mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
fix(recovery): settle a late workspace hold when a remote-epoch continuation already took over (#22387)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Restart recovery must continue a task after a server is killed. A person must never need to press Retry. > - When Paperclip cannot resume a cut-off run exactly, it records a "fresh continuation". For a remote workspace, the continuation abandons the old run's workspace epoch and restores the last save into a new root. It does not apply the old run's unsaved files. > - Workspace recovery can try to save the old run's files after that. The epoch is already abandoned, so the source is "unavailable". Recovery then wrote an active hold that asked a person to press Retry to continue in a new workspace, which the continuation had already arranged. > - That hold made the dispatch gate withhold every queued run of the task, and the task rested on a Retry banner. > - This pull request settles that hold into the recorded continuation at the moment workspace recovery would leave it exhausted. Dispatch and continuation delivery are unchanged. > - The benefit is that the queued continuation starts automatically after a hard kill, with no Retry banner. ## Linked Issues or Issue Description **What happened?** A restart-chaos release-swap wave hard-killed the old server. One task (of 6) then rested on a Retry banner for about 5 minutes. The other 5 tasks passed. The sequence for the failed task: 1. A run started on the incoming server. The wave's injected database delay held its process-identity write for 480 s, so its runner attach timed out. The run saved a same-run resume. 2. Restart recovery found a newer queued run for the task (a child-completion wake). It treated the same-run resume as superseded, and the run ended `failed` with `native_restart_recovery_blocked`. 3. The dispatch gate withheld the queued run (`execution_reconciliation_required`). In the same transaction, it recorded a fresh continuation with `remote_epoch` isolation, which abandoned the old run's remote workspace epoch. 4. 4 s later, workspace preservation for the old run found the abandoned epoch (`source_unavailable`, `use_alternative_strategy`). It left an active hold: "Paperclip cannot recover the unsaved files from the previous run ... Retry continues the task in a new workspace". The retained-source fallback also refused it later, with `workspace_authority_changed`, because the continuation owned the epoch. 5. The gate withheld both bounded retries because of that hold. The retries were spent, and the task showed a Retry banner. 6. A person pressed Retry. Retry started a run in a new workspace, which the recorded continuation had already decided to do. **Expected behavior** The queued continuation starts automatically. No Retry banner shows. **Steps to reproduce** 1. Run the restart-chaos K1 kill variant: 2 release swaps, and the old server is hard-killed 120 s after SIGTERM. 2. Use an OpenCode task on a remote sandbox whose run starts just before the second swap and that gets a child-completion wake while it runs. 3. Watch the task after the kill. Workspace recovery writes the unsaved-files hold, the gate cancels each queued run with `execution_reconciliation_required`, and the task rests on a Retry banner. `server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts` reproduces the database state at step 5. **Paperclip version or commit** `master` at25b5cad7f9(wave) andc41145e72a(base of this branch). ## What Changed - `server/src/services/fresh-continuation-fallback.ts`: new `settleHoldIntoAbandonedEpochContinuation()`. It settles one workspace-recovery hold with the existing `settleLateHolds()` when all of these are true: - the hold is active or escalated, and its saved-work recovery is exhausted; - no person owns it, it has no human-gate evidence, and it is not an unsafe-archive hold; - a fresh-continuation record for the same run has `remote_epoch` isolation and delivery `pending`, `delivered`, `delegated` or `not_requested`; - durable proof shows the run's prepared epoch was abandoned (`isAbandonedPreparedRemoteWorkspace`: an abandoned-epoch receipt for this run, epoch and location, and an authority that moved past it). The hold becomes `resolved` / `restored` (`continued_fresh_from_last_saved`) and is coalesced into the continuation record. Its files are not applied. - `server/src/services/workspace-recovery.ts` calls it at two points: when an attempt records an exhausted failure, and when a recovery pass starts on an already exhausted hold (holds written before this change). - `server/src/services/recovery/human-gate-evidence.ts`: one shared list of human-gate evidence keys (`executionReconciliation`, `OWNER_GONE_ATTESTATION_KEY`, `explicitUserContinuation`), with a SQL helper and a TypeScript helper. Both treat a present key as a person's decision, whatever its value. `isHumanGate`, `completed-turn-cleanup-hold-settlement.ts` and `interrupted-preparation-hold-settlement.ts` use them. - New tests: `server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts`. The dispatch gate, the execution-blocker predicate, the wake-queue lock release, and continuation delivery are unchanged from `master`. An earlier revision of this pull request changed them. That approach was removed because each review round found another dispatch case. `fresh_isolated_workspace` continuations are out of scope on purpose. They do not abandon a remote epoch, so they do not cause this failure. Only a run that carries the isolation marker is protected from the old local directory, so a late hold for such a run keeps blocking. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-isolated-continuation-hold.test.ts` gives 20 of 20 passing. - Recovery settles the late hold for each `remote_epoch` delivery state (`pending`, `delivered`, `delegated`, `not_requested`). The blocker is then empty, and the hold is coalesced into the continuation. - Assembled heartbeat test: after recovery settles the hold, the queued run is the only run that starts. The old run stays failed. With a hold that a person owns, the gate still withholds the queued run. - The hold stays for each of these cases: no continuation, a `remote_epoch` continuation while the run still owns its epoch, a run with no remote workspace, `fresh_isolated_workspace`, `writer_ended`, `invalidated`, `coalesced`, a hold that a person owns, an unsafe archive, each human-gate key, and a null-valued human-gate key. - Red/green: with the two `workspace-recovery.ts` calls disabled, 5 of 20 fail; without the abandoned-epoch check, the 2 epoch cases fail: the 4 settle cases and the assembled start test. - A focused regression run of the server test files for the blocker, fresh continuations, workspace recovery, both hold settlements, owner-gone attestation, and the run-dispatch and wake-queue adapters. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Only exhausted, system-owned workspace-recovery holds of a run whose remote epoch a recorded continuation replaced are settled. All other holds keep blocking. - After settlement, workspace recovery stops for that hold. Its files were already unreachable, because the epoch was abandoned. - A human-gate key with a null or false value now counts as a person's decision in TypeScript too. Before, TypeScript ignored it and SQL counted it. This makes the two sides agree and is more conservative. - No schema change, no migration, and no Runner change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edit, code search) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
414abc2dbf
commit
fe0c4aa95d
7 files changed
+304
-9
No files matched your search
@@ -0,0 +1,250 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq, inArray, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
agentWakeupRequests, agents, authUsers, companies, companyMemberships, createDb, executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues,
|
||||
nativeRunFinalizations,
|
||||
} from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { getExecutionBlocker } from "../services/execution-blocker.ts";
|
||||
import { settleHoldIntoAbandonedEpochContinuation } from "../services/fresh-continuation-fallback.ts";
|
||||
import { HUMAN_GATE_EVIDENCE_KEYS, hasHumanGateEvidence } from "../services/recovery/human-gate-evidence.ts";
|
||||
import { recoverRetainedWorkspace } from "../services/workspace-recovery.ts";
|
||||
import { abandonUnverifiedRemoteWorkspaceEpoch } from "../services/remote-workspace-epoch-abandonment.ts";
|
||||
import { heartbeatService } from "../services/heartbeat.ts";
|
||||
import { runningProcesses } from "../adapters/index.ts";
|
||||
import { createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "../services/system-recovery-state.ts";
|
||||
|
||||
const held = vi.hoisted(() => {
|
||||
const state = {
|
||||
release: () => undefined as void,
|
||||
released: Promise.resolve(),
|
||||
reset() {
|
||||
state.released = new Promise<void>((resolve) => { state.release = resolve; });
|
||||
},
|
||||
};
|
||||
return state;
|
||||
});
|
||||
|
||||
vi.mock("../adapters/index.ts", async () => {
|
||||
const actual = await vi.importActual<typeof import("../adapters/index.ts")>("../adapters/index.ts");
|
||||
return {
|
||||
...actual,
|
||||
getServerAdapter: vi.fn(() => ({
|
||||
supportsLocalAgentJwt: false,
|
||||
execute: async () => {
|
||||
await held.released;
|
||||
return { exitCode: 0, signal: null, timedOut: false, errorMessage: null, summary: "done", provider: "test", model: "test" };
|
||||
},
|
||||
})),
|
||||
};
|
||||
});
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const suite = support.supported ? describe : describe.skip;
|
||||
const minutesAgo = (minutes: number) => new Date(Date.now() - minutes * 60_000);
|
||||
const RESTORE_SCHEMA = "paperclip.workspace-restore-recovery.v1";
|
||||
|
||||
suite("a hold on a run that a fresh continuation already isolated", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("paperclip-isolated-continuation-hold-");
|
||||
db = createDb(temporary.connectionString);
|
||||
}, 60_000);
|
||||
|
||||
afterEach(async () => {
|
||||
held.release();
|
||||
runningProcesses.clear();
|
||||
await db.execute(sql`truncate table ${companies} cascade`);
|
||||
});
|
||||
|
||||
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
|
||||
|
||||
type Shape = {
|
||||
isolation?: "remote_epoch" | "fresh_isolated_workspace" | "writer_ended" | null;
|
||||
personOwned?: boolean;
|
||||
unsafeArchive?: boolean;
|
||||
delivery?: "pending" | "delivered" | "delegated" | "invalidated" | "not_requested" | "coalesced";
|
||||
holdEvidence?: Record<string, unknown>;
|
||||
epoch?: "abandoned" | "still_owned" | "none";
|
||||
};
|
||||
|
||||
async function killedRunAfterFreshContinuation(shape: Shape = {}) {
|
||||
const { isolation = "remote_epoch", personOwned = false, unsafeArchive = false, delivery = "delivered", holdEvidence = {}, epoch = "abandoned" } = shape;
|
||||
const userId = randomUUID(), companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID();
|
||||
const sourceRunId = randomUUID(), continuationId = randomUUID(), holdId = randomUUID();
|
||||
await db.insert(authUsers).values({ id: userId, name: "Board", email: `${userId}@example.test`,
|
||||
emailVerified: true, createdAt: new Date(), updatedAt: new Date() });
|
||||
await db.insert(companies).values({ id: companyId, name: "Kill wave", issuePrefix: `K${companyId.replaceAll("-", "").slice(0, 6)}`,
|
||||
requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: userId });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "owner" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", role: "engineer", status: "idle",
|
||||
adapterType: "codex_local", adapterConfig: {},
|
||||
runtimeConfig: { heartbeat: { enabled: true, wakeOnDemand: true, maxConcurrentRuns: 1 } }, permissions: {} });
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Long task across a hard kill", status: "in_progress",
|
||||
assigneeAgentId: agentId, createdByUserId: userId });
|
||||
|
||||
const recordedAt = minutesAgo(4);
|
||||
const restoreRecovery = { schema: RESTORE_SCHEMA, leaseIds: [randomUUID()] };
|
||||
await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId, agentId, nativeIssueId: issueId, scopeKind: "issue",
|
||||
runtimeMode: "native", nativePhase: "terminal_failure", status: "failed", errorCode: "native_restart_recovery_blocked",
|
||||
invocationSource: "automation", startedAt: minutesAgo(15), finishedAt: minutesAgo(5), createdAt: minutesAgo(16),
|
||||
contextSnapshot: { issueId, taskId: issueId, wakeReason: "issue_commented" },
|
||||
resultJson: { workspaceRestoreFailure: unsafeArchive ? "restore_unsafe_archive" : "restore_failed",
|
||||
workspaceRestoreRecovery: restoreRecovery,
|
||||
...(isolation ? { freshContinuationSupersededAt: recordedAt.toISOString() } : {}) } });
|
||||
if (epoch !== "none") {
|
||||
const workspaceId = randomUUID(), leaseId = randomUUID(), snapshotId = randomUUID();
|
||||
const location = { environmentId: randomUUID(), leaseId, providerLeaseId: `sandbox-${sourceRunId}`,
|
||||
remoteCwd: `/home/daytona/paperclip-workspace/.paperclip-workspaces/${sourceRunId}` };
|
||||
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, mode: "isolated_workspace",
|
||||
strategyType: "project_primary", name: "Task workspace", status: "active", providerType: "cloud_sandbox",
|
||||
cwd: `/workspaces/${workspaceId}`, sourceIssueId: issueId, metadata: { remoteWorkspace: {
|
||||
schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId, source: { kind: "empty" },
|
||||
writerRunId: sourceRunId, dirty: true, location } } });
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1",
|
||||
workspaceId, leaseId, providerLeaseId: location.providerLeaseId, remoteCwd: location.remoteCwd, epoch: 1,
|
||||
state: "prepared", snapshotId } } }).where(eq(heartbeatRuns.id, sourceRunId));
|
||||
if (epoch === "abandoned") await db.transaction(tx => abandonUnverifiedRemoteWorkspaceEpoch(tx, { companyId, workspaceId,
|
||||
runId: sourceRunId, epoch: 1, snapshotId, location }));
|
||||
}
|
||||
await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId: sourceRunId, phase: "terminal_failure",
|
||||
attempt: 1, failureCode: "native_restart_recovery_blocked",
|
||||
failureDetail: isolation ? { replacementDenied: "fresh_continuation" } : {} });
|
||||
|
||||
const queuedRunId = randomUUID(), wakeId = randomUUID();
|
||||
if (isolation) {
|
||||
const freshContinuation = { schema: "paperclip.fresh-continuation.v1", sourceRunId, trigger: "continuation_wake",
|
||||
failureCode: "native_restart_recovery_blocked", delivery: "recorded", recordedAt: recordedAt.toISOString(), isolation,
|
||||
stopProof: isolation === "writer_ended" ? "writer_ended" : "unproven_epoch_abandoned", unverifiedActions: [], createdObjects: [] };
|
||||
const reconciliation = { runId: sourceRunId, providerStopped: true,
|
||||
providerStopProof: isolation === "writer_ended" ? "writer_ended" : "isolated", actionOutcome: "mixed",
|
||||
actorId: "fresh-continuation", recordedAt: recordedAt.toISOString() };
|
||||
await db.insert(issueRecoveryActions).values({ id: continuationId, companyId, sourceIssueId: issueId,
|
||||
kind: "active_run_watchdog", status: "resolved", outcome: "restored", ownerType: "system", returnOwnerAgentId: agentId,
|
||||
cause: "native_continuation_requires_reconciliation", fingerprint: `native-continuation:${sourceRunId}`,
|
||||
nextAction: "Paperclip continues the task in a fresh run from its last saved work.",
|
||||
resolutionNote: "continued_fresh_from_last_saved", resolvedAt: recordedAt, createdAt: recordedAt, updatedAt: recordedAt,
|
||||
evidence: { runId: sourceRunId, executionReconciliation: reconciliation, freshContinuation,
|
||||
continuationDelivery: delivery,
|
||||
...(delivery === "delivered" ? { continuationRunId: queuedRunId, continuationSupersededByRun: true } : {}) } });
|
||||
}
|
||||
|
||||
const spent = failSystemRecovery(reserveSystemRecoveryAttempt(createSystemRecovery(sourceRunId, randomUUID()))!,
|
||||
new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy"), { now: Date.now() - 3 * 60_000 });
|
||||
await db.insert(issueRecoveryActions).values({ id: holdId, companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
||||
status: "active", ownerType: "board", ownerUserId: personOwned ? userId : null, returnOwnerAgentId: agentId,
|
||||
cause: "native_continuation_requires_reconciliation", fingerprint: `workspace-restore:${sourceRunId}`,
|
||||
nextAction: "Paperclip cannot recover the unsaved files from the previous run.",
|
||||
createdAt: minutesAgo(3), updatedAt: minutesAgo(3),
|
||||
evidence: { runId: sourceRunId, workspaceRestoreFailure: unsafeArchive ? "restore_unsafe_archive" : "restore_failed",
|
||||
workspaceRestoreRecovery: restoreRecovery, systemRecovery: spent,
|
||||
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: sourceRunId, status: "blocked", stage: "verifying_stop",
|
||||
attempt: 1, attemptLimit: 6, retryable: true, requestIds: [] }, ...holdEvidence } });
|
||||
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId, agentId, source: "automation", triggerDetail: "system",
|
||||
reason: "issue_children_completed", status: "queued", requestedByActorType: "system", requestedByActorId: "issue-children",
|
||||
payload: { issueId } });
|
||||
await db.insert(heartbeatRuns).values({ id: queuedRunId, companyId, agentId, issueId, invocationSource: "automation",
|
||||
triggerDetail: "system", status: "queued", wakeupRequestId: wakeId, createdAt: minutesAgo(2),
|
||||
scheduledRetryAttempt: 1, scheduledRetryReason: "transient_failure",
|
||||
contextSnapshot: { issueId, taskId: issueId, wakeReason: "issue_children_completed" } });
|
||||
await db.update(agentWakeupRequests).set({ runId: queuedRunId }).where(eq(agentWakeupRequests.id, wakeId));
|
||||
return { userId, companyId, agentId, issueId, sourceRunId, continuationId, holdId, queuedRunId };
|
||||
}
|
||||
|
||||
async function runRow(id: string) {
|
||||
return (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0]!;
|
||||
}
|
||||
|
||||
async function startedRunIds(companyId: string) {
|
||||
const rows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, companyId), sql`${heartbeatRuns.startedAt} is not null`,
|
||||
inArray(heartbeatRuns.status, ["running", "succeeded"])));
|
||||
return rows.map(row => row.id);
|
||||
}
|
||||
|
||||
async function holdRow(id: string) {
|
||||
return (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!;
|
||||
}
|
||||
|
||||
async function recoverHold(f: { companyId: string; issueId: string; sourceRunId: string; holdId: string }) {
|
||||
await recoverRetainedWorkspace({ db, companyId: f.companyId, issueId: f.issueId, runId: f.sourceRunId, actionId: f.holdId,
|
||||
environmentRuntime: {} as Parameters<typeof recoverRetainedWorkspace>[0]["environmentRuntime"] });
|
||||
}
|
||||
|
||||
it.each(["pending", "delivered", "delegated", "not_requested"] as const)(
|
||||
"workspace recovery settles its late hold into a %s remote_epoch continuation of the same run", async delivery => {
|
||||
const f = await killedRunAfterFreshContinuation({ isolation: "remote_epoch", delivery });
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ recoveryActionId: f.holdId });
|
||||
await recoverHold(f);
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
expect(await holdRow(f.holdId)).toMatchObject({ status: "resolved", outcome: "restored",
|
||||
resolutionNote: "continued_fresh_from_last_saved", evidence: { coalescedIntoRecoveryActionId: f.continuationId } });
|
||||
});
|
||||
|
||||
it.each([
|
||||
["no continuation was recorded", { isolation: null }],
|
||||
["the continuation is remote_epoch but the run still owns its epoch", { epoch: "still_owned" }],
|
||||
["the continuation is remote_epoch but the run had no remote workspace", { epoch: "none" }],
|
||||
["the continuation is a local fresh_isolated_workspace", { isolation: "fresh_isolated_workspace" }],
|
||||
["the continuation reuses the old workspace", { isolation: "writer_ended" }],
|
||||
["the continuation was invalidated", { delivery: "invalidated" }],
|
||||
["the continuation was coalesced elsewhere", { delivery: "coalesced" }],
|
||||
["a person owns the hold", { personOwned: true }],
|
||||
["the saved archive is unsafe", { unsafeArchive: true }],
|
||||
["the hold records an owner-gone attestation", { holdEvidence: { ownerGoneAttestation: { attestedBy: "board" } } }],
|
||||
["the hold records an explicit continuation", { holdEvidence: { explicitUserContinuation: { runId: randomUUID() } } }],
|
||||
["the hold records a reconciliation", { holdEvidence: { executionReconciliation: { runId: randomUUID() } } }],
|
||||
["the hold records a null-valued owner-gone attestation", { holdEvidence: { ownerGoneAttestation: null } }],
|
||||
] as const)("keeps the hold when %s", async (_name, shape) => {
|
||||
const f = await killedRunAfterFreshContinuation(shape);
|
||||
expect(await settleHoldIntoAbandonedEpochContinuation(db, { companyId: f.companyId, issueId: f.issueId,
|
||||
runId: f.sourceRunId, actionId: f.holdId })).toBe(false);
|
||||
expect(await holdRow(f.holdId)).toMatchObject({ status: "active" });
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ recoveryActionId: f.holdId });
|
||||
});
|
||||
|
||||
it("treats a present human-gate key as a person's decision in both SQL and TypeScript, whatever its value", () => {
|
||||
for (const key of HUMAN_GATE_EVIDENCE_KEYS) {
|
||||
expect(hasHumanGateEvidence({ [key]: null })).toBe(true);
|
||||
expect(hasHumanGateEvidence({ [key]: false })).toBe(true);
|
||||
}
|
||||
expect(hasHumanGateEvidence({})).toBe(false);
|
||||
expect(hasHumanGateEvidence(null)).toBe(false);
|
||||
});
|
||||
|
||||
it("starts the queued continuation at the execution gate once recovery settled its late hold", async () => {
|
||||
held.reset();
|
||||
const f = await killedRunAfterFreshContinuation({ isolation: "remote_epoch" });
|
||||
const heartbeat = heartbeatService(db);
|
||||
try {
|
||||
await recoverHold(f);
|
||||
await heartbeat.resumeQueuedRunsForAgent(f.companyId, f.agentId);
|
||||
await vi.waitFor(async () => expect((await runRow(f.queuedRunId)).status).not.toBe("queued"), { timeout: 20_000 });
|
||||
expect((await runRow(f.queuedRunId)).errorCode).not.toBe("execution_reconciliation_required");
|
||||
expect(await startedRunIds(f.companyId)).toEqual([f.queuedRunId]);
|
||||
expect(await runRow(f.sourceRunId)).toMatchObject({ status: "failed", errorCode: "native_restart_recovery_blocked" });
|
||||
} finally {
|
||||
held.release();
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
it("still withholds the queued run when a person owns the late hold", async () => {
|
||||
held.reset();
|
||||
const f = await killedRunAfterFreshContinuation({ isolation: "remote_epoch", personOwned: true });
|
||||
const heartbeat = heartbeatService(db);
|
||||
try {
|
||||
await recoverHold(f);
|
||||
await heartbeat.resumeQueuedRunsForAgent(f.companyId, f.agentId);
|
||||
await vi.waitFor(async () => expect((await runRow(f.queuedRunId)).status).toBe("cancelled"), { timeout: 20_000 });
|
||||
expect(await runRow(f.queuedRunId)).toMatchObject({ startedAt: null, errorCode: "execution_reconciliation_required" });
|
||||
expect(await startedRunIds(f.companyId)).toEqual([]);
|
||||
} finally {
|
||||
held.release();
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
}
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -21,7 +21,9 @@ import { issueService } from "./issues.js";
|
||||
import { holdOwnsBlockedStatus } from "./recovery-hold-blocked-status.js";
|
||||
import { proveRunProcessGone } from "./run-process-owner.js";
|
||||
import { hasPendingRunCleanup } from "./execution-blocker.js";
|
||||
import { hasHumanGateEvidence } from "./recovery/human-gate-evidence.js";
|
||||
import { abandonUnverifiedRemoteWorkspaceEpoch } from "./remote-workspace-epoch-abandonment.js";
|
||||
import { isAbandonedPreparedRemoteWorkspace } from "./remote-workspace-abandoned-preparation.js";
|
||||
import { notifyRoutineExecutionStatus } from "./routine-execution-status-notifications.js";
|
||||
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
|
||||
import { readRunCancellation } from "./run-cancellation.js";
|
||||
@@ -294,8 +296,7 @@ async function fenceSourceWorkspace(tx: Db, run: typeof heartbeatRuns.$inferSele
|
||||
* gate again.
|
||||
*/
|
||||
export function isHumanGate(hold: Pick<typeof issueRecoveryActions.$inferSelect, "ownerUserId" | "evidence">): boolean {
|
||||
return Boolean(hold.ownerUserId || hold.evidence.ownerGoneAttestation || hold.evidence.explicitUserContinuation
|
||||
|| hold.evidence.executionReconciliation);
|
||||
return Boolean(hold.ownerUserId) || hasHumanGateEvidence(hold.evidence);
|
||||
}
|
||||
|
||||
const isRestoreHold = (hold: typeof issueRecoveryActions.$inferSelect) =>
|
||||
@@ -702,6 +703,36 @@ export async function continueFreshInTransaction(tx: Db, input: Input): Promise<
|
||||
return { kind: "continued", recoveryActionId: primaryId, delivery, isolation };
|
||||
}
|
||||
|
||||
export async function settleHoldIntoAbandonedEpochContinuation(db: Db, input: {
|
||||
companyId: string; issueId: string; runId: string; actionId: string; now?: Date;
|
||||
}): Promise<boolean> {
|
||||
const now = input.now ?? new Date();
|
||||
return db.transaction(async raw => {
|
||||
const tx = raw as unknown as Db;
|
||||
const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update");
|
||||
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId),
|
||||
eq(heartbeatRuns.id, input.runId)));
|
||||
if (!issue || !run) return false;
|
||||
const [hold] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId),
|
||||
eq(issueRecoveryActions.sourceIssueId, issue.id), eq(issueRecoveryActions.id, input.actionId))).for("update");
|
||||
if (!hold || !["active", "escalated"].includes(hold.status) || isHumanGate(hold) || hold.evidence.runId !== run.id
|
||||
|| hold.evidence.workspaceRestoreFailure === "restore_unsafe_archive"
|
||||
|| readSystemRecovery(hold.evidence)?.state !== "exhausted") return false;
|
||||
const [prior] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId),
|
||||
eq(issueRecoveryActions.sourceIssueId, issue.id), sql`${issueRecoveryActions.id} <> ${hold.id}`,
|
||||
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,schema}' = ${FRESH_CONTINUATION_SCHEMA}`,
|
||||
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,sourceRunId}' = ${run.id}`,
|
||||
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,isolation}' = 'remote_epoch'`,
|
||||
inArray(sql<string>`${issueRecoveryActions.evidence}->>'continuationDelivery'`, ["pending", "delivered", "delegated", "not_requested"])))
|
||||
.orderBy(desc(sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,recordedAt}'`)).limit(1).for("update");
|
||||
if (!prior || !await isAbandonedPreparedRemoteWorkspace(tx, run)) return false;
|
||||
await settleLateHolds(tx, { companyId: input.companyId, issue, run, prior, holds: [hold], now });
|
||||
logger.info({ runId: run.id, issueId: issue.id, recoveryActionId: hold.id, continuationId: prior.id },
|
||||
"Workspace recovery hold settled into the continuation that already moved the task to a new workspace epoch");
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
async function settleLateHolds(tx: Db, input: { companyId: string; issue: typeof issues.$inferSelect; run: typeof heartbeatRuns.$inferSelect;
|
||||
prior: typeof issueRecoveryActions.$inferSelect; holds: Array<typeof issueRecoveryActions.$inferSelect>; now: Date }) {
|
||||
const { companyId, issue, run, prior, holds, now } = input;
|
||||
|
||||
@@ -5,7 +5,7 @@ import { LEGACY_RECOVERY_CAUSE, settleInterruptedNativeBootstrap } from "./legac
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import type { OwnerProof } from "./owner-proof.js";
|
||||
import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
|
||||
import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js";
|
||||
import { humanGateEvidencePredicate } from "./recovery/human-gate-evidence.js";
|
||||
import { abandonedPreparedRemoteWorkspaceExists } from "./remote-workspace-abandoned-preparation.js";
|
||||
|
||||
/**
|
||||
@@ -35,7 +35,7 @@ export function heldInterruptedPreparationWhere() {
|
||||
and hold.status = 'resolved' and hold.outcome = 'blocked' and hold.owner_user_id is null
|
||||
and hold.evidence->'automaticRecovery'->>'policy' = 'preserve_without_replay_v1'
|
||||
and hold.evidence->'automaticRecovery'->>'replay' = 'blocked'
|
||||
and not (hold.evidence ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation'])
|
||||
and not ${humanGateEvidencePredicate(sql`hold.evidence`)}
|
||||
and (not (hold.evidence ?| array['workspaceRestoreFailure', 'workspaceRestoreRecovery'])
|
||||
or ${abandonedPreparedRemoteWorkspaceExists()}))`,
|
||||
sql`not exists (select 1 from environment_leases lease where lease.company_id = ${heartbeatRuns.companyId}
|
||||
|
||||
@@ -7,7 +7,7 @@ import { issueService } from "../issues.js";
|
||||
import { logger } from "../../middleware/logger.js";
|
||||
import type { InProgressRecord, SettleOutcome } from "../recovery/ownership-registry.js";
|
||||
import type { OwnerProof } from "../owner-proof.js";
|
||||
import { OWNER_GONE_ATTESTATION_KEY } from "../recovery/owner-gone-attestation.js";
|
||||
import { humanGateEvidencePredicate } from "../recovery/human-gate-evidence.js";
|
||||
import { holdOwnsBlockedStatus } from "../recovery-hold-blocked-status.js";
|
||||
import { notifyRoutineExecutionStatus } from "../routine-execution-status-notifications.js";
|
||||
import { notifyStatusCardIssueChange } from "../status-card-notifications.js";
|
||||
@@ -66,7 +66,7 @@ function settleableHoldCondition(runId: SQL | string): SQL {
|
||||
isNull(issueRecoveryActions.ownerAgentId),
|
||||
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'policy' = 'preserve_without_replay_v1'`,
|
||||
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
|
||||
sql`not (${issueRecoveryActions.evidence} ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation'])`,
|
||||
sql`not ${humanGateEvidencePredicate(issueRecoveryActions.evidence)}`,
|
||||
sql`(not (${issueRecoveryActions.evidence} ?| array['workspaceRestoreFailure', 'workspaceRestoreRecovery'])
|
||||
or ${completedWorkspaceRecoveryCondition(runId)})`,
|
||||
)!;
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
import { sql, type SQL } from "drizzle-orm";
|
||||
|
||||
export const OWNER_GONE_ATTESTATION_KEY = "ownerGoneAttestation";
|
||||
|
||||
export const HUMAN_GATE_EVIDENCE_KEYS = ["executionReconciliation", OWNER_GONE_ATTESTATION_KEY, "explicitUserContinuation"] as const;
|
||||
|
||||
export function humanGateEvidencePredicate(evidence: SQL | unknown): SQL {
|
||||
return sql`(${evidence} ?| array[${sql.join(HUMAN_GATE_EVIDENCE_KEYS.map(key => sql`${key}`), sql`, `)}])`;
|
||||
}
|
||||
|
||||
export function hasHumanGateEvidence(evidence: Record<string, unknown> | null | undefined): boolean {
|
||||
return Boolean(evidence) && HUMAN_GATE_EVIDENCE_KEYS.some(key => Object.hasOwn(evidence!, key));
|
||||
}
|
||||
@@ -9,8 +9,7 @@ import type { InProgressRecord, OwnableKind } from "./ownership-registry.js";
|
||||
|
||||
/** Recovery action cause: a stuck record whose owner Paperclip cannot judge. */
|
||||
export const OWNER_UNPROVABLE_CAUSE = "ownership_owner_unprovable";
|
||||
/** Lease metadata key the board's confirmation writes; the owner proof reads it as gone. */
|
||||
export const OWNER_GONE_ATTESTATION_KEY = "ownerGoneAttestation";
|
||||
export { OWNER_GONE_ATTESTATION_KEY } from "./human-gate-evidence.js";
|
||||
|
||||
/** Unprovable because the record names no usable owner: saved before owners were recorded. */
|
||||
export const LEGACY_OWNER_REASONS: ReadonlySet<string> = new Set(["no_owner_stamp", "process_space_unrecorded", "missing_identity"]);
|
||||
|
||||
@@ -41,7 +41,7 @@ import { capturePolicyRearmDueCondition, retainedSourceFallbackDueCondition, sto
|
||||
import { rearmExhaustedCapturePolicy } from "./workspace-recovery-capture-policy.js";
|
||||
import { rearmExhaustedStopProof } from "./workspace-recovery-stop-proof.js";
|
||||
import { settleRecoveredWorkspaceRestoreHolds } from "./workspace-restore-hold-settlement.js";
|
||||
import { expediteFreshContinuationChecks } from "./fresh-continuation-fallback.js";
|
||||
import { expediteFreshContinuationChecks, settleHoldIntoAbandonedEpochContinuation } from "./fresh-continuation-fallback.js";
|
||||
import { reusableSandboxNotKeptAfterReplacementSql } from "./reusable-sandbox-kept-marker.js";
|
||||
|
||||
const exportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
|
||||
@@ -206,6 +206,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
|
||||
const { db } = input;
|
||||
await adoptSystemRecovery(db, input);
|
||||
let initial = await loadSystemRecovery(db, input);
|
||||
if (initial?.state === "exhausted" && await settleHoldIntoAbandonedEpochContinuation(db, input)) return;
|
||||
// A hold whose files a sibling saved, or whose source is refused permanently
|
||||
// (including holds saved by older releases), settles without a person.
|
||||
// An adoptable native source keeps its retention path; only the sibling
|
||||
@@ -510,6 +511,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
|
||||
...(cause ? { cause } : {}),
|
||||
});
|
||||
const recorded = await updateSystemRecovery(db, input, current!, failed);
|
||||
if (recorded && failed.state === "exhausted" && await settleHoldIntoAbandonedEpochContinuation(db, input)) return;
|
||||
// A permanent refusal of the recorded source repeats on every retry.
|
||||
// Continue from the last save in this same pass, under this ownership.
|
||||
if (recorded && failed.state === "exhausted" && isUnverifiableSourceFailure(failed.failure)) {
|
||||
|
||||
Reference in new issue
Block a user