diff --git a/doc/plans/2026-10-30-live-recovery-sleep.md b/doc/plans/2026-10-30-live-recovery-sleep.md index a7c075ed292..e045b442e14 100644 --- a/doc/plans/2026-10-30-live-recovery-sleep.md +++ b/doc/plans/2026-10-30-live-recovery-sleep.md @@ -8,7 +8,7 @@ This is the final recovery slice after run dispatch, issue state recovery, and d | Native question cancellation | Cancellable run with committed `nativeQuestionCancellation` marker | Authorized task-completion interaction transaction | Immediate. Existing cancellation owner consumes exact company/run marker. Terminal status removes debt. | | Active execution | Running run | Run lifecycle, controller renewal/revocation, native authority and process evidence writes | Fresh native/legacy controller expiry, saved five-minute preparation grace, or bounded physical-process probe. Local execution promises retain execution ownership. Explicit evidence/manual gates park until their owning producer changes truth. | | Settled native process identity | Terminal native failure/cancellation with persisted process identity | Terminal execution writes, server process identity/stop receipts, environment settlement | Immediate inspection. Only matched local host/PID/group/birth or confirmed remote stop evidence clears identity. A live matched process or local execution owner keeps a bounded probe. Missing authority waits for committed evidence. Startup reconstructs rows independently of the old capped process Map. Successful release removes any obsolete compatibility Map entry. | -| Orphan active environment lease | Active lease with NULL run reference or terminal same-company run | Environment owning transactions and indexed physical-resource peer notifications | Saved lease updatedAt plus five minutes, re-read before due work. Existing export, shutdown, and shared-resource fences remain. Exact-lease dispatch does not mutate timestamps for broad-page fairness. Successful CAS hands work to the existing pending-cleanup owner. | +| Orphan active environment lease | Active lease with NULL run reference or terminal same-company run | Environment owning transactions and indexed physical-resource peer notifications | Saved lease updatedAt plus five minutes, re-read before due work. Existing export, shutdown, and shared-resource fences remain. Workspace preservation lock/statement timeouts retain a bounded thirty-second retry; claim/evidence gates still park. Exact-lease dispatch does not mutate timestamps for broad-page fairness. Successful CAS hands work to the existing pending-cleanup owner. | | Legacy shutdown settlement | Exact lease with pending shutdown marker | Begin, renew, finish, expiry settlement owning transactions | Exact saved marker deadline. Existing owner revalidates boot/controller, source, competing allocations and stop/copyback authority. Expiry records preserved workspace recovery; it never infers file export from process loss. | | Stranded agent status | Ready agent with status running | Agent configuration/lifecycle/instruction settlement, run/resource transitions, accounting acknowledgement, canonical wake status writes, and direct existing-receipt terminal writers | Saved updatedAt plus five minutes, then actual surviving controller expiry when present. Fresh settlement rechecks live execution, native controller, working copy, and lease truth. Settled status removes debt. | | Active output silence | Running run and coalesced last output/process start/run start/created timestamp | Actual output progress transaction, terminal source mutation, same-run terminal activity commit, watchdog decisions, execution state | Five-minute threshold, exact saved snooze/continue expiry, or physical termination probe. Fresh check re-reads output and decision. Existing watchdog alone owns completion evidence, process cleanup and fold. Suppressed/manual/blocked history does not poll. | @@ -37,3 +37,9 @@ A run terminal commit activates resource recovery even if cleanup stops before a The stranded-agent predicate depends on final accounting and wake receipts as well as runs. `accountRunCost` publishes before acknowledgement, including stale acknowledgement cleanup, with no budget policy required. The canonical wake status writer publishes the known agent before mutation. Direct receipt writers in wake-queue cancellation/authorization, task Retry retirement/chat settlement, late chat authorization failure, Slack stop, tree cancellation and assignment cancellation do the same. Run-coupled queue/retry cancellation paths use execution-control notifications. Newly inserted already-terminal receipts do not remove a prior blocker and need no such signal. Assignment cancellation selects only actual unfinished receipts with the canonical idle proof; historical cancellation write semantics remain unchanged. Each producer regression starts with a worker already parked and no unrelated budget/resource transition. The final receipt controls require durable agent status settlement, no scheduler deadline/idle hold, and twenty minutes with zero SQL. The native gate and terminal-resource controls captured the original failure before the missing topic fanout was added. + +## Workspace preservation contention + +The exact-lease owner forwards an optional deadline callback through the existing preservation helper. Only PostgreSQL lock-unavailable (`55P03`) and statement-cancelled (`57014`) outcomes retain a thirty-second retry. A rolled-back write intent may cause earlier bounded reconciliation. `NativeWorkspaceFinalizationBusyError` remains an ownership refusal, and wrong-company or manual/evidence gates do not acquire periodic retries. Legacy explicit callers keep the existing boolean contract. + +The regression holds the real task row `FOR UPDATE` until the existing one-second lock timeout fires, then releases only the transaction, without a mutation or notification. The registered worker retains its deadline and records workspace preservation exactly once on retry, handing the original allocation to stop-only pending cleanup with `retain_on_failure`. A second fixture exceeds the actual fifteen-second statement timeout with a one-shot PostgreSQL trigger. Both require the twenty-minute zero-SQL idle proof after the known producer fanout settles; wrong-company, active-claim and manual-gate controls remain parked. diff --git a/server/src/__tests__/live-recovery-work.test.ts b/server/src/__tests__/live-recovery-work.test.ts index 70abd4a0043..1f53933a73b 100644 --- a/server/src/__tests__/live-recovery-work.test.ts +++ b/server/src/__tests__/live-recovery-work.test.ts @@ -29,6 +29,7 @@ import { createAgentLifecycle, configureAgentLifecycle } from "../services/agent import { issueTreeControlService } from "../services/issue-tree-control.js"; import { terminalizeLegacyExecution } from "../services/legacy-execution-recovery.js"; import { issueService } from "../services/issues.js"; +import { logger } from "../middleware/logger.js"; vi.setConfig({ testTimeout: 90_000 }); describe("residual recovery production matrix", () => { @@ -372,6 +373,89 @@ describe("residual recovery production matrix", () => { }); await quiet(); }); + async function preparedOrphan() { + const f = await fixture(), old = new Date(Date.now() - 6 * 60_000); + const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, + status: "failed", runtimeMode: "native", nativeIssueId: f.issue.id, issueId: f.issue.id, + finishedAt: old, updatedAt: old }).returning(); + const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, + heartbeatRunId: run.id, issueId: f.issue.id, status: "active", provider: "daytona", + providerLeaseId: `sandbox-${run.id}`, metadata: { driver: "sandbox", remoteCwd: "/workspace" }, updatedAt: old }).returning(); + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: { + schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64), + baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId: lease.id, + providerLeaseId: lease.providerLeaseId, remoteCwd: "/workspace", resourceDisposition: "destroy", + } } }).where(eq(heartbeatRuns.id, run.id)); + await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, + runId: run.id, phase: "terminal_failure", recoveryState: "blocked" }); + return { f, run, lease }; + } + it.each(["task lock", "statement timeout"])("retries workspace preservation after real PostgreSQL %s without a producer mutation", async kind => { + const { f, run, lease } = await preparedOrphan(); + const warnings = vi.spyOn(logger, "warn"); + let retryAt = 0; + if (kind === "statement timeout") { + await db.execute(sql`create sequence live_preservation_slow_once`); + await db.execute(sql.raw(`create function live_preservation_slow_once() returns trigger language plpgsql as $$ + begin if new.id = '${run.id}'::uuid and new.result_json->>'workspaceRestoreFailure' = 'restore_failed' + and nextval('live_preservation_slow_once') = 1 then perform pg_sleep(16); end if; return new; end $$`)); + await db.execute(sql`create trigger live_preservation_slow_once before update on heartbeat_runs for each row execute function live_preservation_slow_once()`); + extraStops.push(async () => { + await db.execute(sql`drop trigger if exists live_preservation_slow_once on heartbeat_runs`); + await db.execute(sql`drop function if exists live_preservation_slow_once()`); + await db.execute(sql`drop sequence if exists live_preservation_slow_once`); + }); + } + let release!: () => void, locked!: () => void; + const held = new Promise(resolve => { release = resolve; }); + const ready = new Promise(resolve => { locked = resolve; }); + const lock = kind === "task lock" ? db.transaction(async tx => { + await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, f.issue.id)).for("update"); + locked(); await held; + }) : Promise.resolve(); + if (kind !== "task lock") locked(); + try { + await ready; await start(); + expect((await getRun(run.id)).resultJson).toBeNull(); + expect((await getLease(lease.id)).status).toBe("active"); + expect(warnings).toHaveBeenCalledWith(expect.objectContaining({ runId: run.id, + code: kind === "task lock" ? "55P03" : "57014" }), expect.any(String)); + retryAt = deadlines.get("orphan-active-leases")!()!; + expect(retryAt).toBeGreaterThan(Date.now()); + expect(retryAt).toBeLessThanOrEqual(Date.now() + 30_000); + if (kind === "task lock") expect(retryAt).toBe(Date.now() + 30_000); + } finally { release(); await lock; } + await advance(retryAt - Date.now() - 1); expect((await getRun(run.id)).resultJson).toBeNull(); + await advance(1); + await vi.waitFor(async () => expect((await getRun(run.id)).resultJson).toMatchObject({ + workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { leaseIds: [lease.id] }, + })); + expect(await db.select().from(activityLog).where(and(eq(activityLog.runId, run.id), + eq(activityLog.action, "issue.workspace_recovery_required")))).toHaveLength(1); + expect(await getLease(lease.id)).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure" }); + await quiet(); + expect(await db.select().from(activityLog).where(and(eq(activityLog.runId, run.id), + eq(activityLog.action, "issue.workspace_recovery_required")))).toHaveLength(1); + }); + it.each(["company", "claim", "manual"])("parks workspace preservation authority gates without timeout retries: %s", async gate => { + const { f, run, lease } = await preparedOrphan(); + if (gate === "company") { + const other = await fixture(); + await db.update(environmentLeases).set({ companyId: other.companyId }).where(eq(environmentLeases.id, lease.id)); + } else if (gate === "claim") { + await db.update(nativeRunFinalizations).set({ leaseOwner: "another-workspace-owner", + leaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(nativeRunFinalizations.runId, run.id)); + } else { + await db.update(nativeRunFinalizations).set({ phase: "observed", recoveryState: "blocked" }) + .where(eq(nativeRunFinalizations.runId, run.id)); + } + await start(); + expect((await getRun(run.id)).resultJson).toBeNull(); + expect((await getLease(lease.id)).status).toBe("active"); + expect(deadlines.get("orphan-active-leases")!()).toBeNull(); + expect(await db.select().from(activityLog).where(eq(activityLog.action, "issue.workspace_recovery_required"))).toEqual([]); + await quiet(); + }); it("retains failed orphan work with bounded backoff after a real PostgreSQL write failure", async () => { const f = await fixture(), old = new Date(Date.now() - 6 * 60_000); const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, status: "active", provider: "local", updatedAt: old }).returning(); diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index 6a9eff739bb..43aa81cffb9 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -1496,6 +1496,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD backoffMs: number; companyId?: string; leaseId?: string; + onDeferredDeadline?: (at: number) => void; }): Promise<{ recovered: number }> { const cutoff = new Date(Date.now() - opts.backoffMs); @@ -1528,7 +1529,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD if (await nativeWorkspaceExportPendingForLease(db, lease)) { if (lease.heartbeatRunId && !activeRunExecutions.has(lease.heartbeatRunId) && !adapterExecutionControls.has(lease.heartbeatRunId) - && await preserveOrphanedNativeWorkspace(db, lease)) recovered += 1; + && await preserveOrphanedNativeWorkspace(db, lease, { onDeferredDeadline: opts.onDeferredDeadline })) recovered += 1; if (!opts.leaseId) await deferOrphanedActiveLease(lease.id); continue; } diff --git a/server/src/services/native-runtime/native-failed-workspace-recovery.ts b/server/src/services/native-runtime/native-failed-workspace-recovery.ts index a1c6c914b33..a73471973cd 100644 --- a/server/src/services/native-runtime/native-failed-workspace-recovery.ts +++ b/server/src/services/native-runtime/native-failed-workspace-recovery.ts @@ -91,7 +91,7 @@ export async function preserveNativeFailedWorkspace(db: Db, selected: Pick): Promise { + "companyId" | "heartbeatRunId">, options: { onDeferredDeadline?: (at: number) => void } = {}): Promise { if (!lease.heartbeatRunId) return false; const [run] = await db.select().from(heartbeatRuns).where(and( eq(heartbeatRuns.companyId, lease.companyId), eq(heartbeatRuns.id, lease.heartbeatRunId), @@ -105,6 +105,7 @@ export async function preserveOrphanedNativeWorkspace(db: Db, lease: Pick