From 89ab7e51ff87dd3a7db8751d2f6330e5c21dd211 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 13 Nov 2026 11:58:00 -0800 Subject: [PATCH] fix(recovery): retain a failed run's sandbox in one lease write (#21474) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs execute in remote sandboxes. Each run holds an environment lease row that points to its environment, task, workspace and company > - When a run fails before it copies its files back, Paperclip keeps the sandbox for a file repair. It writes a stop-and-retain intent and a repair-hold policy on the lease > - The recorder wrote the same lease row two times in one transaction. Postgres runs the foreign-key checks again on a row that the current transaction already wrote, and these checks take `FOR KEY SHARE` locks on the referenced rows > - Other runs lock the shared environment row `FOR UPDATE` when they acquire a lease. Then the second write waits, meets the 1 second lock timeout, and the run fails with "Database query failed (55P03)" > - This pull request writes the stop intent and the repair hold in one update > - The benefit is that a busy shared environment no longer fails runs, and the file repair hold is recorded ## Linked Issues or Issue Description **What happened?** On staging, native OpenCode runs failed with `adapter_failed` and the error "Database query failed (55P03)". The failed statement was `update "environment_leases" set "lease_policy" = 'retain_on_failure', "metadata" = "metadata" || {workspaceRestoreRecovery...} where ... "status" = 'pending_cleanup'` in `preserveLegacyWorkspaceRestoreSources`. The cause was `canceling statement due to lock timeout`. The statement does not wait on the lease row, because the transaction already holds it `FOR UPDATE`. It waits on the foreign-key check `SELECT 1 FROM ONLY "public"."environments" x WHERE "id" = $1 FOR KEY SHARE OF x`. Postgres runs this check because the transaction wrote the same row one statement before, in `prepareSandboxStopAndRetain`. **What did you expect?** The run records its workspace restore failure and keeps its sandbox for repair, also when another run holds the shared environment row. **Steps to reproduce** 1. Start a run with an active, ephemeral sandbox lease. 2. In another session, lock the lease's `environments` row with `SELECT ... FOR UPDATE` and keep the transaction open. 3. Record a workspace restore failure for the run (`terminalizeLegacyExecution`, legacy or native `recordRestoreFailureOnly`). 4. Before this change, the second lease write fails after 1 second with 55P03. The new test `server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts` shows this for legacy and native runs. No public issue exists. I searched open and recent pull requests for this lock timeout and found no duplicate. #21071 bounds a different lock wait (workspace source locks in run preparation) and does not overlap. ## What Changed - `server/src/services/sandbox-stop-and-retain.ts`: `prepareSandboxStopAndRetain` accepts an optional `retention` argument (`leasePolicy` and extra `metadata`). The stop-and-retain update sets these values in the same statement. Other callers do not change. - `server/src/services/legacy-workspace-restore-recovery.ts`: the active-source path passes the repair hold (`retain_on_failure` and `workspaceRestoreRecovery` with `originalLeasePolicy`) to that update. It no longer does a second update on the same row. - `server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts`: a regression test for legacy and native runs. Another session holds the environment row `FOR UPDATE` while the run records its restore failure. The test checks the run's recovery evidence and the lease policy, status, stop intent and repair metadata. ## Verification - `cd server && npx vitest run src/__tests__/legacy-workspace-restore-lock-contention.test.ts`: both cases fail on `master` with `canceling statement due to lock timeout` on the `environments` `FOR KEY SHARE` check. Both cases pass with this change. - `cd server && npx vitest run src/__tests__/legacy-workspace-restore-recovery.test.ts`: 73 existing tests pass. - Related suites pass (10 files, 485 tests): `heartbeat-observed-workspace-preservation`, `heartbeat-native-remote-failure-retention`, `live-recovery-work`, `workspace-recovery-stop-proof`, `native-failed-workspace-recovery`, `native-cancelled-workspace-preservation`, `workspace-recovery-admission`, `native-workspace-reconstruction`, `environment-retained-orphan-sandbox`, `abandoned-workspace-files-hold`. - I did not complete a local server typecheck. The process ran out of memory in the 10 GB sandbox. CI runs the typecheck on this head. ## Risks - Low risk. The lease ends with the same status, policy and metadata as before. The values are written in one statement instead of two. The update keeps its ownership guards (status, provider, provider lease ID and exact metadata). - Other code paths can still write a lease row two times in one transaction (for example the shutdown settlement path, which sets a lease back to `active` before it records the restore failure). That path handles 55P03 as a lock timeout and retries. This pull request does not change it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, code execution, embedded Postgres tests, and GitHub tools). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- ...-workspace-restore-lock-contention.test.ts | 66 +++++++++++++++++++ .../legacy-workspace-restore-recovery.ts | 13 ++-- .../src/services/sandbox-stop-and-retain.ts | 15 +++-- 3 files changed, 81 insertions(+), 13 deletions(-) create mode 100644 server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts diff --git a/server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts b/server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts new file mode 100644 index 00000000000..938de577029 --- /dev/null +++ b/server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts @@ -0,0 +1,66 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRuns, issues, projects } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { terminalizeLegacyExecution } from "../services/legacy-execution-recovery.js"; +import { hasRequiredWorkspaceRecovery } from "../services/workspace-restore-recovery-state.js"; + +const externalTestDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL?.trim(); +const support = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("workspace restore hold under environment lock contention", () => { + let db: ReturnType; + let temporary: Awaited>; + beforeAll(async () => { if (externalTestDatabaseUrl) db = createDb(externalTestDatabaseUrl); + else { temporary = await startEmbeddedPostgresTestDatabase("workspace-restore-lock-"); db = createDb(temporary.connectionString); } }, 30_000); + afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); }); + + async function seed(runtimeMode: string) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Workspace recovery", issuePrefix: companyId.slice(0, 8) }); + await db.insert(agents).values({ id: agentId, companyId, name: "Engineer", role: "engineer", adapterType: "codex_local" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Restore files", status: "in_progress", assigneeAgentId: agentId }); + const [project] = await db.insert(projects).values({ companyId, name: "Workspace project" }).returning(); + const [workspace] = await db.insert(executionWorkspaces).values({ companyId, projectId: project.id, + sourceIssueId: issueId, mode: "isolated", strategyType: "git_worktree", name: "Fixture workspace", cwd: "/work/fixture", + }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, status: "failed", runtimeMode, + finishedAt: new Date(), contextSnapshot: { issueId, executionWorkspaceId: workspace.id }, + }).returning(); + const [environment] = await db.insert(environments).values({ name: `Shared sandbox ${runId}`, driver: "sandbox" }).returning(); + const [lease] = await db.insert(environmentLeases).values({ companyId, issueId, executionWorkspaceId: workspace.id, heartbeatRunId: runId, + environmentId: environment.id, status: "active", leasePolicy: "ephemeral", provider: "daytona", providerLeaseId: randomUUID(), + metadata: { remoteCwd: "/work/fixture", workspaceRealization: { authoritativeRoot: "/work/fixture" }, driver: "sandbox", + pluginId: randomUUID(), sandboxProviderPlugin: true }, + }).returning(); + return { runId, run, lease }; + } + + // A second write to a row this transaction already wrote re-runs its + // foreign-key checks, which take KEY SHARE locks on the referenced rows. + // Another run's lease acquisition holds the shared environment FOR UPDATE. + it.each(["legacy", "native"])("retains an active %s source while another run holds its environment", async (runtimeMode) => { + const f = await seed(runtimeMode); + let locked!: () => void, release!: () => void; + const isLocked = new Promise(resolve => { locked = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const holder = db.transaction(async tx => { + await tx.select({ id: environments.id }).from(environments).where(eq(environments.id, f.lease.environmentId!)).for("update"); + locked(); + await released; + }); + await isLocked; + let returned: Awaited>; + try { + returned = await terminalizeLegacyExecution({ db, run: f.run, status: "failed", recordRestoreFailureOnly: runtimeMode === "native", + workspaceRestoreSource: f.lease, patch: { resultJson: { workspaceRestoreFailure: "restore_failed" } } }); + } finally { release(); await holder; } + expect(hasRequiredWorkspaceRecovery(returned!.resultJson)).toBe(true); + const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)); + expect(lease).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure", metadata: expect.objectContaining({ + remoteCwd: "/work/fixture", + workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", runId: f.runId, originalLeasePolicy: "ephemeral" }, + sandboxStopAndRetain: expect.objectContaining({ leaseId: f.lease.id, runId: f.runId }), + }) }); + }); +}); diff --git a/server/src/services/legacy-workspace-restore-recovery.ts b/server/src/services/legacy-workspace-restore-recovery.ts index 99a5ceb5691..9c448052608 100644 --- a/server/src/services/legacy-workspace-restore-recovery.ts +++ b/server/src/services/legacy-workspace-restore-recovery.ts @@ -108,18 +108,13 @@ export async function preserveLegacyWorkspaceRestoreSources(db: Db, run: Run, so // The stop-only worker rechecks exact allocation ownership, including // competing leases, before dispatch. Persist the intent even when cleanup // must wait, so no terminal-run sweep can fall back to destruction. - const prepared = await prepareSandboxStopAndRetain(db, lease); - if (!prepared) throw new Error("Workspace recovery lease ownership changed before terminal commit."); // The original policy lets the settled hold return a warm sandbox to reuse // (settleRecoveredWorkspaceRestoreHolds). It grants nothing while held. - await db.update(environmentLeases).set({ leasePolicy: "retain_on_failure", - metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ workspaceRestoreRecovery: { + const prepared = await prepareSandboxStopAndRetain(db, lease, undefined, { leasePolicy: "retain_on_failure", + metadata: { workspaceRestoreRecovery: { schema: LEGACY_WORKSPACE_RECOVERY_SCHEMA, runId: run.id, originalLeasePolicy: lease.leasePolicy, - } })}::jsonb`, - }).where(and( - eq(environmentLeases.id, prepared.id), eq(environmentLeases.companyId, run.companyId), - eq(environmentLeases.heartbeatRunId, run.id), eq(environmentLeases.status, "pending_cleanup"), - )); + } } }); + if (!prepared) throw new Error("Workspace recovery lease ownership changed before terminal commit."); retained.push(lease.id); } return retained; diff --git a/server/src/services/sandbox-stop-and-retain.ts b/server/src/services/sandbox-stop-and-retain.ts index aea0762201a..20f8c2de44a 100644 --- a/server/src/services/sandbox-stop-and-retain.ts +++ b/server/src/services/sandbox-stop-and-retain.ts @@ -96,9 +96,13 @@ export function readStopOnlyCleanup(lease: Lease) { * an explicit stop into ordinary destructive release. Invalid pins stay pending. * `recovery` marks an ownership settle on the intent and on the lease * (`ORPHANED_SANDBOX_RECOVERY_KEY`); the pending-cleanup sweep dispatches such - * a stop's first attempt without its retry backoff. */ -export async function prepareSandboxStopAndRetain(db: Db, lease: Lease, recovery?: Record) { - return prepareStopAndRetain(db, lease, "active", recovery); + * a stop's first attempt without its retry backoff. `retention` sets a repair + * hold's policy and metadata in the same write: a second write to the row in + * one transaction re-runs its foreign-key checks, which wait on any run that + * holds the shared environment row. */ +export async function prepareSandboxStopAndRetain(db: Db, lease: Lease, recovery?: Record, + retention?: { leasePolicy: string; metadata: Record }) { + return prepareStopAndRetain(db, lease, "active", recovery, retention); } /** Historical release rows can lack a stop receipt. Re-inspect the exact @@ -189,7 +193,8 @@ export async function prepareUnconfirmedSourceSandboxStop(db: Db, lease: typeof return prepareStopAndRetain(db, lease, lease.status); } -async function prepareStopAndRetain(db: Db, lease: Lease, fromStatus: string, recovery: Record = {}) { +async function prepareStopAndRetain(db: Db, lease: Lease, fromStatus: string, recovery: Record = {}, + retention?: { leasePolicy: string; metadata: Record }) { const requestId = randomUUID(), now = new Date(); const intent = { schema: "paperclip.sandbox-stop-and-retain.v1", requestId, companyId: lease.companyId, runId: lease.heartbeatRunId, leaseId: lease.id, @@ -198,12 +203,14 @@ async function prepareStopAndRetain(db: Db, lease: Lease, fromStatus: string, re ? { builtinProvider: lease.provider } : { pluginId: lease.metadata?.pluginId }), ...recovery }; const [updated] = await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ status: "pending_cleanup", cleanupStatus: "failed", failureReason: "sandbox_stop_pending", releasedAt: now, updatedAt: now, + ...(retention ? { leasePolicy: retention.leasePolicy } : {}), metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'remoteExecutionTermination' - 'sandboxStopAndRetainReceipt') || ${JSON.stringify({ [SANDBOX_STOP_AND_RETAIN_KEY]: intent, pendingCleanupAttemptId: requestId, ...(fromStatus === "active" && Object.keys(recovery).length ? { [ORPHANED_SANDBOX_RECOVERY_KEY]: { ...recovery, runId: lease.heartbeatRunId, recoveredAt: now.toISOString() } } : {}), pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0, pendingCleanupRetryAfterMs: 0, pendingCleanupRetryAttempts: 0, pendingCleanupRetryCapWarned: false, + ...retention?.metadata, })}::jsonb`, }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId), eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.status, fromStatus),