fix(recovery): retain a failed run's sandbox in one lease write (#21474)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute in remote sandboxes. Each run holds an
environment lease row that points to its environment, task, workspace
and company
> - When a run fails before it copies its files back, Paperclip keeps
the sandbox for a file repair. It writes a stop-and-retain intent and a
repair-hold policy on the lease
> - The recorder wrote the same lease row two times in one transaction.
Postgres runs the foreign-key checks again on a row that the current
transaction already wrote, and these checks take `FOR KEY SHARE` locks
on the referenced rows
> - Other runs lock the shared environment row `FOR UPDATE` when they
acquire a lease. Then the second write waits, meets the 1 second lock
timeout, and the run fails with "Database query failed (55P03)"
> - This pull request writes the stop intent and the repair hold in one
update
> - The benefit is that a busy shared environment no longer fails runs,
and the file repair hold is recorded

## Linked Issues or Issue Description

**What happened?**
On staging, native OpenCode runs failed with `adapter_failed` and the
error "Database query failed (55P03)". The failed statement was `update
"environment_leases" set "lease_policy" = 'retain_on_failure',
"metadata" = "metadata" || {workspaceRestoreRecovery...} where ...
"status" = 'pending_cleanup'` in
`preserveLegacyWorkspaceRestoreSources`. The cause was `canceling
statement due to lock timeout`. The statement does not wait on the lease
row, because the transaction already holds it `FOR UPDATE`. It waits on
the foreign-key check `SELECT 1 FROM ONLY "public"."environments" x
WHERE "id" = $1 FOR KEY SHARE OF x`. Postgres runs this check because
the transaction wrote the same row one statement before, in
`prepareSandboxStopAndRetain`.

**What did you expect?**
The run records its workspace restore failure and keeps its sandbox for
repair, also when another run holds the shared environment row.

**Steps to reproduce**
1. Start a run with an active, ephemeral sandbox lease.
2. In another session, lock the lease's `environments` row with `SELECT
... FOR UPDATE` and keep the transaction open.
3. Record a workspace restore failure for the run
(`terminalizeLegacyExecution`, legacy or native
`recordRestoreFailureOnly`).
4. Before this change, the second lease write fails after 1 second with
55P03. The new test
`server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts`
shows this for legacy and native runs.

No public issue exists. I searched open and recent pull requests for
this lock timeout and found no duplicate. #21071 bounds a different lock
wait (workspace source locks in run preparation) and does not overlap.

## What Changed

- `server/src/services/sandbox-stop-and-retain.ts`:
`prepareSandboxStopAndRetain` accepts an optional `retention` argument
(`leasePolicy` and extra `metadata`). The stop-and-retain update sets
these values in the same statement. Other callers do not change.
- `server/src/services/legacy-workspace-restore-recovery.ts`: the
active-source path passes the repair hold (`retain_on_failure` and
`workspaceRestoreRecovery` with `originalLeasePolicy`) to that update.
It no longer does a second update on the same row.
-
`server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts`:
a regression test for legacy and native runs. Another session holds the
environment row `FOR UPDATE` while the run records its restore failure.
The test checks the run's recovery evidence and the lease policy,
status, stop intent and repair metadata.

## Verification

- `cd server && npx vitest run
src/__tests__/legacy-workspace-restore-lock-contention.test.ts`: both
cases fail on `master` with `canceling statement due to lock timeout` on
the `environments` `FOR KEY SHARE` check. Both cases pass with this
change.
- `cd server && npx vitest run
src/__tests__/legacy-workspace-restore-recovery.test.ts`: 73 existing
tests pass.
- Related suites pass (10 files, 485 tests):
`heartbeat-observed-workspace-preservation`,
`heartbeat-native-remote-failure-retention`, `live-recovery-work`,
`workspace-recovery-stop-proof`, `native-failed-workspace-recovery`,
`native-cancelled-workspace-preservation`,
`workspace-recovery-admission`, `native-workspace-reconstruction`,
`environment-retained-orphan-sandbox`, `abandoned-workspace-files-hold`.
- I did not complete a local server typecheck. The process ran out of
memory in the 10 GB sandbox. CI runs the typecheck on this head.

## Risks

- Low risk. The lease ends with the same status, policy and metadata as
before. The values are written in one statement instead of two. The
update keeps its ownership guards (status, provider, provider lease ID
and exact metadata).
- Other code paths can still write a lease row two times in one
transaction (for example the shutdown settlement path, which sets a
lease back to `active` before it records the restore failure). That path
handles 55P03 as a lock timeout and retries. This pull request does not
change it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, code execution, embedded Postgres tests, and
GitHub tools).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Devin FoleyandClaude Opus 5.5 authored and GitHub committed 2026-11-13 11:58:00 -08:00
1 parent 7866e9ef05
commit 89ab7e51ff
3 files changed
+81 -13

No files matched your search

@@ -0,0 +1,66 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { agents, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRuns, issues, projects } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { terminalizeLegacyExecution } from "../services/legacy-execution-recovery.js";
import { hasRequiredWorkspaceRecovery } from "../services/workspace-restore-recovery-state.js";
const externalTestDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL?.trim();
const support = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("workspace restore hold under environment lock contention", () => {
let db: ReturnType<typeof createDb>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
beforeAll(async () => { if (externalTestDatabaseUrl) db = createDb(externalTestDatabaseUrl);
else { temporary = await startEmbeddedPostgresTestDatabase("workspace-restore-lock-"); db = createDb(temporary.connectionString); } }, 30_000);
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
async function seed(runtimeMode: string) {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Workspace recovery", issuePrefix: companyId.slice(0, 8) });
await db.insert(agents).values({ id: agentId, companyId, name: "Engineer", role: "engineer", adapterType: "codex_local" });
await db.insert(issues).values({ id: issueId, companyId, title: "Restore files", status: "in_progress", assigneeAgentId: agentId });
const [project] = await db.insert(projects).values({ companyId, name: "Workspace project" }).returning();
const [workspace] = await db.insert(executionWorkspaces).values({ companyId, projectId: project.id,
sourceIssueId: issueId, mode: "isolated", strategyType: "git_worktree", name: "Fixture workspace", cwd: "/work/fixture",
}).returning();
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, status: "failed", runtimeMode,
finishedAt: new Date(), contextSnapshot: { issueId, executionWorkspaceId: workspace.id },
}).returning();
const [environment] = await db.insert(environments).values({ name: `Shared sandbox ${runId}`, driver: "sandbox" }).returning();
const [lease] = await db.insert(environmentLeases).values({ companyId, issueId, executionWorkspaceId: workspace.id, heartbeatRunId: runId,
environmentId: environment.id, status: "active", leasePolicy: "ephemeral", provider: "daytona", providerLeaseId: randomUUID(),
metadata: { remoteCwd: "/work/fixture", workspaceRealization: { authoritativeRoot: "/work/fixture" }, driver: "sandbox",
pluginId: randomUUID(), sandboxProviderPlugin: true },
}).returning();
return { runId, run, lease };
}
// A second write to a row this transaction already wrote re-runs its
// foreign-key checks, which take KEY SHARE locks on the referenced rows.
// Another run's lease acquisition holds the shared environment FOR UPDATE.
it.each(["legacy", "native"])("retains an active %s source while another run holds its environment", async (runtimeMode) => {
const f = await seed(runtimeMode);
let locked!: () => void, release!: () => void;
const isLocked = new Promise<void>(resolve => { locked = resolve; });
const released = new Promise<void>(resolve => { release = resolve; });
const holder = db.transaction(async tx => {
await tx.select({ id: environments.id }).from(environments).where(eq(environments.id, f.lease.environmentId!)).for("update");
locked();
await released;
});
await isLocked;
let returned: Awaited<ReturnType<typeof terminalizeLegacyExecution>>;
try {
returned = await terminalizeLegacyExecution({ db, run: f.run, status: "failed", recordRestoreFailureOnly: runtimeMode === "native",
workspaceRestoreSource: f.lease, patch: { resultJson: { workspaceRestoreFailure: "restore_failed" } } });
} finally { release(); await holder; }
expect(hasRequiredWorkspaceRecovery(returned!.resultJson)).toBe(true);
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
expect(lease).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure", metadata: expect.objectContaining({
remoteCwd: "/work/fixture",
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", runId: f.runId, originalLeasePolicy: "ephemeral" },
sandboxStopAndRetain: expect.objectContaining({ leaseId: f.lease.id, runId: f.runId }),
}) });
});
});
@@ -108,18 +108,13 @@ export async function preserveLegacyWorkspaceRestoreSources(db: Db, run: Run, so
// The stop-only worker rechecks exact allocation ownership, including
// competing leases, before dispatch. Persist the intent even when cleanup
// must wait, so no terminal-run sweep can fall back to destruction.
const prepared = await prepareSandboxStopAndRetain(db, lease);
if (!prepared) throw new Error("Workspace recovery lease ownership changed before terminal commit.");
// The original policy lets the settled hold return a warm sandbox to reuse
// (settleRecoveredWorkspaceRestoreHolds). It grants nothing while held.
await db.update(environmentLeases).set({ leasePolicy: "retain_on_failure",
metadata: sql`${environmentLeases.metadata} || ${JSON.stringify({ workspaceRestoreRecovery: {
const prepared = await prepareSandboxStopAndRetain(db, lease, undefined, { leasePolicy: "retain_on_failure",
metadata: { workspaceRestoreRecovery: {
schema: LEGACY_WORKSPACE_RECOVERY_SCHEMA, runId: run.id, originalLeasePolicy: lease.leasePolicy,
} })}::jsonb`,
}).where(and(
eq(environmentLeases.id, prepared.id), eq(environmentLeases.companyId, run.companyId),
eq(environmentLeases.heartbeatRunId, run.id), eq(environmentLeases.status, "pending_cleanup"),
));
} } });
if (!prepared) throw new Error("Workspace recovery lease ownership changed before terminal commit.");
retained.push(lease.id);
}
return retained;
+11 -4
View File
@@ -96,9 +96,13 @@ export function readStopOnlyCleanup(lease: Lease) {
* an explicit stop into ordinary destructive release. Invalid pins stay pending.
* `recovery` marks an ownership settle on the intent and on the lease
* (`ORPHANED_SANDBOX_RECOVERY_KEY`); the pending-cleanup sweep dispatches such
* a stop's first attempt without its retry backoff. */
export async function prepareSandboxStopAndRetain(db: Db, lease: Lease, recovery?: Record<string, unknown>) {
return prepareStopAndRetain(db, lease, "active", recovery);
* a stop's first attempt without its retry backoff. `retention` sets a repair
* hold's policy and metadata in the same write: a second write to the row in
* one transaction re-runs its foreign-key checks, which wait on any run that
* holds the shared environment row. */
export async function prepareSandboxStopAndRetain(db: Db, lease: Lease, recovery?: Record<string, unknown>,
retention?: { leasePolicy: string; metadata: Record<string, unknown> }) {
return prepareStopAndRetain(db, lease, "active", recovery, retention);
}
/** Historical release rows can lack a stop receipt. Re-inspect the exact
@@ -189,7 +193,8 @@ export async function prepareUnconfirmedSourceSandboxStop(db: Db, lease: typeof
return prepareStopAndRetain(db, lease, lease.status);
}
async function prepareStopAndRetain(db: Db, lease: Lease, fromStatus: string, recovery: Record<string, unknown> = {}) {
async function prepareStopAndRetain(db: Db, lease: Lease, fromStatus: string, recovery: Record<string, unknown> = {},
retention?: { leasePolicy: string; metadata: Record<string, unknown> }) {
const requestId = randomUUID(), now = new Date();
const intent = { schema: "paperclip.sandbox-stop-and-retain.v1", requestId,
companyId: lease.companyId, runId: lease.heartbeatRunId, leaseId: lease.id,
@@ -198,12 +203,14 @@ async function prepareStopAndRetain(db: Db, lease: Lease, fromStatus: string, re
? { builtinProvider: lease.provider } : { pluginId: lease.metadata?.pluginId }), ...recovery };
const [updated] = await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({
status: "pending_cleanup", cleanupStatus: "failed", failureReason: "sandbox_stop_pending", releasedAt: now, updatedAt: now,
...(retention ? { leasePolicy: retention.leasePolicy } : {}),
metadata: sql`(coalesce(${environmentLeases.metadata}, '{}'::jsonb) - 'remoteExecutionTermination' - 'sandboxStopAndRetainReceipt') || ${JSON.stringify({
[SANDBOX_STOP_AND_RETAIN_KEY]: intent, pendingCleanupAttemptId: requestId,
...(fromStatus === "active" && Object.keys(recovery).length
? { [ORPHANED_SANDBOX_RECOVERY_KEY]: { ...recovery, runId: lease.heartbeatRunId, recoveredAt: now.toISOString() } } : {}),
pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0,
pendingCleanupRetryAfterMs: 0, pendingCleanupRetryAttempts: 0, pendingCleanupRetryCapWarned: false,
...retention?.metadata,
})}::jsonb`,
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),
eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.status, fromStatus),