fix(environments): retry recording a started sandbox lease that loses a deadlock (#22392)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A sandbox run creates its sandbox with the provider first. Then
`attachSandboxCreation` records the started sandbox as the run's
`environment_leases` row in one transaction.
> - That transaction locks the environment row, the creation operation
row and the run row. Then the lease insert takes a foreign-key key-share
lock on the parent issue (and execution workspace).
> - Other writers lock the issue row first and then the run row. When
such a writer holds the issue and asks for the run, a lock cycle occurs.
PostgreSQL aborts the lease insert with `deadlock detected` (`40P01`).
> - The transaction rolls back whole. But the run failed during setup
with `Failed to acquire lease ... Failed query: insert into
"environment_leases" ...`, and the sandbox that the provider had already
started was discarded. The bootstrap retry then had to create a new
sandbox.
> - This pull request runs the `attachSandboxCreation` transaction
through the shared bounded `retryTransientLockContention` helper.
#22329, #21703 and #20971 use the same helper for the same kind of lock
cycle.
> - The benefit is that a short lock conflict no longer fails the run
setup or discards a started sandbox.

## Linked Issues or Issue Description

Refs #22329
Refs #21703
Refs #20971

**What happened?**

On staging, a sandbox run failed during setup with `EnvironmentRunError:
Failed to acquire lease for environment "..." (sandbox): Failed query:
insert into "environment_leases" ...`. The cause chain was
`EnvironmentProviderUnavailableError` ("a database query failed while
recording the lease") and then `PostgresError: deadlock detected`. The
provider had already started the sandbox (`sandboxState: started`).

**What did you expect?**

A lost lock race on the lease record must not fail the run. The run must
keep the sandbox that the provider started and continue to its first
turn.

**Steps to reproduce**

1. Start a sandbox creation operation for a running heartbeat run on an
issue.
2. In another transaction, lock the issue row `FOR UPDATE`.
3. Record the created sandbox lease
(`environmentService(db).acquireLease({ creationOperationId, ... })`).
It locks the run row and then waits for the issue key-share lock.
4. In the other transaction, lock the run row `FOR UPDATE`.
5. PostgreSQL aborts the lease insert with `40P01`.

## What Changed

- `server/src/services/environments.ts`: `attachSandboxCreation` now
runs its transaction through `retryTransientLockContention` (5 attempts,
100 ms × attempt backoff, only for `40P01`, `40001` and `55P03`). A
fresh attempt re-reads the pending creation operation and the run row,
and re-proves every check. A doc comment records the lock order and the
rule that `attach` writes only through `tx`.
-
`server/src/services/environment-lease-attach-deadlock.integration.test.ts`:
new embedded-Postgres regression that creates the real issue-then-run
lock cycle, and a negative case that a deterministic failure is not
retried and leaves the operation `pending`.

## Verification

- Red before the fix: `npx vitest run
src/services/environment-lease-attach-deadlock.integration.test.ts`
failed with `PostgresError: deadlock detected` (`routine:
DeadLockReport`) on the same `insert into "environment_leases"`
statement as the staging event.
- Green after the fix: the same file passes (2/2). The lease row is
recorded once, `active`, with the provider lease id, and the creation
operation is `attached` to it.
- Related suites pass:
`src/__tests__/sandbox-creation-operations.test.ts`,
`src/__tests__/environment-task-authority.integration.test.ts`,
`src/__tests__/plugin-warm-tasks.integration.test.ts`,
`src/services/environment-provider-unavailable-error.test.ts` (158
tests), and `src/__tests__/environment-runtime.test.ts` (214 tests).
- Not run locally: the server `tsc` typecheck. The sandbox memory limit
killed it (exit 137). CI typecheck covers this revision.

## Risks

- Low risk. Only a transaction that PostgreSQL already rolled back whole
is retried, and the number of attempts is bounded.
- Both callers of `attachSandboxCreation` pass the root database handle,
and both `attach` callbacks write only through `tx`. If a future caller
passes an open transaction, a retry runs inside it as a savepoint. That
retry stays bounded and then throws the original error.
- No change to the error classification of other lease acquisition
failures.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, embedded-Postgres test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-20 04:53:42 -08:00
1 parent 48025caaca
commit 49adbbe2f6
2 files changed
+114 -2

No files matched your search

@@ -0,0 +1,101 @@
import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns, issues,
sandboxCreationOperations } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { attachSandboxCreation, environmentService } from "./environments.js";
// Staging: a sandbox run failed during setup with "Failed to acquire lease ...
// Failed query: insert into environment_leases ..." caused by 40P01. The
// provider had already started the sandbox. Recording it locks the environment,
// the creation operation and the run row, then the lease insert takes a
// key-share lock on its parent issue. A writer that holds the issue row and then
// asks for the run row (the usual issue-before-run order) closes a lock cycle,
// and PostgreSQL aborts the lease insert. The aborted transaction rolled back
// whole, yet it failed the run and discarded the started sandbox.
describe("recording a created sandbox lease under an issue-then-run lock cycle", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
const companyId = randomUUID();
const agentId = randomUUID();
const environmentId = randomUUID();
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("lease-attach-deadlock-");
db = createDb(temporary.connectionString);
await db.insert(companies).values({ id: companyId, name: "Lease deadlock", issuePrefix: "LAD" });
await db.insert(agents).values({ id: agentId, companyId, name: "Sandbox agent", role: "engineer", adapterType: "paperclip_runner" });
await db.insert(environments).values({ id: environmentId, name: "Sandbox", driver: "sandbox",
config: { provider: "creation-fixture", image: "fixture", reuseLease: false } });
}, 90_000);
afterAll(async () => { await temporary?.cleanup(); });
it("retries a lease insert that lost a deadlock and records the started sandbox once", async () => {
const issueId = randomUUID();
const runId = randomUUID();
const operationId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Fix the bug", status: "in_progress", workMode: "standard",
assigneeAgentId: agentId, createdByUserId: "board-user", responsibleUserId: "board-user" });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", invocationSource: "assignment",
contextSnapshot: { issueId } });
await db.insert(sandboxCreationOperations).values({ id: operationId, companyId, environmentId, originalEnvironmentId: environmentId,
runId, issueId, provider: "creation-fixture", driverKey: "creation-fixture", state: "pending", config: {}, metadata: {},
nextAttemptAt: new Date(Date.now() + 60_000) });
// Another writer holds the issue row before the lease is recorded. Once the
// recording holds the run row and waits for the issue (the lease foreign
// key), the writer asks for the run row: a deadlock. The writer's own
// detector waits longer, so PostgreSQL always aborts the lease recording.
let writerBlockedAttach = false;
let releaseWriter!: () => void;
const writerHoldsIssue = new Promise<void>(resolve => { releaseWriter = resolve; });
const writer = db.transaction(async tx => {
await tx.execute(sql`set local deadlock_timeout = '10s'`);
await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, issueId)).for("update");
const writerPid = (await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`))[0]!.pid;
releaseWriter();
for (let poll = 0; poll < 1_000 && !writerBlockedAttach; poll++) {
const rows = (await db.execute(
sql`select 1 from pg_stat_activity where ${writerPid} = any(pg_blocking_pids(pid))`,
)) as unknown as Array<unknown>;
if (rows[0]) writerBlockedAttach = true;
else await new Promise(resolve => setTimeout(resolve, 10));
}
await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
});
await writerHoldsIssue;
const outcome = await environmentService(db).acquireLease({
creationOperationId: operationId, companyId, environmentId, issueId, heartbeatRunId: runId,
leasePolicy: "ephemeral", provider: "creation-fixture", providerLeaseId: "provider-sandbox-1",
metadata: { agentId, driver: "sandbox" },
}).then(lease => ({ lease }), (error: unknown) => ({ error }));
await writer;
expect(writerBlockedAttach).toBe(true);
expect(outcome).not.toHaveProperty("error");
const leases = await db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, runId));
expect(leases).toEqual([expect.objectContaining({ status: "active", providerLeaseId: "provider-sandbox-1", issueId })]);
expect((outcome as { lease: { id: string } }).lease.id).toBe(leases[0]!.id);
const [operation] = await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id, operationId));
expect(operation).toMatchObject({ state: "attached", leaseId: leases[0]!.id });
}, 120_000);
it("does not retry a deterministic failure while recording the lease", async () => {
const runId = randomUUID();
const operationId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", invocationSource: "assignment" });
await db.insert(sandboxCreationOperations).values({ id: operationId, companyId, environmentId, originalEnvironmentId: environmentId,
runId, provider: "creation-fixture", driverKey: "creation-fixture", state: "pending", config: {}, metadata: {},
nextAttemptAt: new Date(Date.now() + 60_000) });
let attempts = 0;
await expect(attachSandboxCreation(db, operationId, { companyId, environmentId, heartbeatRunId: runId }, async () => {
attempts += 1;
throw new Error("deterministic attach failure");
})).rejects.toThrow("deterministic attach failure");
expect(attempts).toBe(1);
const [operation] = await db.select().from(sandboxCreationOperations).where(eq(sandboxCreationOperations.id, operationId));
expect(operation).toMatchObject({ state: "pending", leaseId: null });
}, 60_000);
});
+13 -2
View File
@@ -8,6 +8,7 @@ import { notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js
import { notifyDeferredExecutionWork } from "./deferred-execution-notifications.js";
import { withoutCleanupVerification } from "./cleanup-verification-proof.js";
import { withEnvironmentLeaseController } from "./environment-lease-controller.js";
import { retryTransientLockContention } from "./transient-lock-contention.js";
import { randomUUID } from "node:crypto";
import { SANDBOX_PLATFORM_PROBE_METADATA_KEY } from "./environment-custom-image-runtime.js";
@@ -66,9 +67,19 @@ import {
* all use this one list, or a delete can orphan a sandbox. */
export const KEPT_SANDBOX_LEASE_POLICIES = ["reuse_by_environment", "retain_on_failure"] as const;
/**
* Record a started sandbox as its creation operation's lease. The provider has
* already created the sandbox, so losing a lock race here must not fail the
* run. This transaction locks the run row, then the lease insert takes a
* key-share lock on its parent issue and workspace; a writer holding one of
* those and then waiting for the run closes a lock cycle and PostgreSQL aborts
* this transaction. The rollback is whole (the operation stays `pending` and
* no lease row exists), so a bounded fresh attempt re-proves every check.
* `attach` must therefore only write through `tx`.
*/
export async function attachSandboxCreation(db: Db, id: string, scope: { companyId: string; environmentId: string; heartbeatRunId?: string | null },
attach: (tx: Db) => Promise<EnvironmentLease>): Promise<EnvironmentLease> {
return db.transaction(async tx => {
return retryTransientLockContention(() => db.transaction(async tx => {
await assertEnvironmentCompanyBinding(tx as unknown as Db, scope.environmentId, scope.companyId);
const [operation] = await tx.select().from(sandboxCreationOperations)
.where(and(eq(sandboxCreationOperations.id, id), eq(sandboxCreationOperations.companyId, scope.companyId),
@@ -89,7 +100,7 @@ export async function attachSandboxCreation(db: Db, id: string, scope: { company
await tx.update(sandboxCreationOperations).set({ state: "attached", leaseId: lease.id, updatedAt: new Date() })
.where(eq(sandboxCreationOperations.id, id));
return lease;
});
}));
}
export async function assertEnvironmentCompanyBinding(db: Db, environmentId: string, companyId: string): Promise<void> {