From 9874608d0aa66e090ce8ad2b93401fdcebe7b37f Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 20 Nov 2026 04:42:41 -0800 Subject: [PATCH] fix(native): retry a workspace failure write that loses a deadlock (#21507) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime finalizes each agent run after the provider turn ends. It copies the workspace back and records the result. > - When the workspace copy-back fails, `recordRetryableFailure` records the failure and a bounded export retry in one transaction. > - That transaction locks the coordinator, issue and run rows. Then the recovery-action insert takes `FOR KEY SHARE` on the agent row for its foreign keys. > - A different transaction that holds the agent row and waits on the run row deadlocks with it. Postgres aborts the failure write with 40P01. > - The run then fails with `Database query failed (40P01)`, and the export retry is never recorded. > - This pull request runs the whole failure write through the existing `retryTransientLockContention` helper. > - The benefit is that a lost deadlock no longer fails the run or drops its workspace export retry. ## Linked Issues or Issue Description Refs #21494 (same mechanism, but for the provider-turn failure write in `native-session-executor.ts`. This PR covers `native-run-finalizer.ts`.) Refs #21506 (a different agent-row deadlock in run setup. It does not change this write.) **What happened?** On a hosted staging server, two runs finished their provider turn, and then the workspace copy-back failed. The server tried to record that failure and lost a Postgres deadlock on `insert into issue_recovery_actions`. The stack was `recordNativeFinalizationFailure` -> `recordRetryableFailure` -> `upsertSourceScoped`. Each run failed with `Database query failed (40P01)`. **Expected behavior** A lost deadlock in the failure write is retried. The run records `retryable_failure` and an active `active_run_watchdog` recovery action for the bounded workspace export retry. **Steps to reproduce** 1. Hold `FOR UPDATE` on the agent row in one transaction. 2. Start `recordRetryableFailure` for a run of that agent. It locks the run and issue rows, then waits on the agent row. 3. In the first transaction, request a lock on the run row. Postgres detects the deadlock and aborts one side with 40P01. The new integration test does these steps with embedded Postgres. **Paperclip version or commit** `master` at 0ebe6d846d. ## What Changed - `recordRetryableFailure` now wraps its transaction in `retryTransientLockContention`. That helper retries only 40P01, 40001 and 55P03, with at most 5 attempts. - Each attempt resets its local state (`input`, `terminalRunToEmit`). Postgres rolls back the aborted attempt completely, and the next attempt re-reads all state under the same lock order. - Post-commit reporting (`emitAgentTaskRun`, `reportRunFailure`) runs one time, after the attempt that commits. - Added `native-workspace-failure-write-deadlock.integration.test.ts`. It makes a real deadlock against the failure write and checks the committed coordinator, recovery action and run rows. ## Verification - Red before the fix: `pnpm exec vitest run src/services/native-runtime/native-workspace-failure-write-deadlock.integration.test.ts` failed with 40P01 while locking a tuple in `agents`. This is the same error and stack as the staging event. - Green after the fix: the same test passed 4 times in a row. - 7 related finalizer and export test files (288 tests) pass. - `pnpm typecheck` (server) passes. ## Risks - Low risk. The change only adds a retry around an existing transaction. The retry is bounded and only applies to lock-contention error codes. - A retried attempt re-reads its state. If a different owner committed a result in the meantime, the existing admission checks reject the stale write as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> --- .../native-runtime/native-run-finalizer.ts | 8 ++- ...failure-write-deadlock.integration.test.ts | 61 +++++++++++++++++++ 2 files changed, 67 insertions(+), 2 deletions(-) create mode 100644 server/src/services/native-runtime/native-workspace-failure-write-deadlock.integration.test.ts diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index b6305dea131..83b50a9bde2 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -81,6 +81,7 @@ import { type NativeStatusCommitFailpoint, } from "./status-decision-committer.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; +import { retryTransientLockContention } from "../transient-lock-contention.js"; import { preservedResultHasStop, readPreservedNativeResultFinalization } from "./preserved-native-result-finalization.js"; import { issueService } from "../issues.js"; import { publishChatPublicationCommitSignal } from "../chat-publication-reconciliation.js"; @@ -483,7 +484,10 @@ async function recordRetryableFailure(input: { ? ("cancelled" as const) : ("failed" as const); let terminalRunToEmit: typeof heartbeatRuns.$inferSelect | null = null; - const outcome = await input.db.transaction(async (tx) => { + const admitted = input; + const outcome = await retryTransientLockContention(() => input.db.transaction(async (tx) => { + input = admitted; + terminalRunToEmit = null; // Admission snapshots cannot authorize a failure write after a different // owner committed success. Match status-committer lock ordering. const current = await tx.select().from(nativeRunFinalizations).where(and( @@ -795,7 +799,7 @@ async function recordRetryableFailure(input: { failureCode, nextAttemptAt: supersededByNewerRun || exhausted ? null : nextAttemptAt, }; - }); + })); if (terminalRunToEmit) { await emitAgentTaskRun(input.db, terminalRunToEmit); void reportRunFailure(input.db, terminalRunToEmit); diff --git a/server/src/services/native-runtime/native-workspace-failure-write-deadlock.integration.test.ts b/server/src/services/native-runtime/native-workspace-failure-write-deadlock.integration.test.ts new file mode 100644 index 00000000000..f4c51ee6ded --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-failure-write-deadlock.integration.test.ts @@ -0,0 +1,61 @@ +import { randomUUID } from "node:crypto"; +import { and, eq, sql } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, companies, createDb, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { recordNativeFinalizationFailure } from "./native-run-finalizer.js"; + +describe("native workspace failure write under a deadlock", () => { + let temporary: Awaited>; + let db: ReturnType; + const companyId = randomUUID(); + const agentId = randomUUID(); + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("native-workspace-failure-deadlock-"); + db = createDb(temporary.connectionString); + await db.insert(companies).values({ id: companyId, name: "Failure deadlock", issuePrefix: "NWD" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Exporter", adapterType: "paperclip_runner" }); + }, 60_000); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + + it("records the retryable export failure after losing a deadlock", async () => { + const issueId = randomUUID(); + const runId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Export saved work", status: "in_progress", assigneeAgentId: agentId }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: issueId }); + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "result_accepted" }); + + let agentLocked!: () => void; + const agentLock = new Promise(resolve => { agentLocked = resolve; }); + const holder = db.transaction(async tx => { + await tx.execute(sql`set local deadlock_timeout = '10s'`); + await tx.select({ id: agents.id }).from(agents).where(eq(agents.id, agentId)).for("update"); + agentLocked(); + for (let waited = 0; ; waited += 25) { + const blocked = await db.execute<{ pid: number }>(sql`select pid from pg_stat_activity + where wait_event_type = 'Lock' and query like 'insert into "issue_recovery_actions"%'`); + if (blocked.length) break; + if (waited > 30_000) throw new Error("failure write never waited on the agent row"); + await new Promise(resolve => setTimeout(resolve, 25)); + } + await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update"); + }); + await Promise.race([agentLock, holder]); + const failure = await recordNativeFinalizationFailure({ db, runId, error: new Error("native_workspace_sync_out_failed"), + failureScope: "workspace", projectRunStatus: true }); + await holder; + + expect(failure).toMatchObject({ phase: "retryable_failure", failureCode: "native_workspace_sync_out_failed" }); + const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)); + expect(coordinator).toMatchObject({ phase: "retryable_failure", failureCode: "native_workspace_sync_out_failed", + failureDetail: expect.objectContaining({ workspaceFinalizeAttempt: 1 }) }); + const actions = await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId))); + expect(actions).toHaveLength(1); + expect(actions[0]).toMatchObject({ status: "active", kind: "active_run_watchdog", cause: "native_workspace_sync_out_failed", + ownerAgentId: agentId, wakePolicy: expect.objectContaining({ kind: "resume_native_run", runId }) }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + expect(run).toMatchObject({ status: "running", nativePhase: "retryable_failure" }); + }, 120_000); +});