fix(native): retry a workspace failure write that loses a deadlock (#21507)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime finalizes each agent run after the provider turn
ends. It copies the workspace back and records the result.
> - When the workspace copy-back fails, `recordRetryableFailure` records
the failure and a bounded export retry in one transaction.
> - That transaction locks the coordinator, issue and run rows. Then the
recovery-action insert takes `FOR KEY SHARE` on the agent row for its
foreign keys.
> - A different transaction that holds the agent row and waits on the
run row deadlocks with it. Postgres aborts the failure write with 40P01.
> - The run then fails with `Database query failed (40P01)`, and the
export retry is never recorded.
> - This pull request runs the whole failure write through the existing
`retryTransientLockContention` helper.
> - The benefit is that a lost deadlock no longer fails the run or drops
its workspace export retry.

## Linked Issues or Issue Description

Refs #21494 (same mechanism, but for the provider-turn failure write in
`native-session-executor.ts`. This PR covers `native-run-finalizer.ts`.)
Refs #21506 (a different agent-row deadlock in run setup. It does not
change this write.)

**What happened?**
On a hosted staging server, two runs finished their provider turn, and
then the workspace copy-back failed. The server tried to record that
failure and lost a Postgres deadlock on `insert into
issue_recovery_actions`. The stack was `recordNativeFinalizationFailure`
-> `recordRetryableFailure` -> `upsertSourceScoped`. Each run failed
with `Database query failed (40P01)`.

**Expected behavior**
A lost deadlock in the failure write is retried. The run records
`retryable_failure` and an active `active_run_watchdog` recovery action
for the bounded workspace export retry.

**Steps to reproduce**
1. Hold `FOR UPDATE` on the agent row in one transaction.
2. Start `recordRetryableFailure` for a run of that agent. It locks the
run and issue rows, then waits on the agent row.
3. In the first transaction, request a lock on the run row. Postgres
detects the deadlock and aborts one side with 40P01.

The new integration test does these steps with embedded Postgres.

**Paperclip version or commit**
`master` at 0ebe6d846d.

## What Changed

- `recordRetryableFailure` now wraps its transaction in
`retryTransientLockContention`. That helper retries only 40P01, 40001
and 55P03, with at most 5 attempts.
- Each attempt resets its local state (`input`, `terminalRunToEmit`).
Postgres rolls back the aborted attempt completely, and the next attempt
re-reads all state under the same lock order.
- Post-commit reporting (`emitAgentTaskRun`, `reportRunFailure`) runs
one time, after the attempt that commits.
- Added `native-workspace-failure-write-deadlock.integration.test.ts`.
It makes a real deadlock against the failure write and checks the
committed coordinator, recovery action and run rows.

## Verification

- Red before the fix: `pnpm exec vitest run
src/services/native-runtime/native-workspace-failure-write-deadlock.integration.test.ts`
failed with 40P01 while locking a tuple in `agents`. This is the same
error and stack as the staging event.
- Green after the fix: the same test passed 4 times in a row.
- 7 related finalizer and export test files (288 tests) pass.
- `pnpm typecheck` (server) passes.

## Risks

- Low risk. The change only adds a retry around an existing transaction.
The retry is bounded and only applies to lock-contention error codes.
- A retried attempt re-reads its state. If a different owner committed a
result in the meantime, the existing admission checks reject the stale
write as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-11-20 04:42:41 -08:00
1 parent b5c53a996b
commit 9874608d0a
2 files changed
+67 -2

No files matched your search

@@ -81,6 +81,7 @@ import {
type NativeStatusCommitFailpoint,
} from "./status-decision-committer.js";
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
import { retryTransientLockContention } from "../transient-lock-contention.js";
import { preservedResultHasStop, readPreservedNativeResultFinalization } from "./preserved-native-result-finalization.js";
import { issueService } from "../issues.js";
import { publishChatPublicationCommitSignal } from "../chat-publication-reconciliation.js";
@@ -483,7 +484,10 @@ async function recordRetryableFailure(input: {
? ("cancelled" as const)
: ("failed" as const);
let terminalRunToEmit: typeof heartbeatRuns.$inferSelect | null = null;
const outcome = await input.db.transaction(async (tx) => {
const admitted = input;
const outcome = await retryTransientLockContention(() => input.db.transaction(async (tx) => {
input = admitted;
terminalRunToEmit = null;
// Admission snapshots cannot authorize a failure write after a different
// owner committed success. Match status-committer lock ordering.
const current = await tx.select().from(nativeRunFinalizations).where(and(
@@ -795,7 +799,7 @@ async function recordRetryableFailure(input: {
failureCode,
nextAttemptAt: supersededByNewerRun || exhausted ? null : nextAttemptAt,
};
});
}));
if (terminalRunToEmit) {
await emitAgentTaskRun(input.db, terminalRunToEmit);
void reportRunFailure(input.db, terminalRunToEmit);
@@ -0,0 +1,61 @@
import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { agents, companies, createDb, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { recordNativeFinalizationFailure } from "./native-run-finalizer.js";
describe("native workspace failure write under a deadlock", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
const companyId = randomUUID();
const agentId = randomUUID();
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("native-workspace-failure-deadlock-");
db = createDb(temporary.connectionString);
await db.insert(companies).values({ id: companyId, name: "Failure deadlock", issuePrefix: "NWD" });
await db.insert(agents).values({ id: agentId, companyId, name: "Exporter", adapterType: "paperclip_runner" });
}, 60_000);
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
it("records the retryable export failure after losing a deadlock", async () => {
const issueId = randomUUID();
const runId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Export saved work", status: "in_progress", assigneeAgentId: agentId });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: issueId });
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "result_accepted" });
let agentLocked!: () => void;
const agentLock = new Promise<void>(resolve => { agentLocked = resolve; });
const holder = db.transaction(async tx => {
await tx.execute(sql`set local deadlock_timeout = '10s'`);
await tx.select({ id: agents.id }).from(agents).where(eq(agents.id, agentId)).for("update");
agentLocked();
for (let waited = 0; ; waited += 25) {
const blocked = await db.execute<{ pid: number }>(sql`select pid from pg_stat_activity
where wait_event_type = 'Lock' and query like 'insert into "issue_recovery_actions"%'`);
if (blocked.length) break;
if (waited > 30_000) throw new Error("failure write never waited on the agent row");
await new Promise(resolve => setTimeout(resolve, 25));
}
await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
});
await Promise.race([agentLock, holder]);
const failure = await recordNativeFinalizationFailure({ db, runId, error: new Error("native_workspace_sync_out_failed"),
failureScope: "workspace", projectRunStatus: true });
await holder;
expect(failure).toMatchObject({ phase: "retryable_failure", failureCode: "native_workspace_sync_out_failed" });
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
expect(coordinator).toMatchObject({ phase: "retryable_failure", failureCode: "native_workspace_sync_out_failed",
failureDetail: expect.objectContaining({ workspaceFinalizeAttempt: 1 }) });
const actions = await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId)));
expect(actions).toHaveLength(1);
expect(actions[0]).toMatchObject({ status: "active", kind: "active_run_watchdog", cause: "native_workspace_sync_out_failed",
ownerAgentId: agentId, wakePolicy: expect.objectContaining({ kind: "resume_native_run", runId }) });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(run).toMatchObject({ status: "running", nativePhase: "retryable_failure" });
}, 120_000);
});