fix(native): retry a coordinator claim that loses a deadlock instead of failing the run (#21703)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native (Paperclip Runner) run claims its
`native_run_finalizations` coordinator before it starts any provider
work. The claim is one transaction. It locks the coordinator row, then
the run row.
> - A writer that holds the run row and then waits for the coordinator
row closes a lock cycle. PostgreSQL then aborts the claim with a
deadlock (`40P01`), because the claim waited first.
> - The aborted claim rolls back whole and starts no provider work. But
the raw database error failed the run before its first turn.
> - The failure path then preserved the prepared workspace. That stamped
the unclaimed coordinator `native_continuation_requires_reconciliation`,
and `failRun` used it as the run's error code. The task waited for
reconciliation.
> - A transient database fault must never end a run like this. The
target rate of runs that need Retry is 0%.
> - This pull request retries the claim transaction on transient lock
contention with the shared bounded helper.
> - The benefit is that a fresh run that loses a lock race at its claim
continues to its provider turn by itself.

## Linked Issues or Issue Description

**What happened?**
On a hosted QA stack, a fresh native assignment run (not a retry) failed
after about 50 s. The run status was `failed`. The error code was
`native_continuation_requires_reconciliation`. The error text was
`Database query failed (40P01)`. The stderr was empty. The task needed
reconciliation.

**Steps to reproduce**
1. Prepare a native run with an unclaimed coordinator (`observed`,
attempt 0), as runtime selection leaves it.
2. In another transaction, lock the run row.
3. Start the native session. Its claim locks the coordinator and waits
for the run row.
4. In the other transaction, lock the coordinator row.
5. PostgreSQL aborts the claim with `40P01`, and
`executePaperclipNativeSession` rejects with that error.

**Expected behavior**
The claim retries after the competing transaction commits. The run
reaches its provider turn.

**Paperclip version or commit**
`master` at 958eda6f16.

## What Changed

- `native-session-executor.ts`: the coordinator claim transaction runs
inside `retryTransientLockContention` (`40P01`, `40001`, `55P03`; 5
attempts, 100 ms × attempt backoff). Each attempt re-reads and re-proves
the coordinator, the run, cancellation and lineage, so a retry admits
nothing that the first attempt would refuse.
- A startup context that a rolled-back attempt opened is finished, and
the startup owner's previous context is restored, before the next
attempt.
- New PostgreSQL test
`native-coordinator-claim-deadlock.integration.test.ts`: `retries a
claim that lost a deadlock and reaches the provider`.

## Verification

- RED on `master` (958eda6f16): the new test fails.
`executePaperclipNativeSession` rejects with a `40P01` cause, and the
provider is not called.
- GREEN with this change: `pnpm --filter @paperclipai/server exec vitest
run
src/services/native-runtime/native-coordinator-claim-deadlock.integration.test.ts`
passes. The provider is called once and the coordinator is at attempt 1.
- `tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` for the affected files: see the PR comments.

## Risks

- Low risk. The normal path has no contention and does not retry.
- Under contention, the claim waits at most about 1 s more (backoff
100–400 ms) before provider work. This is before `turn.start`, so time
to first token grows only in the contended case.
- The lock order (coordinator, then run) is unchanged. It matches
admission and the runner checkpoint owner lock. The writer that took the
run row first on the QA stack is not identified, because no server logs
or SQL were available. This change makes the claim survive any such
writer.
- Out of scope: a transient fault in other pre-dispatch statements of a
run with a prepared remote workspace. There, workspace preservation
still stamps the pre-claim code before `failRun`. The pre-claim
fresh-continuation fallback then continues the task.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, embedded PostgreSQL tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20561 (pre-dispatch transient retry), #20682 (provider-pack
admission lock wait).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 20:23:33 -06:00
1 parent c38a471042
commit adc3356ad3
2 files changed
+121 -2

No files matched your search

@@ -0,0 +1,105 @@
import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, companies, companyMemberships, completionContracts, createDb, heartbeatRuns, issues,
nativeRunFinalizations } from "@paperclipai/db";
import type { NativeExecutionInputV1 } from "@paperclipai/paperclip-runner";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()),
executeNativeSession: provider.execute,
}));
import { executePaperclipNativeSession } from "./native-session-executor.js";
import { buildNativeCompletionContract } from "./completion-contracts.js";
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
// QA Cloud: a fresh assignment run failed before its first provider turn with
// "Database query failed (40P01)" and was left as a task that needed
// reconciliation. The coordinator claim locks the coordinator row, then the
// run row. A writer that holds the run row and then asks for the coordinator
// closes a lock cycle; PostgreSQL aborts the claim, the waiter that checks
// first. The aborted claim rolled back whole, yet it failed the run.
describe("native coordinator claim under a run-then-coordinator lock cycle", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
const companyId = randomUUID();
const agentId = randomUUID();
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("native-claim-deadlock-");
db = createDb(temporary.connectionString);
await db.insert(companies).values({ id: companyId, name: "Claim deadlock", issuePrefix: "NCD" });
await db.insert(agents).values({ id: agentId, companyId, name: "OpenCode agent", status: "active", adapterType: "paperclip_runner",
adapterConfig: { provider: "opencode", model: "openrouter/test-model" } });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "board-user", status: "active", membershipRole: "member" });
});
afterAll(async () => { await temporary?.cleanup(); });
it("retries a claim that lost a deadlock and reaches the provider", async () => {
const issueId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Write the brief", status: "todo", workMode: "standard",
assigneeAgentId: agentId, createdByUserId: "board-user", responsibleUserId: "board-user" });
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
const runId = randomUUID();
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native",
nativeIssueId: issueId, responsibleUserId: "board-user", invocationSource: "assignment", triggerDetail: "system",
contextSnapshot: { issueId } }).returning();
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
// Runtime selection leaves the coordinator unclaimed: observed, attempt 0.
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
const execution: NativeExecutionInputV1 = {
schema: "paperclip.native-execution-input.v1",
binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
provider: { kind: "opencode", model: "openrouter/test-model" },
task: { identifier: issueId, title: issue.title, description: null, prompt: "Write the brief.", workMode: "standard" },
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "opencode_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
contract: buildNativeCompletionContract(issue, { revision: contract.revision }) },
interactionResponses: [], credentialBindings: [],
};
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
// Another writer holds the run row (as a run update does) before the claim
// starts. Once the claim holds the coordinator and waits for the run row,
// the writer asks for the coordinator: a deadlock. The writer's own
// detector waits longer, so PostgreSQL always aborts the claim.
let writerBlockedClaim = false;
let releaseWriter!: () => void;
const writerHoldsRun = new Promise<void>(resolve => { releaseWriter = resolve; });
const writer = db.transaction(async tx => {
await tx.execute(sql`set local deadlock_timeout = '10s'`);
await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("no key update");
const writerPid = (await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`))[0]!.pid;
releaseWriter();
for (let poll = 0; poll < 1_000 && !writerBlockedClaim; poll++) {
const rows = (await db.execute(
sql`select 1 from pg_stat_activity where ${writerPid} = any(pg_blocking_pids(pid))`,
)) as unknown as Array<unknown>;
if (rows[0]) writerBlockedClaim = true;
else await new Promise(resolve => setTimeout(resolve, 10));
}
await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, runId)).for("update");
});
await writerHoldsRun;
provider.execute.mockReset().mockRejectedValue(new Error("provider_reached_after_claim"));
const outcome = await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: {
descriptor: async () => { throw new Error("Unexpected provider call"); },
openSession: async () => { throw new Error("Unexpected provider call"); },
} }).then(() => null, (error: unknown) => error);
await writer;
expect(writerBlockedClaim).toBe(true);
expect((outcome as { cause?: { code?: string } } | null)?.cause?.code).not.toBe("40P01");
expect(provider.execute).toHaveBeenCalledTimes(1);
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
expect(coordinator.attempt).toBe(1);
}, 120_000);
});
@@ -11131,11 +11131,25 @@ async function executePaperclipNativeSessionAttempt(
let claimedStartupContext: NativeExecutionContext | undefined;
let freshClaimAttempt: number | undefined;
let scheduledResumeContinuation = false;
const startupContextBeforeClaim = nativeSessionStartups.get(input.execution.binding.runId)?.executionContext;
try {
attempt = await trace.measure(
"native.coordinator.claim",
() =>
input.db.transaction(async (tx) => {
// The claim locks the coordinator, then the run row. A writer holding
// those rows in the other order can deadlock it (40P01); a lock wait
// can also time out (55P03). Either rolls the claim back whole and it
// starts no provider work, so a fresh attempt re-reads and re-proves
// everything. Without it the run failed before its first turn and the
// task needed reconciliation.
retryTransientLockContention(() => input.db.transaction(async (tx) => {
// A rolled-back attempt may have opened a startup context; drop it.
if (claimedStartupContext) {
claimedStartupContext.finish();
claimedStartupContext = undefined;
const startup = nativeSessionStartups.get(input.execution.binding.runId);
if (startup) startup.executionContext = startupContextBeforeClaim;
}
const coordinator = await tx
.select()
.from(nativeRunFinalizations)
@@ -11369,7 +11383,7 @@ async function executePaperclipNativeSessionAttempt(
})
.where(eq(heartbeatRuns.id, coordinator.runId));
return nextAttempt;
}).then(async claimedAttempt => {
})).then(async claimedAttempt => {
const startup = nativeSessionStartups.get(input.execution.binding.runId);
if (!startup) throw new Error("native_session_startup_owner_missing");
startup.leaseOwner = leaseOwner;