fix(native): retry a contended provider-pack admission write (#20850)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native OpenCode run on a remote sandbox must save a provider-pack
admission receipt on its run row after the provider session opens.
Artifact retention refuses to continue without it.
> - The admission write locks the coordinator row
(`native_run_finalizations`) and then the run row, each with a 1000 ms
`lock_timeout`. The runner's durable checkpoint writes lock the same
coordinator row for the whole checkpoint transaction.
> - On a loaded database one checkpoint transaction can outlast one lock
wait. The admission write then fails with `55P03`, the state stays
`pending`, and artifact retention fails the run as
`native_provider_artifacts_admission_unverified` before the first turn.
The task rests blocked.
> - #20682 fixed the same symptom for a run-event append on the run row.
This is the coordinator-row half.
> - This pull request retries a contended admission write a bounded
number of times, each in a new transaction that re-proves the claim. It
also names the database cause in the run log and adds a chaos option
that makes every fresh sandbox take the pack upload and admission path.
> - The benefit is that a healthy fresh sandbox is no longer failed and
blocked by ordinary lock contention.

## Linked Issues or Issue Description

Refs #20682 (same symptom, run-row lock; this is the coordinator-row
lock).

**What happened?**
On a hosted staging instance, an OpenCode run on a fresh Daytona sandbox
found no matching preinstalled provider pack (`provider_pack.select`
reason `cached_packs_unverified`). It uploaded and verified the pack (61
s), started the runner, and the runner authenticated to the durable
control plane. 3.4 s later the run log shows `native provider-pack
admission receipt could not be saved`.
`nativeProviderPackAdmissionState.state` stayed `pending`. The run
failed with `native_provider_artifacts_admission_unverified` before any
provider turn. The task rests `blocked` with no Retry.

**Steps to reproduce**
1. Start a native run that waits for its claim (`waitForClaim: true`, as
a fresh run with artifact storage does).
2. Hold its `native_run_finalizations` row `FOR UPDATE` in another
transaction for 1.6 s, as a checkpoint write does.
3. Call `recordNativeProviderPackAdmission`. It fails with `canceling
statement due to lock timeout` (`55P03`).

**Expected behavior**
The admission waits out the checkpoint write within a bound, re-proves
the claim, and saves the receipt.

## What Changed

- `recordNativeProviderPackAdmission` retries a write that waits for its
claim when it fails with `55P03`, `40P01` or `40001`, up to 5 attempts
with 100–400 ms back-off. Each attempt is a new transaction with the
same checks (claim, run, epoch, receipt). A stale claim still returns
`false`. Other errors still throw. A write that does not wait for its
claim keeps its `NOWAIT` refusal.
- New `nativeProviderPackAdmissionFailureCode` reads the SQLSTATE from
the error or its cause.
- The executor's run-log line now ends with `(cause=<SQLSTATE or
native_* code>)`, so a lost admission is diagnosable. It carries no
paths or secrets.
- Restart chaos: `--daytona-pack-mismatch <image@sha256:…>` starts
sandboxes from another image while the controller keeps
`--daytona-image`'s runnerd and pack. Every fresh sandbox then takes the
upload and admission path, which the matched-image check made
unreachable. Config validation and README updated.
- `doc/execution-semantics.md` describes the bounded retry.

## Verification

- RED on `origin/master` (`1d67e309e0`): `pnpm --filter
@paperclipai/server exec vitest run
src/services/native-runtime/native-provider-pack-admission.test.ts -t
"checkpoint write holds"` fails with `canceling statement due to lock
timeout` (`55P03`), the staging error.
- GREEN: the same file, 39/39 pass, including the existing fence tests
(`cannot publish while another transaction holds the coordinator claim`,
revoked-claim recheck) and the new bound test (`gives up after the
bounded retry window while the claim stays held`, under 15 s).
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/scheduler.test.ts`: 34/34 pass, including the new
option test.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` and `tsc
--noEmit -p tests/restart-chaos/tsconfig.json` pass.
- Not run: a Daytona chaos wave with `--daytona-pack-mismatch`. It needs
a runner image built at this revision. The staging cause is inferred
from the timing and the code path; the original error was swallowed,
which is why this PR adds the cause to the run log.

## Risks

- Low risk. The normal path has no contention and makes one attempt.
Under contention the write waits at most about 11 s before the first
provider turn. Without this change that case failed the run.
- A retry cannot widen authority: every attempt re-checks the
coordinator claim, lease expiry, run status and admission epoch.
- When admission still fails after the retries, the run fails as before.
This PR does not change that recovery path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/native-runtime/native-provider-pack-admission.test.ts

@@ -257,6 +257,50 @@ describe("provider pack admission PostgreSQL fencing", () => {
     expect((await read(f.identity.runId)).runnerProfileJson?.nativeProviderPackAdmissionState)
       .toEqual({ ...f.profile.nativeProviderPackAdmissionState, state: "verified" });
   });
+  it("keeps the admission when a checkpoint write holds the claim longer than one lock wait", async () => {
+    // Staging run 3f317172 (OpenCode, fresh Daytona sandbox): the pack upload
+    // and runner start succeeded, the runner began writing durable checkpoints,
+    // and the admission write lost one 1000 ms lock wait on the coordinator row
+    // a checkpoint transaction held. The state stayed pending, artifact
+    // retention failed the run as native_provider_artifacts_admission_unverified
+    // and the task rested blocked before its first turn.
+    const f = await fixture();
+    let release!: () => void, acquired!: () => void;
+    const holding = new Promise<void>(resolve => { acquired = resolve; });
+    const gate = new Promise<void>(resolve => { release = resolve; });
+    const checkpoint = db.transaction(async tx => {
+      await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.identity.runId)).for("update");
+      acquired(); await gate;
+    });
+    await holding;
+    const recording = recordNativeProviderPackAdmission(db, { ...f, waitForClaim: true } as Parameters<typeof recordNativeProviderPackAdmission>[1])
+      .then(value => ({ value }), error => ({ error }));
+    // Longer than one lock wait, shorter than the bounded retry window.
+    const held = setTimeout(release, 1_600);
+    try {
+      expect(await recording).toEqual({ value: true });
+    } finally { clearTimeout(held); release(); await checkpoint; }
+    expect((await read(f.identity.runId)).runnerProfileJson?.nativeProviderPackAdmissionState)
+      .toEqual({ ...f.profile.nativeProviderPackAdmissionState, state: "verified" });
+  });
+  it("gives up after the bounded retry window while the claim stays held", async () => {
+    const f = await fixture();
+    let release!: () => void, acquired!: () => void;
+    const holding = new Promise<void>(resolve => { acquired = resolve; });
+    const gate = new Promise<void>(resolve => { release = resolve; });
+    const checkpoint = db.transaction(async tx => {
+      await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.identity.runId)).for("update");
+      acquired(); await gate;
+    });
+    await holding;
+    const started = Date.now();
+    try {
+      await expect(recordNativeProviderPackAdmission(db, { ...f, waitForClaim: true } as Parameters<typeof recordNativeProviderPackAdmission>[1]))
+        .rejects.toMatchObject({ cause: { code: "55P03" } });
+      expect(Date.now() - started).toBeLessThan(15_000);
+    } finally { release(); await checkpoint; }
+    expect((await read(f.identity.runId)).runnerProfileJson).toEqual(f.profile);
+  }, 30_000);
   it.each([false, true])("waits within the durable-capture bound and rechecks the claim after its lock (revoked: %s)", async revoked => {
     const f = await fixture();
     let release!: () => void, acquired!: () => void;

server/src/services/native-runtime/native-provider-pack-admission.ts

@@ -123,13 +123,51 @@ export function admittedNativeProviderPack(session: NativeSession, identity: Nat
     ...(admittedRunId === actual.runId ? {} : { inheritedFromRunId: admittedRunId }) };
 }
 
-export async function recordNativeProviderPackAdmission(db: Db, input: {
+type AdmissionInput = {
   session: NativeSession;
   identity: NativeRunIdentity;
   admissionEpoch: string;
   claim: { leaseOwner: string; attempt: number; controllerGeneration: number; controllerBootId: string };
   waitForClaim?: boolean;
-}): Promise<boolean> {
+};
+
+/** Lock timeouts, deadlocks and serialization aborts that clear on their own. */
+const CONTENDED_ADMISSION_CODES = new Set(["55P03", "40P01", "40001"]);
+/** Attempts of one admission write while its claim is contended. */
+export const NATIVE_PROVIDER_PACK_ADMISSION_ATTEMPTS = 5;
+
+export function nativeProviderPackAdmissionFailureCode(error: unknown): string | null {
+  for (let value = error, depth = 0; value && typeof value === "object" && depth < 4; depth += 1) {
+    const code = (value as { code?: unknown }).code;
+    if (typeof code === "string" && /^[0-9A-Z]{5}$/.test(code)) return code;
+    value = (value as { cause?: unknown }).cause;
+  }
+  return null;
+}
+
+/**
+ * Saves the session's verified provider-pack admission on its run row. A run
+ * that waits for its claim retries a contended write: the runner's durable
+ * checkpoint writes hold the same coordinator row, and on a loaded database one
+ * can outlast a single lock wait. Each attempt is a new transaction that
+ * re-proves the claim, the run and the admission epoch, so a retry never
+ * admits anything the first attempt would have refused. Only contention is
+ * retried; a stale claim still returns false and other errors still throw.
+ */
+export async function recordNativeProviderPackAdmission(db: Db, input: AdmissionInput): Promise<boolean> {
+  for (let attempt = 1; ; attempt += 1) {
+    try {
+      return await recordNativeProviderPackAdmissionOnce(db, input);
+    } catch (error) {
+      const code = nativeProviderPackAdmissionFailureCode(error);
+      if (!input.waitForClaim || attempt >= NATIVE_PROVIDER_PACK_ADMISSION_ATTEMPTS
+        || !code || !CONTENDED_ADMISSION_CODES.has(code)) throw error;
+      await new Promise(resolve => setTimeout(resolve, 100 * attempt));
+    }
+  }
+}
+
+async function recordNativeProviderPackAdmissionOnce(db: Db, input: AdmissionInput): Promise<boolean> {
   const receipt = admittedNativeProviderPack(input.session, input.identity);
   return db.transaction(async tx => {
     // Bounds every lock wait below, including the run-row lock. A run-event

server/src/services/native-runtime/native-session-executor.ts

@@ -44,7 +44,7 @@ import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-
 import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
 import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
 import { providerPackWorkspaceId, recordAdmittedRemoteProviderPack, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js";
-import { nativeProviderPackAdmissionPending, rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack, recordNativeProviderPackAdmission, type NativeProviderPackAdmissionSource } from "./native-provider-pack-admission.js";
+import { nativeProviderPackAdmissionFailureCode, nativeProviderPackAdmissionPending, rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack, recordNativeProviderPackAdmission, type NativeProviderPackAdmissionSource } from "./native-provider-pack-admission.js";
 import { type ProviderPackInstallation, verifyRemoteProviderPack as verifyRemoteProviderPackIdentity } from "./remote-provider-pack-verification.js";
 import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js";
 import { NATIVE_SESSION_EXECUTION_LEASE_TTL_MS, lockNativeExecutionLeaseForReacquire, nativeExecutionLeaseHeldCondition, releaseUnstartedNativeExecutionClaim, startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js";
@@ -12075,8 +12075,12 @@ async function executePaperclipNativeSessionAttempt(
                     claim: { leaseOwner, attempt, controllerGeneration, controllerBootId: controller.bootId },
                     waitForClaim: runnerdBackend?.providerAdmissionRequired?.() === true,
                   });
-                } catch {
-                  try { await input.onLog?.("stderr", "[paperclip-runner] native provider-pack admission receipt could not be saved\n"); } catch {}
+                } catch (error) {
+                  // Name the database cause (a SQLSTATE only), so a lost
+                  // admission is diagnosable from the run log.
+                  const cause = nativeProviderPackAdmissionFailureCode(error)
+                    ?? (error instanceof Error && /^native_[a-z_]+$/.test(error.message) ? error.message : "unknown");
+                  try { await input.onLog?.("stderr", `[paperclip-runner] native provider-pack admission receipt could not be saved (cause=${cause})\n`); } catch {}
                 }
                 await runnerdBackend?.providerAdmissionPersisted?.();
                 if (session.resolveRuntimeRequest) await liveQuestions.attach();