From 6ad4529c5afff32c449dd567258fbc731afff263 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 12:56:57 -0600 Subject: [PATCH] fix(native): preserve pack proof across interrupted reattachment (#21246) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server can connect again to a live Runner after a restart. > - Each new controller claim changes provider-pack admission to pending. > - The claim kept the previous verified proof only in local memory. > - If recovery failed before its callback, the next claim lost that proof. > - This pull request saves a small reference to the verified proof with the claim. > - The next claim can check the same receipt without weakening admission checks. ## Linked Issues or Issue Description **What happened?** A restart recovery attempt timed out before its admission callback. The next attempt refused the saved pack with `source_state_pending`. It then failed with `native_provider_artifacts_admission_unverified`. **Expected behavior** The next attempt can reuse the previous verified proof for the same receipt, session, Runner, and lease. Current admission must remain pending until its normal callback succeeds. **Steps to reproduce** 1. Save verified provider-pack admission for a run. 2. Claim that run again. Commit the pending admission state. 3. Interrupt recovery before the session admission callback. 4. Claim the same run again and reconnect to the same Runner. 5. Observe that the old code refuses the saved receipt. Related work: #20682 and #20850 address lock waits. #21085 adds refusal logs. #21161 addresses command waits after restart. This change addresses lost admission proof across interrupted claims. The original transport timeout remains a separate investigation. ## What Changed - The admission module prepares the claim patch and its recovery source. - The claim saves a versioned reference with the source run, receipt digest, and previous verified state. It adds no copy of provider files or credentials. - A later pending claim can use that reference only if the receipt is unchanged. Normal company, agent, session, Runner, and lease checks still apply. - Missing proof, unknown state, a changed receipt, or a different Runner stays refused. The callback clears the reference when it records its decision. - The executor uses this helper under its existing claim lock. Existing artifact-retention and ownership checks stay in place. - The execution guide states the previous-release behavior. Old verified records seed the reference. Old pending records without a reference remain refused. Older readers ignore the field and keep their existing refusal behavior. No migration is needed. ## Verification - The new interrupted-claim regression failed before the fix: admission returned false instead of true. - Focused admission, PostgreSQL fencing, and executor lease tests passed after the rebase: 61 tests on commit c8123cc35d. - The database case saves two interrupted claims, reads their persisted state, checks a different Runner is refused, and publishes verified admission for the original Runner. - Negative cases cover missing proof, unknown state, a changed receipt, a different source run, a different Runner, and a different lease. Existing company and claim fences also pass. - `pnpm verify` returned exit 2 because the execution guide requires broader CI. Required CI must cover the final commit. - Local server typecheck was killed with SIGKILL, including its single-thread retry. This is an environment gap. Required CI must provide typecheck and build results. - Repository preflight installed dependencies and built shared packages. It could not finish Runner preparation because cargo is absent. - A live OpenCode restart workflow and recovery of the original task are not verified. No task replay or deployment was performed. ## Risks - The added profile field carries historical evidence. It does not mark the current claim verified. Its use is limited to an unchanged receipt with the normal identity checks. - Existing pending or unknown incidents without this evidence cannot gain proof from this fix. They keep their normal recovery path. - No provider attempt limit, approval gate, workspace hold, or company boundary changes. ## Model Used OpenAI GPT-6, through Codex, with code search, tool use, and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 14 +++++ .../native-provider-pack-admission.test.ts | 63 ++++++++++++++++++- .../native-provider-pack-admission.ts | 31 ++++++++- .../native-session-executor.test.ts | 5 +- .../native-runtime/native-session-executor.ts | 10 ++- 5 files changed, 113 insertions(+), 10 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 74e550ad52a..faf96be0013 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -6204,6 +6204,20 @@ can be accepted without a terminal acknowledgement. The selected run and its history refresh after either outcome. Stop does not resume the provider or change a manual task hold. +### Provider-pack admission during restart recovery + +Provider-pack admission keeps a digest-bound historical proof when a controller +claim changes its current admission to pending. If a restart or a failed recovery +barrier interrupts that claim before its admission callback, the next same-run +claim can reuse that proof only for the unchanged receipt. Normal session, +company, agent, Runner and environment-lease checks still apply. Current artifact +retention remains blocked until the new claim publishes verified admission. +Unknown admission state discards the historical proof; a callback clears it when +it records its decision. This adds no background work or provider replay. +Previous-release verified receipts seed this proof on their first new claim. +Older pending or unknown records without it remain unverified, as before. Older +readers ignore the added profile field and retain their existing refusal behavior. + ### Native session diagnostic ownership Native session diagnostics follow the current execution's recorder epoch, including diff --git a/server/src/services/native-runtime/native-provider-pack-admission.test.ts b/server/src/services/native-runtime/native-provider-pack-admission.test.ts index da6f2f9a312..204663dab7c 100644 --- a/server/src/services/native-runtime/native-provider-pack-admission.test.ts +++ b/server/src/services/native-runtime/native-provider-pack-admission.test.ts @@ -5,7 +5,7 @@ import { agents, companies, createDb, heartbeatRuns, issues, nativeRunFinalizati import type { NativeSession } from "../../vendor/paperclip-runner/index.js"; import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { canonicalProviderPackJson, type ProviderPackIdentity } from "./remote-provider-pack-verification.js"; -import { admittedNativeProviderPack, nativeProviderPackAdmissionPending, recordNativeProviderPackAdmission, +import { admittedNativeProviderPack, nativeProviderPackAdmissionPending, prepareNativeProviderPackAdmission, recordNativeProviderPackAdmission, rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack } from "./native-provider-pack-admission.js"; const hash = (value: string) => `sha256:${createHash("sha256").update(value).digest("hex")}`; @@ -32,6 +32,41 @@ function receiptDb(receipt: unknown, state: unknown = verified): Db { } describe("provider pack admission identity", () => { + it("reattaches after a previous claim was interrupted before its admission callback", async () => { + const original = sessionFixture(); remember(original.session); + const receipt = admittedNativeProviderPack(original.session, initial)!; + const profile = { nativeProviderPackAdmission: receipt, nativeProviderPackAdmissionState: verified }; + const first = prepareNativeProviderPackAdmission(initial.runId, profile, randomUUID()); + // The claim commits, then snapshot/drain fails before the callback. The + // replacement controller has only persisted data, not the first's locals. + const pending = JSON.parse(JSON.stringify({ ...profile, ...first.patch })); + expect(pending.nativeProviderPackAdmissionState.state).toBe("pending"); + const second = prepareNativeProviderPackAdmission(initial.runId, pending, randomUUID()); + const recovered = sessionFixture(); + await expect(rememberRecoveredNativeProviderPack(receiptDb(receipt, pending.nativeProviderPackAdmissionState), recovered.session, + { identity: initial, sourceRunId: initial.runId, ...binding, source: second.source })).resolves.toBe(true); + expect(admittedNativeProviderPack(recovered.session, initial)).toEqual(receipt); + }); + it.each(["unknown", "missing", "different receipt", "different run", "different runner", "different lease"])( + "does not reuse a pending recovery proof with %s", async change => { + const original = sessionFixture(); remember(original.session); + const receipt = admittedNativeProviderPack(original.session, initial)!; + const profile = { nativeProviderPackAdmission: receipt, nativeProviderPackAdmissionState: verified }; + const first = prepareNativeProviderPackAdmission(initial.runId, profile, randomUUID()); + const pending = JSON.parse(JSON.stringify({ ...profile, ...first.patch })); + if (change === "unknown") pending.nativeProviderPackAdmissionState.state = "unknown"; + if (change === "missing") delete pending.nativeProviderPackAdmissionRecovery; + if (change === "different receipt") pending.nativeProviderPackAdmission.providerPackDigest = hash("replacement"); + if (change === "different run") pending.nativeProviderPackAdmissionRecovery.runId = "other-run"; + const next = prepareNativeProviderPackAdmission(initial.runId, pending, randomUUID()); + const recovered = sessionFixture(); + await expect(rememberRecoveredNativeProviderPack(receiptDb(null), recovered.session, { + identity: initial, sourceRunId: initial.runId, ...binding, source: next.source, + ...(change === "different runner" ? { runnerInstanceId: "replacement" } : {}), + ...(change === "different lease" ? { environmentLeaseId: "replacement" } : {}), + })).resolves.toBe(false); + expect(admittedNativeProviderPack(recovered.session, initial)).toBeNull(); + }); it("does not publish selected configuration before successful session admission", async () => { const { session } = sessionFixture(); expect(admittedNativeProviderPack(session, initial)).toBeNull(); @@ -179,6 +214,32 @@ describe("provider pack admission PostgreSQL fencing", () => { await expect(recordNativeProviderPackAdmission(db, { ...f, claim: { ...f.claim, [key]: changed } as typeof f.claim })).resolves.toBe(false); expect((await read(f.identity.runId)).runnerProfileJson).toEqual(f.profile); }); + it("republishes admission after two persisted interrupted claims without admitting a different Runner", async () => { + const f = await fixture(); + await expect(recordNativeProviderPackAdmission(db, f)).resolves.toBe(true); + for (let i = 0; i < 2; i++) { + const prior = (await read(f.identity.runId)).runnerProfileJson!; + const claim = prepareNativeProviderPackAdmission(f.identity.runId, prior, randomUUID()); + await db.update(heartbeatRuns).set({ runnerProfileJson: { ...prior, ...claim.patch } }).where(eq(heartbeatRuns.id, f.identity.runId)); + } + const prior = (await read(f.identity.runId)).runnerProfileJson!; + const claim = prepareNativeProviderPackAdmission(f.identity.runId, prior, randomUUID()); + const recovered = sessionFixture(f.identity); + await expect(rememberRecoveredNativeProviderPack(db, recovered.session, { + identity: f.identity, sourceRunId: f.identity.runId, ...binding, source: claim.source, + runnerInstanceId: "replacement", + })).resolves.toBe(false); + await expect(rememberRecoveredNativeProviderPack(db, recovered.session, { + identity: f.identity, sourceRunId: f.identity.runId, ...binding, source: claim.source, + })).resolves.toBe(true); + await db.update(heartbeatRuns).set({ runnerProfileJson: { ...prior, ...claim.patch } }).where(eq(heartbeatRuns.id, f.identity.runId)); + await expect(recordNativeProviderPackAdmission(db, { ...f, session: recovered.session, + admissionEpoch: claim.patch.nativeProviderPackAdmissionState.admissionEpoch })).resolves.toBe(true); + const final = (await read(f.identity.runId)).runnerProfileJson!; + expect(final.nativeProviderPackAdmissionState).toMatchObject({ state: "verified" }); + expect(final.nativeProviderPackAdmission).toEqual(prior.nativeProviderPackAdmission); + expect(final.nativeProviderPackAdmissionRecovery).toBeNull(); + }); it("rejects an expired claim and a terminal run", async () => { const f = await fixture(); await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(0) }).where(eq(nativeRunFinalizations.runId, f.identity.runId)); diff --git a/server/src/services/native-runtime/native-provider-pack-admission.ts b/server/src/services/native-runtime/native-provider-pack-admission.ts index 6de5cd840a9..6a2e3d05f32 100644 --- a/server/src/services/native-runtime/native-provider-pack-admission.ts +++ b/server/src/services/native-runtime/native-provider-pack-admission.ts @@ -41,11 +41,39 @@ export function nativeProviderPackAdmissionPending(admissionEpoch: string) { return { schema: "paperclip.native-provider-pack-admission-state.v1" as const, state: "pending" as const, admissionEpoch }; } +/** Called under the executor's claim lock, before provider admission. */ +export function prepareNativeProviderPackAdmission(runId: string, profile: Record | null | undefined, admissionEpoch: string) { + const receipt = profile?.nativeProviderPackAdmission; + let state = profile?.nativeProviderPackAdmissionState; + const saved = profile?.nativeProviderPackAdmissionRecovery as Record | null | undefined; + const receiptDigest = receipt && typeof receipt === "object" && !Array.isArray(receipt) + ? `sha256:${createHash("sha256").update(canonicalProviderPackJson(receipt)).digest("hex")}` : null; + // A pending claim has not selected a different pack. It can carry forward + // the exact verified proof that the previous claim captured under its lock. + // Unknown state means selection was refused and must never regain that proof. + if (admissionState(state, "pending") + && saved?.schema === "paperclip.native-provider-pack-admission-recovery.v1" + && Object.keys(saved).sort().join(",") === "receiptDigest,runId,schema,state" + && saved.runId === runId && receiptDigest !== null && saved.receiptDigest === receiptDigest + && verifiedState(saved.state)) state = saved.state; + const recovery = receiptDigest !== null && verifiedState(state) + ? { schema: "paperclip.native-provider-pack-admission-recovery.v1", runId, receiptDigest, state } : null; + return { + source: { runId, receipt, state, artifacts: profile?.nativeProviderArtifacts }, + patch: { nativeProviderPackAdmissionState: nativeProviderPackAdmissionPending(admissionEpoch), + nativeProviderPackAdmissionRecovery: recovery }, + }; +} + function verifiedState(value: unknown): boolean { + return admissionState(value, "verified"); +} + +function admissionState(value: unknown, expected: "pending" | "verified"): boolean { if (!value || typeof value !== "object" || Array.isArray(value)) return false; const state = value as Record; return Object.keys(state).sort().join(",") === "admissionEpoch,schema,state" - && state.schema === "paperclip.native-provider-pack-admission-state.v1" && state.state === "verified" + && state.schema === "paperclip.native-provider-pack-admission-state.v1" && state.state === expected && typeof state.admissionEpoch === "string" && /^[a-f0-9-]{36}$/.test(state.admissionEpoch); } @@ -210,6 +238,7 @@ async function recordNativeProviderPackAdmissionOnce(db: Db, input: AdmissionInp === canonicalProviderPackJson({ ...receipt, inheritedFromRunId: null })); if (verified && current.state === "verified") return true; const patch = { ...(verified && previous == null ? { nativeProviderPackAdmission: receipt } : {}), + ...(run.profile?.nativeProviderPackAdmissionRecovery != null ? { nativeProviderPackAdmissionRecovery: null } : {}), nativeProviderPackAdmissionState: { ...nativeProviderPackAdmissionPending(input.admissionEpoch), state: verified ? "verified" : "unknown" } }; const saved = await tx.update(heartbeatRuns).set({ runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) || ${JSON.stringify(patch)}::jsonb`, diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 898df3a20e6..d2efd0d2b06 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -10683,9 +10683,10 @@ describe("native session execution lease fencing", () => { }); await executePaperclipNativeSession({ db: leaseDb(execution, {}, {}, updates, profile), execution, runnerInstanceId: "runner" }); expect(state.execute).toHaveBeenCalledOnce(); - expect(observed).toEqual({ nativeProviderPackAdmissionState: { schema: "paperclip.native-provider-pack-admission-state.v1", + expect(observed).toMatchObject({ nativeProviderPackAdmissionState: { schema: "paperclip.native-provider-pack-admission-state.v1", state: "pending", admissionEpoch: expect.any(String) } }); - expect(JSON.stringify(observed)).not.toContain(profile.nativeProviderPackAdmissionState.admissionEpoch); + expect((observed as unknown as Record).nativeProviderPackAdmissionState.admissionEpoch) + .not.toBe(profile.nativeProviderPackAdmissionState.admissionEpoch); expect(profile.nativeProviderPackAdmissionState.state).toBe("verified"); }); it("does not admit a session if mandatory claim invalidation fails", async () => { diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index a2d43d50212..64ff926b87f 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -45,7 +45,7 @@ import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export- import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; import { providerPackWorkspaceId, recordAdmittedRemoteProviderPack, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js"; -import { nativeProviderPackAdmissionFailureCode, nativeProviderPackAdmissionPending, rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack, recordNativeProviderPackAdmission, type NativeProviderPackAdmissionSource } from "./native-provider-pack-admission.js"; +import { nativeProviderPackAdmissionFailureCode, prepareNativeProviderPackAdmission, rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack, recordNativeProviderPackAdmission, type NativeProviderPackAdmissionSource } from "./native-provider-pack-admission.js"; import { type ProviderPackInstallation, verifyRemoteProviderPack as verifyRemoteProviderPackIdentity } from "./remote-provider-pack-verification.js"; import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js"; import { NATIVE_SESSION_EXECUTION_LEASE_TTL_MS, lockNativeExecutionLeaseForReacquire, nativeExecutionLeaseHeldCondition, releaseUnstartedNativeExecutionClaim, startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js"; @@ -11259,10 +11259,8 @@ async function executePaperclipNativeSessionAttempt( .returning({ runId: nativeRunFinalizations.runId }) .then((rows) => rows[0] ?? null); if (!claimed) throw new Error("native_session_lease_lost"); - providerPackAdmissionSource = { runId: coordinator.runId, - receipt: boundRun.runnerProfileJson?.nativeProviderPackAdmission, - state: boundRun.runnerProfileJson?.nativeProviderPackAdmissionState, - artifacts: boundRun.runnerProfileJson?.nativeProviderArtifacts }; + const providerPackAdmission = prepareNativeProviderPackAdmission(coordinator.runId, boundRun.runnerProfileJson, providerPackAdmissionEpoch); + providerPackAdmissionSource = providerPackAdmission.source; controllerGeneration = nextControllerGeneration; claimedStopCleanup = stopRecovery; await tx @@ -11270,7 +11268,7 @@ async function executePaperclipNativeSessionAttempt( .set({ nativePhase: "observed", nativePhaseUpdatedAt: leaseNow, - runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) || ${JSON.stringify({ nativeProviderPackAdmissionState: nativeProviderPackAdmissionPending(providerPackAdmissionEpoch), + runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) || ${JSON.stringify({ ...providerPackAdmission.patch, ...(claimedStartupContext ? { nativeStartupContext: { schema: "paperclip.native-startup-context.v1", companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId, runId: input.execution.binding.runId, contextId: claimedStartupContext.id, deadlineAtMs: claimedStartupContext.deadlineAtMs, leaseId: target?.kind === "remote" ? target.leaseId : null,