From 3eca7a6112afc5b75efe2ece9b2e8de02c49e9b7 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 12 Nov 2026 05:38:23 -0600 Subject: [PATCH] fix(runner): log which check refuses a saved provider-pack admission on reattach (#21085) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - OpenCode turns run in a remote Runner. When a server restarts, the new server reconnects to the Runner and reuses the provider pack admission that the run saved earlier. > - On staging, a run reconnected cleanly after an upgrade but then failed with `native_provider_artifacts_admission_unverified`. Its saved admission state stayed "pending". > - The reuse step refuses a saved admission for several reasons. It returns only false and logs nothing, so we cannot tell which check refused it. > - This pull request adds diagnostics only. It names the refused check in the run log and the server log. It also logs when the admission stays unverified for a session that needs it. > - The benefit is that the next upgrade audit shows the real cause, so the next fix can target it. ## Linked Issues or Issue Description **What happened?** After an automatic staging upgrade, a run reconnected to its live Runner without a new process. The next checkpoint failed with `native_provider_artifacts_admission_unverified`, which used one of the run's 3 attempts. The run log had no line that explained why the saved admission was not reused. **Expected behavior** When a saved admission is not reused, the run log and the server log name the check that refused it. **Steps to reproduce** 1. Start an OpenCode run on a remote sandbox. 2. Restart the server, so the run reconnects to its Runner. 3. If the saved admission is refused, the run fails later with `native_provider_artifacts_admission_unverified` and no reason in the logs. ## What Changed - `native-provider-pack-admission.ts`: `rememberRecoveredNativeProviderPack` accepts an optional `onRefused` callback. It is called with one reason: `source_state_pending`, `source_state_unknown`, `source_state_missing`, `receipt_missing`, `runner_instance_differs`, `environment_lease_differs`, `receipt_invalid`, or a session identity reason. The result of every check stays the same. An error in the callback is ignored. - `native-session-executor.ts`: the reattach path logs the reason to the run log and the server log. When `recordNativeProviderPackAdmission` returns false for a session that requires admission, it logs that too. - `native-provider-pack-admission.test.ts`: each refusal names its reason, and a reused admission reports nothing. ## Verification - `npx vitest run src/services/native-runtime/native-provider-pack-admission.test.ts` passes 46/46. The new reason tests fail on the old code because `onRefused` is never called. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts` passes 1412/1412. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. This change adds diagnostics only, and admission decisions do not change. The new log lines contain only a fixed reason code and run ids. They do not contain secrets or file paths. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, code search, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../native-provider-pack-admission.test.ts | 25 +++++++++++++++++ .../native-provider-pack-admission.ts | 27 ++++++++++++++----- .../native-runtime/native-session-executor.ts | 12 +++++++-- 3 files changed, 56 insertions(+), 8 deletions(-) diff --git a/server/src/services/native-runtime/native-provider-pack-admission.test.ts b/server/src/services/native-runtime/native-provider-pack-admission.test.ts index 90a421485f9..da6f2f9a312 100644 --- a/server/src/services/native-runtime/native-provider-pack-admission.test.ts +++ b/server/src/services/native-runtime/native-provider-pack-admission.test.ts @@ -105,6 +105,31 @@ describe("provider pack admission identity", () => { { identity: initial, sourceRunId: initial.runId, ...binding })).resolves.toBe(false); expect(admittedNativeProviderPack(recovered.session, initial)).toBeNull(); }); + it.each([ + ["source_state_pending", {}, { ...verified, state: "pending" }], + ["source_state_unknown", {}, { ...verified, state: "unknown" }], + ["source_state_missing", {}, null], + ["runner_instance_differs", { runnerInstanceId: "other-runner" }, verified], + ["environment_lease_differs", { runnerEnvironmentLeaseId: "other-lease" }, verified], + ["receipt_invalid", { sessionId: "foreign" }, verified], + ] as const)("names the refused check %s without admitting", async (reason, change, state) => { + const source = sessionFixture(); remember(source.session); + const saved = { ...admittedNativeProviderPack(source.session, initial)!, ...change }; + const recovered = sessionFixture(); + const onRefused = vi.fn(); + await expect(rememberRecoveredNativeProviderPack(receiptDb(saved, state), recovered.session, + { identity: initial, sourceRunId: initial.runId, ...binding, onRefused })).resolves.toBe(false); + expect(onRefused).toHaveBeenCalledExactlyOnceWith(reason); + expect(admittedNativeProviderPack(recovered.session, initial)).toBeNull(); + }); + it("reports nothing when the saved admission is reused", async () => { + const source = sessionFixture(); remember(source.session); + const recovered = sessionFixture(); + const onRefused = vi.fn(); + await expect(rememberRecoveredNativeProviderPack(receiptDb(admittedNativeProviderPack(source.session, initial)), recovered.session, + { identity: initial, sourceRunId: initial.runId, ...binding, onRefused })).resolves.toBe(true); + expect(onRefused).not.toHaveBeenCalled(); + }); it("uses only the exact same-run locked pre-invalidation source snapshot", async () => { const source = sessionFixture(); remember(source.session); const saved = admittedNativeProviderPack(source.session, initial)!; diff --git a/server/src/services/native-runtime/native-provider-pack-admission.ts b/server/src/services/native-runtime/native-provider-pack-admission.ts index 9d4ab3e40ba..6de5cd840a9 100644 --- a/server/src/services/native-runtime/native-provider-pack-admission.ts +++ b/server/src/services/native-runtime/native-provider-pack-admission.ts @@ -31,7 +31,11 @@ const admitted = new WeakMap>(); const digest = /^sha256:[a-f0-9]{64}$/; type NativeRunIdentity = ReturnType; -export type NativeProviderPackAdmissionSource = { runId: string; receipt: unknown; state: unknown; artifacts?: unknown }; +export type NativeProviderPackAdmissionRefusal = "session_identity_unavailable" | "session_identity_differs" + | "source_state_pending" | "source_state_unknown" | "source_state_missing" | "receipt_missing" + | "runner_instance_differs" | "environment_lease_differs" | "receipt_invalid"; + +export type NativeProviderPackAdmissionSource ={ runId: string; receipt: unknown; state: unknown; artifacts?: unknown }; export function nativeProviderPackAdmissionPending(admissionEpoch: string) { return { schema: "paperclip.native-provider-pack-admission-state.v1" as const, state: "pending" as const, admissionEpoch }; @@ -83,19 +87,30 @@ export async function rememberRecoveredNativeProviderPack(db: Db, session: Nativ runnerInstanceId: string; environmentLeaseId: string; source?: NativeProviderPackAdmissionSource; + /** Diagnostics only: names the check that refused the saved admission. */ + onRefused?: (reason: NativeProviderPackAdmissionRefusal) => void | Promise; }): Promise { admitted.delete(session); - if (typeof session.identity !== "function") return false; - if (canonicalProviderPackJson(session.identity()) !== canonicalProviderPackJson(input.identity)) return false; + const refuse = async (reason: NativeProviderPackAdmissionRefusal) => { + try { await input.onRefused?.(reason); } catch { /* diagnostics never change admission */ } + return false; + }; + if (typeof session.identity !== "function") return refuse("session_identity_unavailable"); + if (canonicalProviderPackJson(session.identity()) !== canonicalProviderPackJson(input.identity)) return refuse("session_identity_differs"); const source = input.source?.runId === input.sourceRunId ? input.source : (await db.select({ receipt: sql`${heartbeatRuns.runnerProfileJson}->'nativeProviderPackAdmission'`, state: sql`${heartbeatRuns.runnerProfileJson}->'nativeProviderPackAdmissionState'`, }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.sourceRunId), eq(heartbeatRuns.companyId, input.identity.companyId), eq(heartbeatRuns.agentId, input.identity.agentId), eq(heartbeatRuns.runtimeMode, "native"))).limit(1))[0]; - if (!verifiedState(source?.state)) return false; + if (!verifiedState(source?.state)) { + const state = source?.state && typeof source.state === "object" ? (source.state as Record).state : undefined; + return refuse(state === "pending" ? "source_state_pending" : state === "unknown" ? "source_state_unknown" : "source_state_missing"); + } const value = source?.receipt; - if (!value || typeof value !== "object" || Array.isArray(value)) return false; + if (!value || typeof value !== "object" || Array.isArray(value)) return refuse("receipt_missing"); const receipt = value as Record; + if (receipt.runnerInstanceId !== input.runnerInstanceId) return refuse("runner_instance_differs"); + if (receipt.runnerEnvironmentLeaseId !== input.environmentLeaseId) return refuse("environment_lease_differs"); const keys = new Set(["schema", "stage", "authority", "companyId", "agentId", "runId", "sessionId", "runnerInstanceId", "runnerEnvironmentLeaseId", "providerPackDigest", "opencodeProxySha256", "sourceRevision", "inheritedFromRunId"]); if (Object.keys(receipt).some(key => !keys.has(key)) || receipt.schema !== "paperclip.native-provider-pack-admission.v1" @@ -104,7 +119,7 @@ export async function rememberRecoveredNativeProviderPack(db: Db, session: Nativ || receipt.sessionId !== input.identity.sessionId || receipt.runnerInstanceId !== input.runnerInstanceId || receipt.runnerEnvironmentLeaseId !== input.environmentLeaseId || !digest.test(String(receipt.providerPackDigest)) || !digest.test(String(receipt.opencodeProxySha256)) || !/^[a-f0-9]{40}$/.test(String(receipt.sourceRevision)) - || (receipt.inheritedFromRunId !== undefined && (typeof receipt.inheritedFromRunId !== "string" || !receipt.inheritedFromRunId))) return false; + || (receipt.inheritedFromRunId !== undefined && (typeof receipt.inheritedFromRunId !== "string" || !receipt.inheritedFromRunId))) return refuse("receipt_invalid"); return rememberSelection(session, { admittedRunId: String(receipt.inheritedFromRunId ?? receipt.runId), companyId: input.identity.companyId, agentId: input.identity.agentId, sessionId: input.identity.sessionId, runnerInstanceId: input.runnerInstanceId, environmentLeaseId: input.environmentLeaseId, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 28028a99062..fd67d8f80b9 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -12495,7 +12495,7 @@ async function executePaperclipNativeSessionAttempt( throw new NativeControllerDetachedForRestartError(); } try { - await recordNativeProviderPackAdmission(input.db, { + const admissionVerified = await recordNativeProviderPackAdmission(input.db, { session, admissionEpoch: providerPackAdmissionEpoch, identity: { runId: input.execution.binding.runId, companyId: input.execution.binding.companyId, @@ -12504,6 +12504,10 @@ async function executePaperclipNativeSessionAttempt( claim: { leaseOwner, attempt, controllerGeneration, controllerBootId: controller.bootId }, waitForClaim: runnerdBackend?.providerAdmissionRequired?.() === true, }); + if (!admissionVerified && runnerdBackend?.providerAdmissionRequired?.() === true) { + logger.warn({ runId: input.execution.binding.runId }, "native provider-pack admission was not verified for this session"); + try { await input.onLog?.("stderr", "[paperclip-runner] native provider-pack admission was not verified for this session\n"); } catch {} + } } catch (error) { // Name the database cause (a SQLSTATE only), so a lost // admission is diagnosable from the run log. @@ -18238,7 +18242,11 @@ async function createRunnerdBackendWithinSessionClaim( try { await rememberRecoveredNativeProviderPack(input.db, session, { identity, sourceRunId, runnerInstanceId: effectiveRunnerInstanceId, environmentLeaseId: effectiveEnvironmentLeaseId, - source: input.providerPackAdmissionSource }); + source: input.providerPackAdmissionSource, + onRefused: async reason => { + logger.warn({ runId: input.execution.binding.runId, sourceRunId, reason }, "saved provider-pack admission was not reused"); + await input.onLog?.("stderr", `[paperclip-runner] saved provider-pack admission was not reused (reason=${reason})\n`); + } }); } catch { try { await input.onLog?.("stderr", "[paperclip-runner] prior provider-pack admission receipt is unavailable\n"); } catch {} }