From 7e30ba2c453b7de36c40c10b90ae67b0a47cf2bd Mon Sep 17 00:00:00 2001 From: paperclipbugbot Date: Sat, 21 Nov 2026 19:52:24 -0800 Subject: [PATCH] fix(recovery): report an unprepared saved skill on resume instead of a skill access change (#22395) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Runner executions can be suspended, for example by a server restart, and then resumed from their saved admitted input > - On resume, runtime selection checks that every saved skill in the admitted runtime context is still available > - The check treated a permitted skill whose saved version could not be prepared on this host the same as a skill the agent can no longer read > - The run failed as `configuration_incomplete` with "Project skill access changed while this execution was suspended", the task went to the board, and the real preparation error was lost > - This pull request separates the two cases: an absent saved skill is still an access change, and a present skill with a missing source fails with its actual preparation detail > - The benefit is a correct failure message and recovery route, and a recorded cause for the underlying preparation fault ## Linked Issues or Issue Description **What happened?** Two resumed native runs on staging failed right after a server restart with `ConfigurationIncompleteFailure: Project skill access changed while this execution was suspended. Restore access or start a new execution.` Skill access did not change. The first attempt of each run prepared all four saved skills. After the restart, the resumed attempt failed. The tasks went to board recovery with a "secret is not set" next action and stayed blocked. **Expected behavior** Preparation already refuses a saved skill that the agent can no longer read ("A pinned skill is unavailable or no longer permitted"). So, at the resume guard, a saved skill that is absent from the current entries is an access change. A saved skill that is present but whose source is `missing` had a preparation failure (cache, storage or materialization). That run must fail with the actual `missingDetail`, as a fresh admission does in `materializeSelectedSkills`. It must not claim an access change. **Steps to reproduce** 1. Admit a native run whose saved `skillVersions` pin a skill. 2. Resume it with the same skill entry, but with `sourceStatus: "missing"` and a materialization `missingDetail`. 3. Before this change, the resume rejects with `ConfigurationIncompleteFailure` "Project skill access changed". ## What Changed - `selectHeartbeatRuntime` resume guard: a saved skill that is absent from the current runtime skill entries still raises `skill_access_changed`. - A saved skill that is present with `sourceStatus: "missing"` now throws a plain error that names the skill key and its preparation detail. The admitted revision and saved skill pins stay unchanged. - Regression test for both the unprepared case (red before, green after) and the available case. The existing revoked-access test is unchanged and still passes. ## Verification - `cd server && npx vitest run src/services/heartbeat/runtime-selection.test.ts`: 92 passed. The new `missing` case failed before the fix with `expected ConfigurationIncompleteFailure ... to not be an instance of ConfigurationIncompleteFailure`. - `cd server && npx vitest run src/__tests__/heartbeat-runtime-skills.test.ts src/__tests__/heartbeat-execution-configuration.test.ts`: 56 passed. - `pnpm run typecheck` (server): passed. - The broad `pnpm verify` selection did not finish within the local time limit. Required CI supplies that coverage. ## Risks - Low risk. The change is in one guard. Real access revocation keeps its existing error and board route. - An unprepared skill on resume now follows the ordinary failed-run path (`adapter_failed`) instead of `configuration_incomplete`. This is the same as a fresh admission with a missing skill source. - This change does not fix the underlying host-side preparation fault. It records its actual cause so that the next occurrence identifies it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, through Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- .../heartbeat/runtime-selection.test.ts | 36 +++++++++++++++++++ .../services/heartbeat/runtime-selection.ts | 12 +++++-- 2 files changed, 45 insertions(+), 3 deletions(-) diff --git a/server/src/services/heartbeat/runtime-selection.test.ts b/server/src/services/heartbeat/runtime-selection.test.ts index dcb96533ada..d1b313c1d72 100644 --- a/server/src/services/heartbeat/runtime-selection.test.ts +++ b/server/src/services/heartbeat/runtime-selection.test.ts @@ -27,6 +27,7 @@ import { claimNativeRestartRecoveries } from "../native-runtime/native-restart-r import { buildNativeModelEnvelope, canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js"; import * as runner from "../../vendor/paperclip-runner/index.js"; import { prepareAndSelectHeartbeatRuntime, selectHeartbeatRuntime, type HeartbeatRuntimeSelectionInput } from "./runtime-selection.js"; +import { ConfigurationIncompleteFailure } from "./run-preparation.js"; import { resolveRunnerEnvironmentForRun, resolveWarmRunnerIdleTimeoutMsForRun } from "../runner-environment-lifecycle.js"; import { reserveRunBudget } from "../budget-reservations.js"; import { legacyControllerClaim } from "../legacy-controller-lease.js"; @@ -772,6 +773,41 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = expect((await read(run.id)).contextSnapshot?.skillVersions).toEqual({ [key]: versionId }); }); + it.each(["missing", "available"] as const)("distinguishes an unprepared permitted skill from revoked access on resume (%s)", async sourceStatus => { + const { input, run } = await fixture(); + const key = "company/test/project-skill", versionId = randomUUID(); + await db.update(heartbeatRuns).set({ contextSnapshot: { skillVersions: { [key]: versionId } } }).where(eq(heartbeatRuns.id, run.id)); + input.run = await read(run.id); + vi.mocked(buildNativeRuntimeContext).mockImplementationOnce(async () => { + const context = nativeRuntimeContextFixture(); + context.instructions.bundle.rootPath = instructionRoot; + context.skills = [{ key, runtimeName: "project-skill", versionId, bundle: { ...context.instructions.bundle } }]; + return { ...context, aggregateDigest: canonicalNativeRuntimeContextDigest(context) }; + }); + const selected = await selectHeartbeatRuntime(db, input); + if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection"); + const saved = await read(run.id); + input.run = saved; + input.session.persistedNativeExecutionInput = selected.nativeExecution; + input.session.persistedRunnerProfile = saved.runnerProfileJson ?? {}; + const detail = "Failed to materialize the selected skill version: synthetic storage read timed out"; + input.config.runtimeSkillEntries = [{ key, runtimeName: "project-skill", versionId, source: "/unavailable/project-skill", sourceStatus, + missingDetail: sourceStatus === "missing" ? detail : null }]; + const resuming = selectHeartbeatRuntime(db, input); + if (sourceStatus === "missing") { + const error = await resuming.then(() => null, (failure: unknown) => failure); + expect(error).toBeInstanceOf(Error); + expect(error).not.toBeInstanceOf(ConfigurationIncompleteFailure); + expect((error as Error).message).not.toContain("access changed"); + expect((error as Error).message).toContain(key); + expect((error as Error).message).toContain(detail); + expect((await read(run.id)).runnerProfileJson).toEqual(saved.runnerProfileJson); + } else { + expect(await resuming).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution }); + } + expect((await read(run.id)).contextSnapshot?.skillVersions).toEqual({ [key]: versionId }); + }); + it.each(["exact", "fresh", "bootstrap", "dead-runner", "expired", "foreign-owner", "missing-audit"] as const)( "admits only exact pending Stop cleanup at runtime selection (%s)", async mode => { const { input, run, issue } = await fixture(); diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts index 2462b365c50..fd10d8ad1c1 100644 --- a/server/src/services/heartbeat/runtime-selection.ts +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -610,13 +610,19 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele ); if ("runtimeContext" in nativeExecution) { const savedVersions = parseObject(run.contextSnapshot?.skillVersions); - const accessibleKeys = new Set(runtimeSkillEntries.filter(entry => entry.sourceStatus !== "missing").map(entry => entry.key)); - if (nativeExecution.runtimeContext.skills.some(skill => - Object.prototype.hasOwnProperty.call(savedVersions, skill.key) && !accessibleKeys.has(skill.key))) { + const currentEntries = new Map(runtimeSkillEntries.map(entry => [entry.key, entry])); + const savedSkills = nativeExecution.runtimeContext.skills.filter(skill => + Object.prototype.hasOwnProperty.call(savedVersions, skill.key)); + if (savedSkills.some(skill => !currentEntries.has(skill.key))) { throw new ConfigurationIncompleteFailure("Project skill access changed while this execution was suspended. Restore access or start a new execution.", { configurationIncomplete: { reason: "skill_access_changed", actionUrl: "/skills" }, }); } + const unprepared = savedSkills.find(skill => currentEntries.get(skill.key)?.sourceStatus === "missing"); + if (unprepared) { + const detail = currentEntries.get(unprepared.key)?.missingDetail?.trim() || "its saved files are unavailable"; + throw new Error(`Saved skill "${unprepared.key}" could not be prepared for this resumed execution: ${detail}`); + } } // Recover only the originally admitted request. A stored idle // checkpoint can precede an already-started provider turn, so even