fix(recovery): report an unprepared saved skill on resume instead of a skill access change (#22395)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Runner executions can be suspended, for example by a server
restart, and then resumed from their saved admitted input
> - On resume, runtime selection checks that every saved skill in the
admitted runtime context is still available
> - The check treated a permitted skill whose saved version could not be
prepared on this host the same as a skill the agent can no longer read
> - The run failed as `configuration_incomplete` with "Project skill
access changed while this execution was suspended", the task went to the
board, and the real preparation error was lost
> - This pull request separates the two cases: an absent saved skill is
still an access change, and a present skill with a missing source fails
with its actual preparation detail
> - The benefit is a correct failure message and recovery route, and a
recorded cause for the underlying preparation fault

## Linked Issues or Issue Description

**What happened?**
Two resumed native runs on staging failed right after a server restart
with `ConfigurationIncompleteFailure: Project skill access changed while
this execution was suspended. Restore access or start a new execution.`
Skill access did not change. The first attempt of each run prepared all
four saved skills. After the restart, the resumed attempt failed. The
tasks went to board recovery with a "secret is not set" next action and
stayed blocked.

**Expected behavior**
Preparation already refuses a saved skill that the agent can no longer
read ("A pinned skill is unavailable or no longer permitted"). So, at
the resume guard, a saved skill that is absent from the current entries
is an access change. A saved skill that is present but whose source is
`missing` had a preparation failure (cache, storage or materialization).
That run must fail with the actual `missingDetail`, as a fresh admission
does in `materializeSelectedSkills`. It must not claim an access change.

**Steps to reproduce**
1. Admit a native run whose saved `skillVersions` pin a skill.
2. Resume it with the same skill entry, but with `sourceStatus:
"missing"` and a materialization `missingDetail`.
3. Before this change, the resume rejects with
`ConfigurationIncompleteFailure` "Project skill access changed".

## What Changed

- `selectHeartbeatRuntime` resume guard: a saved skill that is absent
from the current runtime skill entries still raises
`skill_access_changed`.
- A saved skill that is present with `sourceStatus: "missing"` now
throws a plain error that names the skill key and its preparation
detail. The admitted revision and saved skill pins stay unchanged.
- Regression test for both the unprepared case (red before, green after)
and the available case. The existing revoked-access test is unchanged
and still passes.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/runtime-selection.test.ts`: 92 passed. The new
`missing` case failed before the fix with `expected
ConfigurationIncompleteFailure ... to not be an instance of
ConfigurationIncompleteFailure`.
- `cd server && npx vitest run
src/__tests__/heartbeat-runtime-skills.test.ts
src/__tests__/heartbeat-execution-configuration.test.ts`: 56 passed.
- `pnpm run typecheck` (server): passed.
- The broad `pnpm verify` selection did not finish within the local time
limit. Required CI supplies that coverage.

## Risks

- Low risk. The change is in one guard. Real access revocation keeps its
existing error and board route.
- An unprepared skill on resume now follows the ordinary failed-run path
(`adapter_failed`) instead of `configuration_incomplete`. This is the
same as a fresh admission with a missing skill source.
- This change does not fix the underlying host-side preparation fault.
It records its actual cause so that the next occurrence identifies it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution,
through Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-21 19:52:24 -08:00
1 parent 4ec1447a67
commit 7e30ba2c45
2 files changed
+45 -3

No files matched your search

@@ -27,6 +27,7 @@ import { claimNativeRestartRecoveries } from "../native-runtime/native-restart-r
import { buildNativeModelEnvelope, canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js"; import { buildNativeModelEnvelope, canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js";
import * as runner from "../../vendor/paperclip-runner/index.js"; import * as runner from "../../vendor/paperclip-runner/index.js";
import { prepareAndSelectHeartbeatRuntime, selectHeartbeatRuntime, type HeartbeatRuntimeSelectionInput } from "./runtime-selection.js"; import { prepareAndSelectHeartbeatRuntime, selectHeartbeatRuntime, type HeartbeatRuntimeSelectionInput } from "./runtime-selection.js";
import { ConfigurationIncompleteFailure } from "./run-preparation.js";
import { resolveRunnerEnvironmentForRun, resolveWarmRunnerIdleTimeoutMsForRun } from "../runner-environment-lifecycle.js"; import { resolveRunnerEnvironmentForRun, resolveWarmRunnerIdleTimeoutMsForRun } from "../runner-environment-lifecycle.js";
import { reserveRunBudget } from "../budget-reservations.js"; import { reserveRunBudget } from "../budget-reservations.js";
import { legacyControllerClaim } from "../legacy-controller-lease.js"; import { legacyControllerClaim } from "../legacy-controller-lease.js";
@@ -772,6 +773,41 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
expect((await read(run.id)).contextSnapshot?.skillVersions).toEqual({ [key]: versionId }); expect((await read(run.id)).contextSnapshot?.skillVersions).toEqual({ [key]: versionId });
}); });
it.each(["missing", "available"] as const)("distinguishes an unprepared permitted skill from revoked access on resume (%s)", async sourceStatus => {
const { input, run } = await fixture();
const key = "company/test/project-skill", versionId = randomUUID();
await db.update(heartbeatRuns).set({ contextSnapshot: { skillVersions: { [key]: versionId } } }).where(eq(heartbeatRuns.id, run.id));
input.run = await read(run.id);
vi.mocked(buildNativeRuntimeContext).mockImplementationOnce(async () => {
const context = nativeRuntimeContextFixture();
context.instructions.bundle.rootPath = instructionRoot;
context.skills = [{ key, runtimeName: "project-skill", versionId, bundle: { ...context.instructions.bundle } }];
return { ...context, aggregateDigest: canonicalNativeRuntimeContextDigest(context) };
});
const selected = await selectHeartbeatRuntime(db, input);
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection");
const saved = await read(run.id);
input.run = saved;
input.session.persistedNativeExecutionInput = selected.nativeExecution;
input.session.persistedRunnerProfile = saved.runnerProfileJson ?? {};
const detail = "Failed to materialize the selected skill version: synthetic storage read timed out";
input.config.runtimeSkillEntries = [{ key, runtimeName: "project-skill", versionId, source: "/unavailable/project-skill", sourceStatus,
missingDetail: sourceStatus === "missing" ? detail : null }];
const resuming = selectHeartbeatRuntime(db, input);
if (sourceStatus === "missing") {
const error = await resuming.then(() => null, (failure: unknown) => failure);
expect(error).toBeInstanceOf(Error);
expect(error).not.toBeInstanceOf(ConfigurationIncompleteFailure);
expect((error as Error).message).not.toContain("access changed");
expect((error as Error).message).toContain(key);
expect((error as Error).message).toContain(detail);
expect((await read(run.id)).runnerProfileJson).toEqual(saved.runnerProfileJson);
} else {
expect(await resuming).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution });
}
expect((await read(run.id)).contextSnapshot?.skillVersions).toEqual({ [key]: versionId });
});
it.each(["exact", "fresh", "bootstrap", "dead-runner", "expired", "foreign-owner", "missing-audit"] as const)( it.each(["exact", "fresh", "bootstrap", "dead-runner", "expired", "foreign-owner", "missing-audit"] as const)(
"admits only exact pending Stop cleanup at runtime selection (%s)", async mode => { "admits only exact pending Stop cleanup at runtime selection (%s)", async mode => {
const { input, run, issue } = await fixture(); const { input, run, issue } = await fixture();
@@ -610,13 +610,19 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
); );
if ("runtimeContext" in nativeExecution) { if ("runtimeContext" in nativeExecution) {
const savedVersions = parseObject(run.contextSnapshot?.skillVersions); const savedVersions = parseObject(run.contextSnapshot?.skillVersions);
const accessibleKeys = new Set(runtimeSkillEntries.filter(entry => entry.sourceStatus !== "missing").map(entry => entry.key)); const currentEntries = new Map(runtimeSkillEntries.map(entry => [entry.key, entry]));
if (nativeExecution.runtimeContext.skills.some(skill => const savedSkills = nativeExecution.runtimeContext.skills.filter(skill =>
Object.prototype.hasOwnProperty.call(savedVersions, skill.key) && !accessibleKeys.has(skill.key))) { Object.prototype.hasOwnProperty.call(savedVersions, skill.key));
if (savedSkills.some(skill => !currentEntries.has(skill.key))) {
throw new ConfigurationIncompleteFailure("Project skill access changed while this execution was suspended. Restore access or start a new execution.", { throw new ConfigurationIncompleteFailure("Project skill access changed while this execution was suspended. Restore access or start a new execution.", {
configurationIncomplete: { reason: "skill_access_changed", actionUrl: "/skills" }, configurationIncomplete: { reason: "skill_access_changed", actionUrl: "/skills" },
}); });
} }
const unprepared = savedSkills.find(skill => currentEntries.get(skill.key)?.sourceStatus === "missing");
if (unprepared) {
const detail = currentEntries.get(unprepared.key)?.missingDetail?.trim() || "its saved files are unavailable";
throw new Error(`Saved skill "${unprepared.key}" could not be prepared for this resumed execution: ${detail}`);
}
} }
// Recover only the originally admitted request. A stored idle // Recover only the originally admitted request. A stored idle
// checkpoint can precede an already-started provider turn, so even // checkpoint can precede an already-started provider turn, so even