mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(native): name a resume that finds no saved Runner state, so Retry starts a fresh session (#20757)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can continue the provider session of the previous run on the same task. In a remote sandbox, the Runner keeps the session state inside the sandbox, and the server saves a harness backup after each run. > - On staging, five runs in 48 hours failed with `native_session_recovery_failed: Error: PRP provider resume state is unavailable` (class P in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`). > - In each case we read, the previous run had an accepted result, but its `harness_state.backup.persist` failed. Its per-turn sandbox was then destroyed. The next run started in a new sandbox with no Runner state for the session. > - The Runner transport threw an untyped error. The runtime could not tell it apart from a transport drop. A governed-wait or same-session checkpoint ended as a generic `native_session_interrupted` with an "ambiguous state" hold that said not to open a replacement session. > - This pull request types that refusal and lets the driver classify it as `provider_state_incompatible` when nothing is in flight. The existing recovery policy then decides. > - The benefit is that the run ends with the permanent named code `native_provider_state_incompatible`. Retry, or the next message, continues in a fresh provider session in the same workspace. No provider turn or tool call can be repeated, because the follow-up never started a turn. ## Linked Issues or Issue Description Refs #20712, #20723 (the same classification for a moved remote workspace root). Refs #20691 (fixes the `provider_identity_incomplete` part of class P). **What happened?** 1. Run A on a chat task finished. Its result was accepted. Its harness backup failed to save (`harness_state.backup.persist` failed, `native.post_result_failure`). 2. Run B started for the next chat message in a new sandbox. It rebound run A's checkpoint with `providerRecoveryPolicy: allow_replacement_after_governed_wait`. 3. The Runner found no control-plane state and no archived authority epoch. It threw `PRP provider resume state is unavailable` as a plain `Error`. 4. The runtime refused a replacement for the governed-wait policy and failed run B with the generic `native_session_recovery_failed`. The task stayed blocked with an "ambiguous state" hold. **Expected behavior** The run ends with a permanent named code and a working Retry, as it does for other deterministic refusals. **Steps to reproduce** 1. Create a Runner transport that resumes a saved provider session from an empty state directory. 2. Request `thread/read`. On `master` it rejects with a plain `Error`. 3. Recover a saved Codex, OpenCode or ACPX session through the driver when the transport fails that way. On `master` the result has no `failureClass`. **Paperclip version or commit** `master` at `9574065a28`. ## What Changed - `packages/paperclip-runner/src/contracts/harness-driver.ts`: add `NativeRunnerResumeStateUnavailableError` (same message as before) and `isRunnerResumeStateUnavailable`. - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: throw the typed error at the three branches where the control-plane state is absent. The branch with no PRP identity and the quarantine branch are unchanged. - `packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts`: classify the typed error as `provider_state_incompatible` when the saved session has no turn or runtime request in flight. Extract the in-flight test that the moved-root check already used, without changing it. - `doc/execution-semantics.md`: describe the new classification (additive). - Tests: a transport test (local and external Runner state), driver tests for Codex, OpenCode and ACPX Claude, in-flight cases that stay unclassified, and a server case that shows the blocked task offers Retry and that one click starts exactly one fresh-session run. ## Verification - `cd packages/paperclip-runner && npx vitest run src/backends/native-backend-factory.test.ts -t "no saved state"`: 5 pass. With the driver change reverted, the 3 "reports … as provider state incompatible" cases fail (no `failureClass`). - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-cold-restore.test.ts`: 5 pass. On `master` the error is a plain `Error`. - `cd packages/paperclip-runner && npx vitest run src/backends/native-backend-factory.test.ts src/drivers/codex/codex-app-server-driver.recovery.test.ts src/live/runnerd-codex-transport-cold-restore.test.ts src/native-session-runtime.test.ts`: 410 pass. - `cd server && npx vitest run src/services/workspace-recovery-retry-copyback.test.ts -t "does not continue automatically"`: 5 pass, including the new case. It checks `canRetry: true`, an accepted Retry, exactly one queued successor, and `forceFreshSession: true`. - `cd packages/paperclip-runner && npx tsc --noEmit -p .`: clean. - `pnpm verify`: exit 2 (broader verification required) because `doc/execution-semantics.md` is a global CI input. The focused checks above pass. Required CI supplies the broad coverage. ## Risks - Low. The change only adds a class to a failure that already failed. A checkpoint with `allow_replacement_after_resume_failure` already replaced the session for this failure, and still does. - For `same_session_only` and `allow_replacement_after_governed_wait`, the run now ends with `native_provider_state_incompatible` instead of a retryable `native_session_interrupted`. Same-run retries stop at once. They could not succeed, because the state is absent. - A session with a turn or runtime request in flight keeps the old unclassified behavior. This PR does not change it. - The server code path is unchanged. The Rust Runner is unchanged, so no new runner image is needed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution) and read-only staging API reads. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
1 parent
9574065a28
commit
ea25af759e
7 files changed
+146
-16
No files matched your search
@@ -166,6 +166,10 @@ describe("automatic retained workspace recovery", () => {
|
||||
{ title: "its provider turn ended without an observed provider terminal", change: "unproven" },
|
||||
{ title: "the source was a system wake", change: "system" },
|
||||
{ title: "the failure was not a lost runner or allocation", change: "error" },
|
||||
// Staging class P: the new sandbox had no Runner state to resume the saved
|
||||
// provider session. The Runner now names it, so the run ends with the
|
||||
// permanent code and Retry continues in a fresh provider session.
|
||||
{ title: "the Runner had no saved state to resume the provider session", change: "unrestorable" },
|
||||
])("does not continue automatically when $title, and offers a Retry that works on first use", async ({ change }) => {
|
||||
const f = await restartAllocationLostSource(change === "turn" ? "in_flight" : change === "unproven" ? "unproven" : "none");
|
||||
if (change === "system") await db.update(agentWakeupRequests).set({ requestedByActorType: "system", requestedByActorId: null,
|
||||
@@ -173,6 +177,9 @@ describe("automatic retained workspace recovery", () => {
|
||||
if (change === "system") await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, wakeReason: "issue_children_completed" } })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
if (change === "error") await db.update(heartbeatRuns).set({ error: "native_provider_terminal_failed" }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (change === "unrestorable") await db.update(heartbeatRuns).set({ errorCode: "native_provider_state_incompatible",
|
||||
error: "native_session_recovery_failed: NativeRunnerResumeStateUnavailableError: PRP provider resume state is unavailable" })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
const service = heartbeatService(db, { environmentRuntime: f.environmentRuntime });
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
||||
expect(await requestRestartInterruptedTurnRetry(db, f.claim)).toMatchObject({ requested: false,
|
||||
@@ -195,7 +202,8 @@ describe("automatic retained workspace recovery", () => {
|
||||
await service.resumeTaskRetry(f.companyId, receipt.requestId!);
|
||||
const successors = await queuedSuccessors(f.companyId);
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0].contextSnapshot).toMatchObject({ previousRunId: f.runId, taskRetryRequestId: receipt.requestId });
|
||||
expect(successors[0].contextSnapshot).toMatchObject({ previousRunId: f.runId, taskRetryRequestId: receipt.requestId,
|
||||
forceFreshSession: true });
|
||||
}, 30_000);
|
||||
|
||||
// S54: the automatic Retry's own run then lost its runner too. Saved-work
|
||||
|
||||
Reference in new issue
Block a user