From ea25af759e09297f1154e447031d1b536c27193d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 01:30:27 -0600 Subject: [PATCH] fix(native): name a resume that finds no saved Runner state, so Retry starts a fresh session (#20757) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can continue the provider session of the previous run on the same task. In a remote sandbox, the Runner keeps the session state inside the sandbox, and the server saves a harness backup after each run. > - On staging, five runs in 48 hours failed with `native_session_recovery_failed: Error: PRP provider resume state is unavailable` (class P in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`). > - In each case we read, the previous run had an accepted result, but its `harness_state.backup.persist` failed. Its per-turn sandbox was then destroyed. The next run started in a new sandbox with no Runner state for the session. > - The Runner transport threw an untyped error. The runtime could not tell it apart from a transport drop. A governed-wait or same-session checkpoint ended as a generic `native_session_interrupted` with an "ambiguous state" hold that said not to open a replacement session. > - This pull request types that refusal and lets the driver classify it as `provider_state_incompatible` when nothing is in flight. The existing recovery policy then decides. > - The benefit is that the run ends with the permanent named code `native_provider_state_incompatible`. Retry, or the next message, continues in a fresh provider session in the same workspace. No provider turn or tool call can be repeated, because the follow-up never started a turn. ## Linked Issues or Issue Description Refs #20712, #20723 (the same classification for a moved remote workspace root). Refs #20691 (fixes the `provider_identity_incomplete` part of class P). **What happened?** 1. Run A on a chat task finished. Its result was accepted. Its harness backup failed to save (`harness_state.backup.persist` failed, `native.post_result_failure`). 2. Run B started for the next chat message in a new sandbox. It rebound run A's checkpoint with `providerRecoveryPolicy: allow_replacement_after_governed_wait`. 3. The Runner found no control-plane state and no archived authority epoch. It threw `PRP provider resume state is unavailable` as a plain `Error`. 4. The runtime refused a replacement for the governed-wait policy and failed run B with the generic `native_session_recovery_failed`. The task stayed blocked with an "ambiguous state" hold. **Expected behavior** The run ends with a permanent named code and a working Retry, as it does for other deterministic refusals. **Steps to reproduce** 1. Create a Runner transport that resumes a saved provider session from an empty state directory. 2. Request `thread/read`. On `master` it rejects with a plain `Error`. 3. Recover a saved Codex, OpenCode or ACPX session through the driver when the transport fails that way. On `master` the result has no `failureClass`. **Paperclip version or commit** `master` at `9574065a28`. ## What Changed - `packages/paperclip-runner/src/contracts/harness-driver.ts`: add `NativeRunnerResumeStateUnavailableError` (same message as before) and `isRunnerResumeStateUnavailable`. - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: throw the typed error at the three branches where the control-plane state is absent. The branch with no PRP identity and the quarantine branch are unchanged. - `packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts`: classify the typed error as `provider_state_incompatible` when the saved session has no turn or runtime request in flight. Extract the in-flight test that the moved-root check already used, without changing it. - `doc/execution-semantics.md`: describe the new classification (additive). - Tests: a transport test (local and external Runner state), driver tests for Codex, OpenCode and ACPX Claude, in-flight cases that stay unclassified, and a server case that shows the blocked task offers Retry and that one click starts exactly one fresh-session run. ## Verification - `cd packages/paperclip-runner && npx vitest run src/backends/native-backend-factory.test.ts -t "no saved state"`: 5 pass. With the driver change reverted, the 3 "reports … as provider state incompatible" cases fail (no `failureClass`). - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-cold-restore.test.ts`: 5 pass. On `master` the error is a plain `Error`. - `cd packages/paperclip-runner && npx vitest run src/backends/native-backend-factory.test.ts src/drivers/codex/codex-app-server-driver.recovery.test.ts src/live/runnerd-codex-transport-cold-restore.test.ts src/native-session-runtime.test.ts`: 410 pass. - `cd server && npx vitest run src/services/workspace-recovery-retry-copyback.test.ts -t "does not continue automatically"`: 5 pass, including the new case. It checks `canRetry: true`, an accepted Retry, exactly one queued successor, and `forceFreshSession: true`. - `cd packages/paperclip-runner && npx tsc --noEmit -p .`: clean. - `pnpm verify`: exit 2 (broader verification required) because `doc/execution-semantics.md` is a global CI input. The focused checks above pass. Required CI supplies the broad coverage. ## Risks - Low. The change only adds a class to a failure that already failed. A checkpoint with `allow_replacement_after_resume_failure` already replaced the session for this failure, and still does. - For `same_session_only` and `allow_replacement_after_governed_wait`, the run now ends with `native_provider_state_incompatible` instead of a retryable `native_session_interrupted`. Same-run retries stop at once. They could not succeed, because the state is absent. - A session with a turn or runtime request in flight keeps the old unclassified behavior. This PR does not change it. - The server code path is unchanged. The Rust Runner is unchanged, so no new runner image is needed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution) and read-only staging API reads. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude --- doc/execution-semantics.md | 11 ++++ .../backends/native-backend-factory.test.ts | 51 ++++++++++++++++++- .../src/contracts/harness-driver.ts | 19 +++++++ .../codex/codex-app-server-driver-impl.ts | 31 +++++++---- ...nnerd-codex-transport-cold-restore.test.ts | 32 ++++++++++++ .../src/live/runnerd-codex-transport.ts | 8 +-- .../workspace-recovery-retry-copyback.test.ts | 10 +++- 7 files changed, 146 insertions(+), 16 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 5a2f0494b60..5f4951a9620 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -3000,6 +3000,17 @@ replacement behavior. When a replacement does archive remote continuity state and the remote move fails, the local state is not moved, and the run error keeps the bounded, redacted exit status and output of that move. +A Runner that keeps no control-plane state for the saved session (none is live +and no archived authority epoch matches it) reports a typed refusal. This +happens when a follow-up run starts in a new sandbox and the previous run's +harness backup was not saved. Resuming again reads the same empty state. When the +saved session has no turn or runtime request in flight, the driver classifies +the refusal as `provider_state_incompatible`. The checkpoint's recovery policy +then decides: a replacement session when the policy allows it, otherwise the run +ends with `native_provider_state_incompatible`, and Retry or the next message +continues in a fresh provider session in the same workspace. A session with +work in flight keeps the unclassified behavior. + Bootstrap retries, exact-checkpoint resumes, and fresh replacement sessions share three total provider attempts, including the original attempt. Linked run IDs, controller restarts, and duplicate wakes do not reset this budget. A resume after a graceful shutdown drain suspended the run for reattachment does not consume an attempt, because the run did not cause that restart. A crash restart (no drain record) consumes an attempt only when the attempt it interrupted spent provider work without progress. This is one rule over recorded outcomes, not over intents. The attempt counts as started only when the provider accepted a turn: a `turn.accepted` event or the provider's own `turn.started`. A startup still marked `beforeSpawn`, a runner that spawned but was never admitted, and a `turn.start` the provider did not accept are not charged. Progress means that an accepted turn then completed work: a tool execution or a completed item other than a usage report. An accepted turn with only usage, a failure, or no events is charged, so a resume that fails each time still uses up the budget. The first restart judges the original execution. Each later restart claim records the run's event-log position, and the next claim judges only the attempt recorded after it; a normal retry in between starts a new attempt (at its `native.process_start_requested`), so earlier work does not count for it. A reattachment adopts a provider that already started, so for it only progress counts. A resume that launches a new runner and that the controller died before starting used no provider work, so the next claim replaces it and the attempt ends at the attempt that resume was to run as. Uncharged restarts stop after 12 controller generations, so a crash loop that never starts the provider, or that makes progress and then crashes the controller each time, is still bounded. Legacy runs follow the same rule: a preparation interrupted before provider work started (`providerWorkStarted: false`) is retried in its own `interrupted_preparation` lane, which never spends the failure retry budget and is bounded at 12 retries. This holds for a hard kill as well as a graceful drain: once the killed controller's recorded process is proven gone, restart recovery fails the run as `process_lost` with the bootstrap certificate and schedules it in that lane. A run saved by an older release without a controller process stamp cannot be proven and stays in the charged lane. Exact-checkpoint resumes retain the 30-second delay. Replacement scheduling and predecessor lineage commit together, with one successor per predecessor and admission through the normal task locks, authorization, pause, approval, and budget gates. A due failed remote retry can retain its provider PID after heartbeat completion diff --git a/packages/paperclip-runner/src/backends/native-backend-factory.test.ts b/packages/paperclip-runner/src/backends/native-backend-factory.test.ts index 134c048e310..8ccefae294d 100644 --- a/packages/paperclip-runner/src/backends/native-backend-factory.test.ts +++ b/packages/paperclip-runner/src/backends/native-backend-factory.test.ts @@ -32,7 +32,7 @@ import { WORKSPACE, } from "../drivers/codex/codex-app-server-driver.test-support.js"; import { createNativeSessionBackend } from "../index.js"; -import { PROVIDER_SIDE_EFFECT_PENDING, PROVIDER_STATE_INCOMPATIBLE } from "../contracts/harness-driver.js"; +import { NativeRunnerResumeStateUnavailableError, PROVIDER_SIDE_EFFECT_PENDING, PROVIDER_STATE_INCOMPATIBLE } from "../contracts/harness-driver.js"; import { createCodexNativeSessionBackend, createRunnerdNativeSessionBackend } from "./codex-native-backend.js"; import { createOpenCodeNativeSessionBackend } from "./opencode-native-backend.js"; @@ -453,6 +453,55 @@ describe("native backend factory", () => { expect(factory).not.toHaveBeenCalled(); }); + // Staging class P ("PRP provider resume state is unavailable"): the previous + // run's result was accepted but its harness backup failed to save, and its + // per-turn sandbox was destroyed. The follow-up's new sandbox holds no Runner + // control-plane state for the saved session, so every resume reads the same + // empty state. That failure was untyped, so a governed-wait or same-session + // checkpoint ended in a generic interruption, and each new message resumed + // the same session and failed again. + describe("a Runner that has no saved state for the session", () => { + function statelessRecovery(input: NativeExecutionInputV1) { + const transport = new FakeCodexTransport(); + transport.rejectMethods.set("initialize", new NativeRunnerResumeStateUnavailableError()); + const backend = createNativeSessionBackend(input, { + codexTransportFactory: () => transport, + environment: { ...process.env, PAPERCLIP_WORKSPACE_CWD: WORKSPACE }, + }); + const snapshot = { + schema: "paperclip.native-session.v1", backendKind: "codex_app_server", driverKind: "codex_app_server", + sessionId: transport.threadId, providerSessionId: transport.providerSessionId, workingDirectory: WORKSPACE, + identity: { runId: "run", sessionId: "session", companyId: "company", issueId: "issue", agentId: "agent" }, + activeTurnId: null, terminalTurns: [], pendingRuntimeRequests: [], lastCursor: null, + }; + return { backend, snapshot }; + } + + it.each([ + ["Codex", execution()], + ["OpenCode", opencodeExecution()], + ["ACPX Claude", acpxExecution("claude")], + ])("reports a %s session with nothing in flight as provider state incompatible", async (_label, input) => { + const { backend, snapshot } = statelessRecovery(input); + await expect(backend.recoverSession!(snapshot as never, { signal: new AbortController().signal })) + .resolves.toEqual({ + recovered: false, + reason: expect.stringContaining("PRP provider resume state is unavailable"), + failureClass: PROVIDER_STATE_INCOMPATIBLE, + }); + }); + + it.each([ + ["an active turn", { activeTurnId: "turn-in-flight" }], + ["a pending runtime request", { pendingRuntimeRequests: [{ requestId: "request-1" }] }], + ])("leaves a session that holds %s unclassified", async (_label, inFlight) => { + const { backend, snapshot } = statelessRecovery(acpxExecution("claude")); + const recovered = await backend.recoverSession!({ ...snapshot, ...inFlight } as never, { signal: new AbortController().signal }); + expect(recovered).toMatchObject({ recovered: false, reason: expect.stringContaining("PRP provider resume state is unavailable") }); + expect(recovered).not.toHaveProperty("failureClass"); + }); + }); + it("names the agent generically when a remote Claude session opens outside its workspace", async () => { const { factory, backend } = movedRootRecovery(acpxExecution("claude")); const failure = await backend.openSession({ diff --git a/packages/paperclip-runner/src/contracts/harness-driver.ts b/packages/paperclip-runner/src/contracts/harness-driver.ts index 406523d3b5f..b8903443c8c 100644 --- a/packages/paperclip-runner/src/contracts/harness-driver.ts +++ b/packages/paperclip-runner/src/contracts/harness-driver.ts @@ -532,6 +532,25 @@ export class NativeRunnerSourceReconciliationRequiredError extends Error { } } +/** The Runner keeps no control-plane state for the saved session: none is + * live and no archived authority epoch matches it. A new sandbox whose + * harness backup was never saved has nothing to resume, and recovering again + * reads the same empty state. The driver classifies it only for a session + * with no turn or runtime request in flight. */ +export class NativeRunnerResumeStateUnavailableError extends Error { + readonly code = "native_runner_resume_state_unavailable"; + constructor() { + super("PRP provider resume state is unavailable"); + this.name = "NativeRunnerResumeStateUnavailableError"; + } +} + +export function isRunnerResumeStateUnavailable(error: unknown, depth = 0): boolean { + if (!error || typeof error !== "object" || depth > 5) return false; + if (error instanceof NativeRunnerResumeStateUnavailableError) return true; + return isRunnerResumeStateUnavailable((error as { cause?: unknown }).cause, depth + 1); +} + export interface HarnessSessionRecoveryResult { recovered: boolean; session?: HarnessSession; diff --git a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts index a006348de21..834fdf100e8 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts @@ -20,7 +20,7 @@ import type { PersistedHarnessTurnTerminal, } from "../../contracts/harness-driver.js"; import { NativeSessionProtocolIntegrityError, nativeRestartInterruptedTurnId } from "../../contracts/native-session-backend.js"; -import { HarnessReconciliationError, PROVIDER_SIDE_EFFECT_PENDING, PROVIDER_STATE_INCOMPATIBLE, runnerRecoveryFailureClass } from "../../contracts/harness-driver.js"; +import { HarnessReconciliationError, isRunnerResumeStateUnavailable, PROVIDER_SIDE_EFFECT_PENDING, PROVIDER_STATE_INCOMPATIBLE, runnerRecoveryFailureClass } from "../../contracts/harness-driver.js"; import { CODEX_CODEX_PROTOCOL_VERSION, CODEX_SKILLLESS_BASE_INSTRUCTIONS, @@ -127,6 +127,18 @@ function bootstrapCancellation( }; } + +/** A saved turn or runtime request whose outcome is unknown. A crash can save a + * turn's terminal record before clearing its active marker; that turn finished + * and is not in flight. */ +function persistedSessionInFlight(snapshot: PersistedHarnessSession): boolean { + const activeTurnIsTerminal = (snapshot.terminalTurns ?? []).some( + (terminal) => terminal.turnId === snapshot.activeTurnId, + ); + return (Boolean(snapshot.activeTurnId) && !activeTurnIsTerminal) + || Boolean(snapshot.pendingRuntimeRequests?.length); +} + export class CodexAppServerDriver implements HarnessDriver { readonly #options: CodexAppServerDriverOptions; readonly #caps: CodexCapabilities; @@ -411,14 +423,7 @@ export class CodexAppServerDriver implements HarnessDriver { this.#options.workingDirectoryAuthority === "remote_runner" && remoteRunnerWorkingDirectoryMoved(snapshot.workingDirectory, this.#options.environment) ) { - // A crash can save a turn's terminal record before clearing its active - // marker; that turn finished and is not in flight. - const activeTurnIsTerminal = (snapshot.terminalTurns ?? []).some( - (terminal) => terminal.turnId === snapshot.activeTurnId, - ); - const inFlight = (Boolean(snapshot.activeTurnId) && !activeTurnIsTerminal) - || Boolean(snapshot.pendingRuntimeRequests?.length); - return inFlight + return persistedSessionInFlight(snapshot) ? { recovered: false, reason: "persisted session working directory is not the assigned remote workspace, and a provider turn or runtime request was in flight", @@ -764,7 +769,13 @@ export class CodexAppServerDriver implements HarnessDriver { await cancellation.close().catch(() => {}); if (error instanceof NativeSessionProtocolIntegrityError) throw error; if (options.signal.aborted) options.signal.throwIfAborted(); - const failureClass = runnerRecoveryFailureClass(error); + // A Runner with no saved control-plane state for this session can never + // resume it here. With nothing in flight, the runtime's recovery policy + // decides, as for a moved workspace root. An in-flight session keeps the + // unclassified failure. + const failureClass = runnerRecoveryFailureClass(error) + ?? (isRunnerResumeStateUnavailable(error) && !persistedSessionInFlight(snapshot) + ? PROVIDER_STATE_INCOMPATIBLE : undefined); return { recovered: false, reason: redactCodexDiagnostic(String(error)), diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport-cold-restore.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport-cold-restore.test.ts index d7598b004fa..ae7c3859ffa 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport-cold-restore.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport-cold-restore.test.ts @@ -3,6 +3,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { expect, it, vi } from "vitest"; import { createCapabilityRunnerdCodexTransport } from "./runnerd-codex-transport.js"; +import { isRunnerResumeStateUnavailable, NativeRunnerResumeStateUnavailableError } from "../contracts/harness-driver.js"; import { defaultCapabilityRunnerdBinary, fakeCodex, fakeCodexArgs } from "../../test/support/runnerd-transport.js"; it.each(["restore", "refuse", "foreign-state"] as const)( @@ -105,3 +106,34 @@ it.each(["restore", "refuse", "foreign-state"] as const)( }, 30_000, ); + +// Staging class P: a follow-up's new sandbox holds no Runner control-plane +// state and no archived authority epoch for the saved session. The refusal is +// typed so the driver can tell the runtime the saved state cannot resume here. +it.each(["local", "external"] as const)("names a resume that finds no saved %s Runner state", async (placement) => { + const root = await mkdtemp(join(tmpdir(), "runnerd-resume-state-absent-")); + const prepare = vi.fn(async () => {}); + try { + const resumed = createCapabilityRunnerdCodexTransport({ + runnerBinary: defaultCapabilityRunnerdBinary(), + codexCommand: fakeCodex, + codexArgs: fakeCodexArgs(root), + stateDirectory: root, + prpIdentity: { runnerInstanceId: "absent-runner", environmentLeaseId: "absent-lease", runId: "follow-up-run", + normalizedSessionId: "absent-session", turnId: "follow-up-turn", itemId: "follow-up-item" }, + resumeProviderSession: { driverSessionId: "thread-saved", providerSessionId: "provider-saved" }, + ...(placement === "external" ? { + readRunnerState: vi.fn(async () => { throw new Error("no runner state"); }), + prepareExternalRunnerState: prepare, + archiveExternalRunnerState: vi.fn(async () => { throw new Error("nothing to archive"); }), + } : {}), + }); + const failure = await resumed.transport.request("thread/read", {}).then(() => null, (error: unknown) => error); + expect(failure).toBeInstanceOf(NativeRunnerResumeStateUnavailableError); + expect(isRunnerResumeStateUnavailable(failure)).toBe(true); + expect((failure as Error).message).toBe("PRP provider resume state is unavailable"); + await resumed.transport.close().catch(() => {}); + } finally { + await rm(root, { recursive: true, force: true }); + } +}); diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index 03dcf56ff2d..e28ca0fa2c1 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -70,7 +70,7 @@ import type { HarnessRuntimeRequestResolution, PersistedHarnessProviderIdentity, } from "../contracts/harness-driver.js"; -import { NativeRunnerSourceReconciliationRequiredError } from "../contracts/harness-driver.js"; +import { NativeRunnerResumeStateUnavailableError, NativeRunnerSourceReconciliationRequiredError } from "../contracts/harness-driver.js"; import { DurablePrpControlPlane, inspectStagedRunAttachment, @@ -6498,7 +6498,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { let rotationSource: Record | null = null; if (controlPlaneState === null) { if (!supportsColdRotation) { - throw new Error("PRP provider resume state is unavailable"); + throw new NativeRunnerResumeStateUnavailableError(); } // An interrupted rotation leaves only its archive marker. rotationSource = committedRotationSource ?? latestArchivedControlPlaneState( @@ -6506,7 +6506,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { desiredIdentity, ); if (!rotationSource) { - throw new Error("PRP provider resume state is unavailable"); + throw new NativeRunnerResumeStateUnavailableError(); } } else if (!exactAuthority && warmRecovery === null) { if (!supportsColdRotation) { @@ -6540,7 +6540,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } // A missing control plane either rotated from its archive above or threw. if (controlPlaneState === null) { - throw new Error("PRP provider resume state is unavailable"); + throw new NativeRunnerResumeStateUnavailableError(); } const runnerBinaryPath = this.options.runnerBinary ?? (this.options.prepareLaunch ? "paperclip-runnerd" : defaultCapabilityRunnerdBinary()); diff --git a/server/src/services/workspace-recovery-retry-copyback.test.ts b/server/src/services/workspace-recovery-retry-copyback.test.ts index cdc45a308ef..39bbad5f77b 100644 --- a/server/src/services/workspace-recovery-retry-copyback.test.ts +++ b/server/src/services/workspace-recovery-retry-copyback.test.ts @@ -166,6 +166,10 @@ describe("automatic retained workspace recovery", () => { { title: "its provider turn ended without an observed provider terminal", change: "unproven" }, { title: "the source was a system wake", change: "system" }, { title: "the failure was not a lost runner or allocation", change: "error" }, + // Staging class P: the new sandbox had no Runner state to resume the saved + // provider session. The Runner now names it, so the run ends with the + // permanent code and Retry continues in a fresh provider session. + { title: "the Runner had no saved state to resume the provider session", change: "unrestorable" }, ])("does not continue automatically when $title, and offers a Retry that works on first use", async ({ change }) => { const f = await restartAllocationLostSource(change === "turn" ? "in_flight" : change === "unproven" ? "unproven" : "none"); if (change === "system") await db.update(agentWakeupRequests).set({ requestedByActorType: "system", requestedByActorId: null, @@ -173,6 +177,9 @@ describe("automatic retained workspace recovery", () => { if (change === "system") await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, wakeReason: "issue_children_completed" } }) .where(eq(heartbeatRuns.id, f.runId)); if (change === "error") await db.update(heartbeatRuns).set({ error: "native_provider_terminal_failed" }).where(eq(heartbeatRuns.id, f.runId)); + if (change === "unrestorable") await db.update(heartbeatRuns).set({ errorCode: "native_provider_state_incompatible", + error: "native_session_recovery_failed: NativeRunnerResumeStateUnavailableError: PRP provider resume state is unavailable" }) + .where(eq(heartbeatRuns.id, f.runId)); const service = heartbeatService(db, { environmentRuntime: f.environmentRuntime }); await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" }); expect(await requestRestartInterruptedTurnRetry(db, f.claim)).toMatchObject({ requested: false, @@ -195,7 +202,8 @@ describe("automatic retained workspace recovery", () => { await service.resumeTaskRetry(f.companyId, receipt.requestId!); const successors = await queuedSuccessors(f.companyId); expect(successors).toHaveLength(1); - expect(successors[0].contextSnapshot).toMatchObject({ previousRunId: f.runId, taskRetryRequestId: receipt.requestId }); + expect(successors[0].contextSnapshot).toMatchObject({ previousRunId: f.runId, taskRetryRequestId: receipt.requestId, + forceFreshSession: true }); }, 30_000); // S54: the automatic Retry's own run then lost its runner too. Saved-work