From 611dc68581c523e9cdc50de99f891d571e1f5394 Mon Sep 17 00:00:00 2001 From: paperclipbugbot Date: Fri, 20 Nov 2026 16:52:11 -0800 Subject: [PATCH] fix(runner): continue a lost turn on a resumed warm session (#22447) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents in a managed sandbox run under the Paperclip runner (`runnerd`). When `runnerd` exits mid-run, the transport relaunches it, and the runner restores the same conversation > - #22235 lets the execution that started a turn send one continuation turn when the restored runner reports that turn as `provider_turn_lost_on_restore` > - That rule applies only to a session that the execution opened fresh (`!recovered`). A run that resumes the warm conversation of its previous run is a recovered session. A chat follow-up does this on every new message > - So a resumed session that loses its own task turn gets no continuation. The run fails, and the user must press Retry > - This pull request makes the task turn that this execution started its own turn, whether the session was opened fresh or resumed > - The benefit is that a chat follow-up keeps its conversation and workspace after a mid-run `runnerd` restore, as #22235 intended ## Linked Issues or Issue Description Refs #22235 (continue a turn lost to a mid-run runnerd restore). Refs #22068 (disk reserve). Refs #22239 (stop retrying a run that filled its sandbox disk). **What happened?** On staging, an agent in a chat task filled its 10 GiB managed sandbox disk. It fetched the full history of a large Git repository and searched it. `runnerd` exited with `failed to create private durable state temporary file: No space left on device (os error 28)`. The disk reserve let the transport relaunch `runnerd`, and the runner restored the same conversation. Then the runner reported `turn.failed {code: provider_turn_lost_on_restore, recoverable: true}` for the turn that this run had started. The runtime did not send the continuation turn. The run failed as `sandbox_disk_full` with the message "The runner restored the conversation after process loss, but the previous turn is no longer active." A manual Retry on a fresh provider session finished the work. The run started from the warm checkpoint of the previous run in the same conversation (`startup reuse: provider_session (reused)`, then `session.resumed` with the same provider session id). `loadWarmNativeCheckpoint` rebinds that checkpoint to the new run and clears its turns. `executeNativeSession` recovers it, so `recovered` is true. The run then starts its own task turn. `restoredInThisExecution` required `!recovered`, so that turn could not be continued. **Expected behavior** When a mid-run `runnerd` restore loses the task turn that this execution started, the execution sends one continuation turn in the restored conversation. This applies to a fresh session and to a resumed warm session. **Steps to reproduce** 1. Run a native agent through `paperclip_runner` in a managed sandbox, as a follow-up on an existing conversation (a warm resumed session). 2. During the turn, make `runnerd` exit, for example by filling the sandbox disk. 3. The transport relaunches `runnerd`, and the runner reports the turn as lost on restore. 4. The run fails instead of continuing the turn. ## What Changed - `packages/paperclip-runner/src/native-session-runtime.ts`: `restoredInThisExecution` no longer requires a fresh session. The turn id that this execution records when it starts its task turn is now enough. - The execution records that turn id only for a turn that carries the task work. A disposition-only recovery turn (the turn that asks only for `paperclip_finish` after a completed turn) is still not continued, as before. - All other guards stay: one continuation per run, the provider allowlist, no compaction or goal runs, no pending runtime requests, no saved semantic result, and the snapshot must show that same turn as restart-interrupted. The control-plane restart claim path (`restoredAtStartup`) does not change. - `native-session-runtime.test.ts`: three new cases for a recovered warm session: - it continues its own task turn once after a mid-run restore; - it does not continue a lost turn that it never started; - it does not continue a lost disposition-only recovery turn. ## Verification - Red before the fix: `npx vitest run src/native-session-runtime.test.ts` in `packages/paperclip-runner` failed `continues its own turn on a resumed warm session once when runnerd is restored mid-execution` with `NativeProviderTerminalFailure` (`code: native_provider_terminal_failed`, `providerCode: provider_turn_lost_on_restore`). This is the same failure as on staging. - Green after the fix: the same file has 303 passed, 1 failed. The one failure is `settles a governed wait through real Runner shutdown after a missing Codex terminal`. It needs the Rust `paperclip-runnerd` binary (`ENOENT runner/target/debug/paperclip-runnerd`). It fails the same way before and after the change. - The existing `requires the control plane restart claim` and `does not continue a lost turn this execution never started` cases still pass. - To prove that the disposition-only guard is necessary, I removed it locally. The new disposition-only case then failed (the run continued). I put the guard back. - `native-session-runtime.provider-turn.test.ts`: 36/36 passed. The Codex driver provider-turn, recovery, stale-generation and terminal-replay suites and `terminal-call-freshness.test.ts` passed. - Not run locally: `src/live/runnerd-restart-continuation.test.ts`. It needs the real Rust `runnerd` binary, and this sandbox has no Rust toolchain. CI builds the runner and runs these tests. - Live acceptance: after the automatic staging deploy, a chat follow-up whose `runnerd` restarts mid-turn must show one continuation turn in the same sandbox instead of a failed run. ## Risks - **Second executor.** The continuation starts only for the turn id that this execution received from its own `startTurn` call. A turn from an earlier process, or a turn that the execution adopted, is not this execution's turn. That path still needs the control-plane restart claim. The continuation runs at most once per run (`restartContinuationStarted`). - **Disk still full.** If the continuation fills the disk again, the second lost turn finds `restartContinuationStarted` set. The run then fails as `sandbox_disk_full`, and automatic retries stay off (#22239). The worst case is the current behavior. - **Disposition-only turns.** A recovered session that only asks for the disposition does not change. This change does not add a continuation for that turn. - Low risk otherwise. The change is 1 condition and 1 assignment in the runner runtime. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, Paperclip API and Sentry connectors). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (except one test that needs the Rust runner binary; see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no doc change is necessary: the runtime comment states the rule) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- .../src/native-session-runtime.test.ts | 80 +++++++++++++++++++ .../src/native-session-runtime.ts | 11 ++- 2 files changed, 88 insertions(+), 3 deletions(-) diff --git a/packages/paperclip-runner/src/native-session-runtime.test.ts b/packages/paperclip-runner/src/native-session-runtime.test.ts index 3b49eadb3ba..770afe9ba6e 100644 --- a/packages/paperclip-runner/src/native-session-runtime.test.ts +++ b/packages/paperclip-runner/src/native-session-runtime.test.ts @@ -769,6 +769,86 @@ describe("executeNativeSession recovery", () => { expect(events.filter(event => event.eventType === "run.terminal")).toHaveLength(1); }); + // A chat follow-up resumes the warm conversation of its previous run, so its + // session is recovered, and no restart claim exists. The turn it starts is + // still its own when a mid-run runnerd restore reports that turn lost. + it.each([ + { label: "continues its own turn on a resumed warm session once when runnerd is restored mid-execution", lostTurnId: "turn-own", dispositionOnly: false, continues: true }, + { label: "does not continue a lost turn a resumed warm session never started", lostTurnId: "turn-elsewhere", dispositionOnly: false, continues: false }, + { label: "does not continue a lost disposition-only recovery turn", lostTurnId: "turn-own", dispositionOnly: true, continues: false }, + ])("$label", async ({ lostTurnId, dispositionOnly, continues }) => { + const failure = { code: "provider_turn_lost_on_restore", recoverable: true }; + // A warm checkpoint rebound to a new run keeps only its provider identity + // (loadWarmNativeCheckpoint). A same-run recovery after a completed turn + // keeps that turn and asks only for the disposition. + const warm: PersistedNativeSession = { + backendKind: "runner", driverKind: "acpx_runtime", sessionId: "thread-recovery", identity, + providerSessionId: "provider-recovery", cursor: "0", activeTurnId: null, semanticResult: null, + pendingRuntimeRequests: [], lineage: [], + terminalTurns: dispositionOnly + ? [{ turnId: "turn-completed", fingerprint: JSON.stringify({ terminalState: "completed", error: null, result: null }) }] + : [], + }; + const interrupted: PersistedNativeSession = { + ...warm, + terminalTurns: [...warm.terminalTurns!, + { turnId: lostTurnId, fingerprint: JSON.stringify({ terminalState: "failed", error: failure, result: null }) }], + }; + let startedTurns = 0; + const startTurn = vi.fn(async () => { + startedTurns += 1; + return { turnId: startedTurns === 1 ? "turn-own" : "turn-continuation" }; + }); + // The runner reports the loss only after this execution submitted its turn. + const turnSubmitted = vi.waitFor(() => { expect(startedTurns).toBeGreaterThan(0); }); + const session: NativeSession = { + identity: () => identity, + async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; }, + async *events() { + await turnSubmitted; + yield { ...runnerEvent(1, "turn.failed", { status: "failed", error: failure }), turnId: lostTurnId }; + if (startTurn.mock.calls.length < 2) return; + yield { ...runnerEvent(2, "turn.completed", { status: "completed" }), turnId: "turn-continuation" }; + }, + startTurn, + async result() { return startTurn.mock.calls.length > 1 ? { result, terminal, turnId: "turn-continuation" } : null; }, + async snapshot() { return structuredClone(startTurn.mock.calls.length ? interrupted : warm); }, + async close() {}, + }; + const events: PrpEvent[] = []; + const backend: NativeSessionBackend = { + async descriptor() { return { kind: "runner", name: "warm-restore", version: "1", capabilities: await session.capabilities() }; }, + async openSession() { throw new Error("must resume the warm provider session"); }, + async recoverSession() { return { recovered: true, session }; }, + }; + const port: ControlPlanePort = { + async openRun() {}, + async checkpointSession() {}, + async appendEvent(event: PrpEvent) { events.push(event); return { cursor: events.length, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" }; }, + async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; }, + async completeRun() {}, + }; + const execution = executeNativeSession({ + input, backend, controlPlane: port, persistedSession: warm, + runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery", + }); + if (!continues) { + await expect(execution).rejects.toMatchObject({ code: "native_provider_terminal_failed" }); + expect(startTurn).toHaveBeenCalledOnce(); + return; + } + await expect(execution).resolves.toMatchObject({ result, turnId: "turn-continuation", terminal: { runTerminalState: "succeeded" } }); + expect(startTurn).toHaveBeenCalledTimes(2); + // The lost turn carried this run's task, not a disposition-only request. + const first = startTurn.mock.calls[0]![0] as Parameters[0]; + expect(first.message.text).toContain(JSON.stringify(input.task.prompt).slice(1, -1)); + const submitted = startTurn.mock.calls[1]![0] as Parameters[0]; + expect(submitted.continuation).toBe(true); + expectRestartContinuationGuidance(submitted.message.text); + expect(submitted.message.text).not.toContain(input.task.prompt); + expect(events.filter(event => event.eventType === "run.terminal")).toHaveLength(1); + }); + it.each([ { label: "after the continuation starts", staleFirst: false, staleTurnId: "turn-unrecorded-continuation" }, { label: "before the continuation starts", staleFirst: true, staleTurnId: "turn-unrecorded-continuation" }, diff --git a/packages/paperclip-runner/src/native-session-runtime.ts b/packages/paperclip-runner/src/native-session-runtime.ts index 795df4003b7..7fed5d55bf4 100644 --- a/packages/paperclip-runner/src/native-session-runtime.ts +++ b/packages/paperclip-runner/src/native-session-runtime.ts @@ -3103,7 +3103,9 @@ export async function executeNativeSession( if (await requestContinuationDisposition(event, signal)) return true; const restoredAtStartup = recovered && options.resumeInterruptedTurn === true && event.turnId === persistedSession?.activeTurnId; - const restoredInThisExecution = !recovered && ownTurnId !== null && event.turnId === ownTurnId; + // The task turn this execution started is its own, whether the session + // was opened fresh or resumed from a warm or recovered checkpoint. + const restoredInThisExecution = ownTurnId !== null && event.turnId === ownTurnId; if (restartContinuationStarted || !(restoredAtStartup || restoredInThisExecution) || // A compaction-only run never continues task work after a restart. // Runnerd records a restart-lost turn only for these providers, after @@ -3428,7 +3430,7 @@ export async function executeNativeSession( // the fenced checkpoint right before the submit. A new session's // normal path adds nothing before turn.start. if (recovered) await checkpoint(); - ({ turnId: ownTurnId } = await session.startTurn({ + const startedTaskTurn = await session.startTurn({ message: { role: "user", text: JSON.stringify(modelEnvelope) }, ...(recovered && modelEnvelope.schema === "paperclip.native-continuation.v1" ? { continuation: true as const } : {}), @@ -3438,7 +3440,10 @@ export async function executeNativeSession( input.provider.kind === "cursor_cloud" ? "default" : "executionMode" in input ? input.executionMode : "default", - })); + }); + // Only a turn carrying this run's task work can be continued after a + // mid-run runnerd restore. A disposition-only recovery turn cannot. + if (!dispositionOnlyRecovery || effectFreeInitialAcpxTurn) ownTurnId = startedTaskTurn.turnId; await checkpoint(); } settleStartupDecision();