fix(runner): continue a lost turn on a resumed warm session (#22447)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a managed sandbox run under the Paperclip runner
(`runnerd`). When `runnerd` exits mid-run, the transport relaunches it,
and the runner restores the same conversation
> - #22235 lets the execution that started a turn send one continuation
turn when the restored runner reports that turn as
`provider_turn_lost_on_restore`
> - That rule applies only to a session that the execution opened fresh
(`!recovered`). A run that resumes the warm conversation of its previous
run is a recovered session. A chat follow-up does this on every new
message
> - So a resumed session that loses its own task turn gets no
continuation. The run fails, and the user must press Retry
> - This pull request makes the task turn that this execution started
its own turn, whether the session was opened fresh or resumed
> - The benefit is that a chat follow-up keeps its conversation and
workspace after a mid-run `runnerd` restore, as #22235 intended

## Linked Issues or Issue Description

Refs #22235 (continue a turn lost to a mid-run runnerd restore). Refs
#22068 (disk reserve). Refs #22239 (stop retrying a run that filled its
sandbox disk).

**What happened?**
On staging, an agent in a chat task filled its 10 GiB managed sandbox
disk. It fetched the full history of a large Git repository and searched
it. `runnerd` exited with `failed to create private durable state
temporary file: No space left on device (os error 28)`. The disk reserve
let the transport relaunch `runnerd`, and the runner restored the same
conversation. Then the runner reported `turn.failed {code:
provider_turn_lost_on_restore, recoverable: true}` for the turn that
this run had started. The runtime did not send the continuation turn.
The run failed as `sandbox_disk_full` with the message "The runner
restored the conversation after process loss, but the previous turn is
no longer active." A manual Retry on a fresh provider session finished
the work.

The run started from the warm checkpoint of the previous run in the same
conversation (`startup reuse: provider_session (reused)`, then
`session.resumed` with the same provider session id).
`loadWarmNativeCheckpoint` rebinds that checkpoint to the new run and
clears its turns. `executeNativeSession` recovers it, so `recovered` is
true. The run then starts its own task turn. `restoredInThisExecution`
required `!recovered`, so that turn could not be continued.

**Expected behavior**
When a mid-run `runnerd` restore loses the task turn that this execution
started, the execution sends one continuation turn in the restored
conversation. This applies to a fresh session and to a resumed warm
session.

**Steps to reproduce**
1. Run a native agent through `paperclip_runner` in a managed sandbox,
as a follow-up on an existing conversation (a warm resumed session).
2. During the turn, make `runnerd` exit, for example by filling the
sandbox disk.
3. The transport relaunches `runnerd`, and the runner reports the turn
as lost on restore.
4. The run fails instead of continuing the turn.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`restoredInThisExecution` no longer requires a fresh session. The turn
id that this execution records when it starts its task turn is now
enough.
- The execution records that turn id only for a turn that carries the
task work. A disposition-only recovery turn (the turn that asks only for
`paperclip_finish` after a completed turn) is still not continued, as
before.
- All other guards stay: one continuation per run, the provider
allowlist, no compaction or goal runs, no pending runtime requests, no
saved semantic result, and the snapshot must show that same turn as
restart-interrupted. The control-plane restart claim path
(`restoredAtStartup`) does not change.
- `native-session-runtime.test.ts`: three new cases for a recovered warm
session:
  - it continues its own task turn once after a mid-run restore;
  - it does not continue a lost turn that it never started;
  - it does not continue a lost disposition-only recovery turn.

## Verification

- Red before the fix: `npx vitest run
src/native-session-runtime.test.ts` in `packages/paperclip-runner`
failed `continues its own turn on a resumed warm session once when
runnerd is restored mid-execution` with `NativeProviderTerminalFailure`
(`code: native_provider_terminal_failed`, `providerCode:
provider_turn_lost_on_restore`). This is the same failure as on staging.
- Green after the fix: the same file has 303 passed, 1 failed. The one
failure is `settles a governed wait through real Runner shutdown after a
missing Codex terminal`. It needs the Rust `paperclip-runnerd` binary
(`ENOENT runner/target/debug/paperclip-runnerd`). It fails the same way
before and after the change.
- The existing `requires the control plane restart claim` and `does not
continue a lost turn this execution never started` cases still pass.
- To prove that the disposition-only guard is necessary, I removed it
locally. The new disposition-only case then failed (the run continued).
I put the guard back.
- `native-session-runtime.provider-turn.test.ts`: 36/36 passed. The
Codex driver provider-turn, recovery, stale-generation and
terminal-replay suites and `terminal-call-freshness.test.ts` passed.
- Not run locally: `src/live/runnerd-restart-continuation.test.ts`. It
needs the real Rust `runnerd` binary, and this sandbox has no Rust
toolchain. CI builds the runner and runs these tests.
- Live acceptance: after the automatic staging deploy, a chat follow-up
whose `runnerd` restarts mid-turn must show one continuation turn in the
same sandbox instead of a failed run.

## Risks

- **Second executor.** The continuation starts only for the turn id that
this execution received from its own `startTurn` call. A turn from an
earlier process, or a turn that the execution adopted, is not this
execution's turn. That path still needs the control-plane restart claim.
The continuation runs at most once per run
(`restartContinuationStarted`).
- **Disk still full.** If the continuation fills the disk again, the
second lost turn finds `restartContinuationStarted` set. The run then
fails as `sandbox_disk_full`, and automatic retries stay off (#22239).
The worst case is the current behavior.
- **Disposition-only turns.** A recovered session that only asks for the
disposition does not change. This change does not add a continuation for
that turn.
- Low risk otherwise. The change is 1 condition and 1 assignment in the
runner runtime.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, Paperclip API and Sentry connectors).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (except one test that needs
the Rust runner binary; see Verification)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
doc change is necessary: the runtime comment states the rule)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-20 16:52:11 -08:00
1 parent b6a904554d
commit 611dc68581
2 files changed
+88 -3

No files matched your search

@@ -769,6 +769,86 @@ describe("executeNativeSession recovery", () => {
expect(events.filter(event => event.eventType === "run.terminal")).toHaveLength(1);
});
// A chat follow-up resumes the warm conversation of its previous run, so its
// session is recovered, and no restart claim exists. The turn it starts is
// still its own when a mid-run runnerd restore reports that turn lost.
it.each([
{ label: "continues its own turn on a resumed warm session once when runnerd is restored mid-execution", lostTurnId: "turn-own", dispositionOnly: false, continues: true },
{ label: "does not continue a lost turn a resumed warm session never started", lostTurnId: "turn-elsewhere", dispositionOnly: false, continues: false },
{ label: "does not continue a lost disposition-only recovery turn", lostTurnId: "turn-own", dispositionOnly: true, continues: false },
])("$label", async ({ lostTurnId, dispositionOnly, continues }) => {
const failure = { code: "provider_turn_lost_on_restore", recoverable: true };
// A warm checkpoint rebound to a new run keeps only its provider identity
// (loadWarmNativeCheckpoint). A same-run recovery after a completed turn
// keeps that turn and asks only for the disposition.
const warm: PersistedNativeSession = {
backendKind: "runner", driverKind: "acpx_runtime", sessionId: "thread-recovery", identity,
providerSessionId: "provider-recovery", cursor: "0", activeTurnId: null, semanticResult: null,
pendingRuntimeRequests: [], lineage: [],
terminalTurns: dispositionOnly
? [{ turnId: "turn-completed", fingerprint: JSON.stringify({ terminalState: "completed", error: null, result: null }) }]
: [],
};
const interrupted: PersistedNativeSession = {
...warm,
terminalTurns: [...warm.terminalTurns!,
{ turnId: lostTurnId, fingerprint: JSON.stringify({ terminalState: "failed", error: failure, result: null }) }],
};
let startedTurns = 0;
const startTurn = vi.fn<NativeSession["startTurn"]>(async () => {
startedTurns += 1;
return { turnId: startedTurns === 1 ? "turn-own" : "turn-continuation" };
});
// The runner reports the loss only after this execution submitted its turn.
const turnSubmitted = vi.waitFor(() => { expect(startedTurns).toBeGreaterThan(0); });
const session: NativeSession = {
identity: () => identity,
async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; },
async *events() {
await turnSubmitted;
yield { ...runnerEvent(1, "turn.failed", { status: "failed", error: failure }), turnId: lostTurnId };
if (startTurn.mock.calls.length < 2) return;
yield { ...runnerEvent(2, "turn.completed", { status: "completed" }), turnId: "turn-continuation" };
},
startTurn,
async result() { return startTurn.mock.calls.length > 1 ? { result, terminal, turnId: "turn-continuation" } : null; },
async snapshot() { return structuredClone(startTurn.mock.calls.length ? interrupted : warm); },
async close() {},
};
const events: PrpEvent[] = [];
const backend: NativeSessionBackend = {
async descriptor() { return { kind: "runner", name: "warm-restore", version: "1", capabilities: await session.capabilities() }; },
async openSession() { throw new Error("must resume the warm provider session"); },
async recoverSession() { return { recovered: true, session }; },
};
const port: ControlPlanePort = {
async openRun() {},
async checkpointSession() {},
async appendEvent(event: PrpEvent) { events.push(event); return { cursor: events.length, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" }; },
async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; },
async completeRun() {},
};
const execution = executeNativeSession({
input, backend, controlPlane: port, persistedSession: warm,
runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery",
});
if (!continues) {
await expect(execution).rejects.toMatchObject({ code: "native_provider_terminal_failed" });
expect(startTurn).toHaveBeenCalledOnce();
return;
}
await expect(execution).resolves.toMatchObject({ result, turnId: "turn-continuation", terminal: { runTerminalState: "succeeded" } });
expect(startTurn).toHaveBeenCalledTimes(2);
// The lost turn carried this run's task, not a disposition-only request.
const first = startTurn.mock.calls[0]![0] as Parameters<NativeSession["startTurn"]>[0];
expect(first.message.text).toContain(JSON.stringify(input.task.prompt).slice(1, -1));
const submitted = startTurn.mock.calls[1]![0] as Parameters<NativeSession["startTurn"]>[0];
expect(submitted.continuation).toBe(true);
expectRestartContinuationGuidance(submitted.message.text);
expect(submitted.message.text).not.toContain(input.task.prompt);
expect(events.filter(event => event.eventType === "run.terminal")).toHaveLength(1);
});
it.each([
{ label: "after the continuation starts", staleFirst: false, staleTurnId: "turn-unrecorded-continuation" },
{ label: "before the continuation starts", staleFirst: true, staleTurnId: "turn-unrecorded-continuation" },
@@ -3103,7 +3103,9 @@ export async function executeNativeSession(
if (await requestContinuationDisposition(event, signal)) return true;
const restoredAtStartup = recovered && options.resumeInterruptedTurn === true &&
event.turnId === persistedSession?.activeTurnId;
const restoredInThisExecution = !recovered && ownTurnId !== null && event.turnId === ownTurnId;
// The task turn this execution started is its own, whether the session
// was opened fresh or resumed from a warm or recovered checkpoint.
const restoredInThisExecution = ownTurnId !== null && event.turnId === ownTurnId;
if (restartContinuationStarted || !(restoredAtStartup || restoredInThisExecution) ||
// A compaction-only run never continues task work after a restart.
// Runnerd records a restart-lost turn only for these providers, after
@@ -3428,7 +3430,7 @@ export async function executeNativeSession(
// the fenced checkpoint right before the submit. A new session's
// normal path adds nothing before turn.start.
if (recovered) await checkpoint();
({ turnId: ownTurnId } = await session.startTurn({
const startedTaskTurn = await session.startTurn({
message: { role: "user", text: JSON.stringify(modelEnvelope) },
...(recovered && modelEnvelope.schema === "paperclip.native-continuation.v1"
? { continuation: true as const } : {}),
@@ -3438,7 +3440,10 @@ export async function executeNativeSession(
input.provider.kind === "cursor_cloud"
? "default"
: "executionMode" in input ? input.executionMode : "default",
}));
});
// Only a turn carrying this run's task work can be continued after a
// mid-run runnerd restore. A disposition-only recovery turn cannot.
if (!dispositionOnlyRecovery || effectFreeInitialAcpxTurn) ownTurnId = startedTaskTurn.turnId;
await checkpoint();
}
settleStartupDecision();