mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
fix(runner): continue a lost turn on a resumed warm session (#22447)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents in a managed sandbox run under the Paperclip runner (`runnerd`). When `runnerd` exits mid-run, the transport relaunches it, and the runner restores the same conversation > - #22235 lets the execution that started a turn send one continuation turn when the restored runner reports that turn as `provider_turn_lost_on_restore` > - That rule applies only to a session that the execution opened fresh (`!recovered`). A run that resumes the warm conversation of its previous run is a recovered session. A chat follow-up does this on every new message > - So a resumed session that loses its own task turn gets no continuation. The run fails, and the user must press Retry > - This pull request makes the task turn that this execution started its own turn, whether the session was opened fresh or resumed > - The benefit is that a chat follow-up keeps its conversation and workspace after a mid-run `runnerd` restore, as #22235 intended ## Linked Issues or Issue Description Refs #22235 (continue a turn lost to a mid-run runnerd restore). Refs #22068 (disk reserve). Refs #22239 (stop retrying a run that filled its sandbox disk). **What happened?** On staging, an agent in a chat task filled its 10 GiB managed sandbox disk. It fetched the full history of a large Git repository and searched it. `runnerd` exited with `failed to create private durable state temporary file: No space left on device (os error 28)`. The disk reserve let the transport relaunch `runnerd`, and the runner restored the same conversation. Then the runner reported `turn.failed {code: provider_turn_lost_on_restore, recoverable: true}` for the turn that this run had started. The runtime did not send the continuation turn. The run failed as `sandbox_disk_full` with the message "The runner restored the conversation after process loss, but the previous turn is no longer active." A manual Retry on a fresh provider session finished the work. The run started from the warm checkpoint of the previous run in the same conversation (`startup reuse: provider_session (reused)`, then `session.resumed` with the same provider session id). `loadWarmNativeCheckpoint` rebinds that checkpoint to the new run and clears its turns. `executeNativeSession` recovers it, so `recovered` is true. The run then starts its own task turn. `restoredInThisExecution` required `!recovered`, so that turn could not be continued. **Expected behavior** When a mid-run `runnerd` restore loses the task turn that this execution started, the execution sends one continuation turn in the restored conversation. This applies to a fresh session and to a resumed warm session. **Steps to reproduce** 1. Run a native agent through `paperclip_runner` in a managed sandbox, as a follow-up on an existing conversation (a warm resumed session). 2. During the turn, make `runnerd` exit, for example by filling the sandbox disk. 3. The transport relaunches `runnerd`, and the runner reports the turn as lost on restore. 4. The run fails instead of continuing the turn. ## What Changed - `packages/paperclip-runner/src/native-session-runtime.ts`: `restoredInThisExecution` no longer requires a fresh session. The turn id that this execution records when it starts its task turn is now enough. - The execution records that turn id only for a turn that carries the task work. A disposition-only recovery turn (the turn that asks only for `paperclip_finish` after a completed turn) is still not continued, as before. - All other guards stay: one continuation per run, the provider allowlist, no compaction or goal runs, no pending runtime requests, no saved semantic result, and the snapshot must show that same turn as restart-interrupted. The control-plane restart claim path (`restoredAtStartup`) does not change. - `native-session-runtime.test.ts`: three new cases for a recovered warm session: - it continues its own task turn once after a mid-run restore; - it does not continue a lost turn that it never started; - it does not continue a lost disposition-only recovery turn. ## Verification - Red before the fix: `npx vitest run src/native-session-runtime.test.ts` in `packages/paperclip-runner` failed `continues its own turn on a resumed warm session once when runnerd is restored mid-execution` with `NativeProviderTerminalFailure` (`code: native_provider_terminal_failed`, `providerCode: provider_turn_lost_on_restore`). This is the same failure as on staging. - Green after the fix: the same file has 303 passed, 1 failed. The one failure is `settles a governed wait through real Runner shutdown after a missing Codex terminal`. It needs the Rust `paperclip-runnerd` binary (`ENOENT runner/target/debug/paperclip-runnerd`). It fails the same way before and after the change. - The existing `requires the control plane restart claim` and `does not continue a lost turn this execution never started` cases still pass. - To prove that the disposition-only guard is necessary, I removed it locally. The new disposition-only case then failed (the run continued). I put the guard back. - `native-session-runtime.provider-turn.test.ts`: 36/36 passed. The Codex driver provider-turn, recovery, stale-generation and terminal-replay suites and `terminal-call-freshness.test.ts` passed. - Not run locally: `src/live/runnerd-restart-continuation.test.ts`. It needs the real Rust `runnerd` binary, and this sandbox has no Rust toolchain. CI builds the runner and runs these tests. - Live acceptance: after the automatic staging deploy, a chat follow-up whose `runnerd` restarts mid-turn must show one continuation turn in the same sandbox instead of a failed run. ## Risks - **Second executor.** The continuation starts only for the turn id that this execution received from its own `startTurn` call. A turn from an earlier process, or a turn that the execution adopted, is not this execution's turn. That path still needs the control-plane restart claim. The continuation runs at most once per run (`restartContinuationStarted`). - **Disk still full.** If the continuation fills the disk again, the second lost turn finds `restartContinuationStarted` set. The run then fails as `sandbox_disk_full`, and automatic retries stay off (#22239). The worst case is the current behavior. - **Disposition-only turns.** A recovered session that only asks for the disposition does not change. This change does not add a continuation for that turn. - Low risk otherwise. The change is 1 condition and 1 assignment in the runner runtime. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, Paperclip API and Sentry connectors). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (except one test that needs the Rust runner binary; see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no doc change is necessary: the runtime comment states the rule) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2 files changed
+88
-3
No files matched your search
@@ -769,6 +769,86 @@ describe("executeNativeSession recovery", () => {
|
||||
expect(events.filter(event => event.eventType === "run.terminal")).toHaveLength(1);
|
||||
});
|
||||
|
||||
// A chat follow-up resumes the warm conversation of its previous run, so its
|
||||
// session is recovered, and no restart claim exists. The turn it starts is
|
||||
// still its own when a mid-run runnerd restore reports that turn lost.
|
||||
it.each([
|
||||
{ label: "continues its own turn on a resumed warm session once when runnerd is restored mid-execution", lostTurnId: "turn-own", dispositionOnly: false, continues: true },
|
||||
{ label: "does not continue a lost turn a resumed warm session never started", lostTurnId: "turn-elsewhere", dispositionOnly: false, continues: false },
|
||||
{ label: "does not continue a lost disposition-only recovery turn", lostTurnId: "turn-own", dispositionOnly: true, continues: false },
|
||||
])("$label", async ({ lostTurnId, dispositionOnly, continues }) => {
|
||||
const failure = { code: "provider_turn_lost_on_restore", recoverable: true };
|
||||
// A warm checkpoint rebound to a new run keeps only its provider identity
|
||||
// (loadWarmNativeCheckpoint). A same-run recovery after a completed turn
|
||||
// keeps that turn and asks only for the disposition.
|
||||
const warm: PersistedNativeSession = {
|
||||
backendKind: "runner", driverKind: "acpx_runtime", sessionId: "thread-recovery", identity,
|
||||
providerSessionId: "provider-recovery", cursor: "0", activeTurnId: null, semanticResult: null,
|
||||
pendingRuntimeRequests: [], lineage: [],
|
||||
terminalTurns: dispositionOnly
|
||||
? [{ turnId: "turn-completed", fingerprint: JSON.stringify({ terminalState: "completed", error: null, result: null }) }]
|
||||
: [],
|
||||
};
|
||||
const interrupted: PersistedNativeSession = {
|
||||
...warm,
|
||||
terminalTurns: [...warm.terminalTurns!,
|
||||
{ turnId: lostTurnId, fingerprint: JSON.stringify({ terminalState: "failed", error: failure, result: null }) }],
|
||||
};
|
||||
let startedTurns = 0;
|
||||
const startTurn = vi.fn<NativeSession["startTurn"]>(async () => {
|
||||
startedTurns += 1;
|
||||
return { turnId: startedTurns === 1 ? "turn-own" : "turn-continuation" };
|
||||
});
|
||||
// The runner reports the loss only after this execution submitted its turn.
|
||||
const turnSubmitted = vi.waitFor(() => { expect(startedTurns).toBeGreaterThan(0); });
|
||||
const session: NativeSession = {
|
||||
identity: () => identity,
|
||||
async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; },
|
||||
async *events() {
|
||||
await turnSubmitted;
|
||||
yield { ...runnerEvent(1, "turn.failed", { status: "failed", error: failure }), turnId: lostTurnId };
|
||||
if (startTurn.mock.calls.length < 2) return;
|
||||
yield { ...runnerEvent(2, "turn.completed", { status: "completed" }), turnId: "turn-continuation" };
|
||||
},
|
||||
startTurn,
|
||||
async result() { return startTurn.mock.calls.length > 1 ? { result, terminal, turnId: "turn-continuation" } : null; },
|
||||
async snapshot() { return structuredClone(startTurn.mock.calls.length ? interrupted : warm); },
|
||||
async close() {},
|
||||
};
|
||||
const events: PrpEvent[] = [];
|
||||
const backend: NativeSessionBackend = {
|
||||
async descriptor() { return { kind: "runner", name: "warm-restore", version: "1", capabilities: await session.capabilities() }; },
|
||||
async openSession() { throw new Error("must resume the warm provider session"); },
|
||||
async recoverSession() { return { recovered: true, session }; },
|
||||
};
|
||||
const port: ControlPlanePort = {
|
||||
async openRun() {},
|
||||
async checkpointSession() {},
|
||||
async appendEvent(event: PrpEvent) { events.push(event); return { cursor: events.length, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" }; },
|
||||
async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; },
|
||||
async completeRun() {},
|
||||
};
|
||||
const execution = executeNativeSession({
|
||||
input, backend, controlPlane: port, persistedSession: warm,
|
||||
runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery",
|
||||
});
|
||||
if (!continues) {
|
||||
await expect(execution).rejects.toMatchObject({ code: "native_provider_terminal_failed" });
|
||||
expect(startTurn).toHaveBeenCalledOnce();
|
||||
return;
|
||||
}
|
||||
await expect(execution).resolves.toMatchObject({ result, turnId: "turn-continuation", terminal: { runTerminalState: "succeeded" } });
|
||||
expect(startTurn).toHaveBeenCalledTimes(2);
|
||||
// The lost turn carried this run's task, not a disposition-only request.
|
||||
const first = startTurn.mock.calls[0]![0] as Parameters<NativeSession["startTurn"]>[0];
|
||||
expect(first.message.text).toContain(JSON.stringify(input.task.prompt).slice(1, -1));
|
||||
const submitted = startTurn.mock.calls[1]![0] as Parameters<NativeSession["startTurn"]>[0];
|
||||
expect(submitted.continuation).toBe(true);
|
||||
expectRestartContinuationGuidance(submitted.message.text);
|
||||
expect(submitted.message.text).not.toContain(input.task.prompt);
|
||||
expect(events.filter(event => event.eventType === "run.terminal")).toHaveLength(1);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ label: "after the continuation starts", staleFirst: false, staleTurnId: "turn-unrecorded-continuation" },
|
||||
{ label: "before the continuation starts", staleFirst: true, staleTurnId: "turn-unrecorded-continuation" },
|
||||
|
||||
@@ -3103,7 +3103,9 @@ export async function executeNativeSession(
|
||||
if (await requestContinuationDisposition(event, signal)) return true;
|
||||
const restoredAtStartup = recovered && options.resumeInterruptedTurn === true &&
|
||||
event.turnId === persistedSession?.activeTurnId;
|
||||
const restoredInThisExecution = !recovered && ownTurnId !== null && event.turnId === ownTurnId;
|
||||
// The task turn this execution started is its own, whether the session
|
||||
// was opened fresh or resumed from a warm or recovered checkpoint.
|
||||
const restoredInThisExecution = ownTurnId !== null && event.turnId === ownTurnId;
|
||||
if (restartContinuationStarted || !(restoredAtStartup || restoredInThisExecution) ||
|
||||
// A compaction-only run never continues task work after a restart.
|
||||
// Runnerd records a restart-lost turn only for these providers, after
|
||||
@@ -3428,7 +3430,7 @@ export async function executeNativeSession(
|
||||
// the fenced checkpoint right before the submit. A new session's
|
||||
// normal path adds nothing before turn.start.
|
||||
if (recovered) await checkpoint();
|
||||
({ turnId: ownTurnId } = await session.startTurn({
|
||||
const startedTaskTurn = await session.startTurn({
|
||||
message: { role: "user", text: JSON.stringify(modelEnvelope) },
|
||||
...(recovered && modelEnvelope.schema === "paperclip.native-continuation.v1"
|
||||
? { continuation: true as const } : {}),
|
||||
@@ -3438,7 +3440,10 @@ export async function executeNativeSession(
|
||||
input.provider.kind === "cursor_cloud"
|
||||
? "default"
|
||||
: "executionMode" in input ? input.executionMode : "default",
|
||||
}));
|
||||
});
|
||||
// Only a turn carrying this run's task work can be continued after a
|
||||
// mid-run runnerd restore. A disposition-only recovery turn cannot.
|
||||
if (!dispositionOnlyRecovery || effectFreeInitialAcpxTurn) ownTurnId = startedTaskTurn.turnId;
|
||||
await checkpoint();
|
||||
}
|
||||
settleStartupDecision();
|
||||
|
||||
Reference in new issue
Block a user