mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents in a managed sandbox run under the Paperclip runner (`runnerd`). When `runnerd` exits mid-run, the transport relaunches it, and the runner restores the same conversation > - #22235 lets the execution that started a turn send one continuation turn when the restored runner reports that turn as `provider_turn_lost_on_restore` > - That rule applies only to a session that the execution opened fresh (`!recovered`). A run that resumes the warm conversation of its previous run is a recovered session. A chat follow-up does this on every new message > - So a resumed session that loses its own task turn gets no continuation. The run fails, and the user must press Retry > - This pull request makes the task turn that this execution started its own turn, whether the session was opened fresh or resumed > - The benefit is that a chat follow-up keeps its conversation and workspace after a mid-run `runnerd` restore, as #22235 intended ## Linked Issues or Issue Description Refs #22235 (continue a turn lost to a mid-run runnerd restore). Refs #22068 (disk reserve). Refs #22239 (stop retrying a run that filled its sandbox disk). **What happened?** On staging, an agent in a chat task filled its 10 GiB managed sandbox disk. It fetched the full history of a large Git repository and searched it. `runnerd` exited with `failed to create private durable state temporary file: No space left on device (os error 28)`. The disk reserve let the transport relaunch `runnerd`, and the runner restored the same conversation. Then the runner reported `turn.failed {code: provider_turn_lost_on_restore, recoverable: true}` for the turn that this run had started. The runtime did not send the continuation turn. The run failed as `sandbox_disk_full` with the message "The runner restored the conversation after process loss, but the previous turn is no longer active." A manual Retry on a fresh provider session finished the work. The run started from the warm checkpoint of the previous run in the same conversation (`startup reuse: provider_session (reused)`, then `session.resumed` with the same provider session id). `loadWarmNativeCheckpoint` rebinds that checkpoint to the new run and clears its turns. `executeNativeSession` recovers it, so `recovered` is true. The run then starts its own task turn. `restoredInThisExecution` required `!recovered`, so that turn could not be continued. **Expected behavior** When a mid-run `runnerd` restore loses the task turn that this execution started, the execution sends one continuation turn in the restored conversation. This applies to a fresh session and to a resumed warm session. **Steps to reproduce** 1. Run a native agent through `paperclip_runner` in a managed sandbox, as a follow-up on an existing conversation (a warm resumed session). 2. During the turn, make `runnerd` exit, for example by filling the sandbox disk. 3. The transport relaunches `runnerd`, and the runner reports the turn as lost on restore. 4. The run fails instead of continuing the turn. ## What Changed - `packages/paperclip-runner/src/native-session-runtime.ts`: `restoredInThisExecution` no longer requires a fresh session. The turn id that this execution records when it starts its task turn is now enough. - The execution records that turn id only for a turn that carries the task work. A disposition-only recovery turn (the turn that asks only for `paperclip_finish` after a completed turn) is still not continued, as before. - All other guards stay: one continuation per run, the provider allowlist, no compaction or goal runs, no pending runtime requests, no saved semantic result, and the snapshot must show that same turn as restart-interrupted. The control-plane restart claim path (`restoredAtStartup`) does not change. - `native-session-runtime.test.ts`: three new cases for a recovered warm session: - it continues its own task turn once after a mid-run restore; - it does not continue a lost turn that it never started; - it does not continue a lost disposition-only recovery turn. ## Verification - Red before the fix: `npx vitest run src/native-session-runtime.test.ts` in `packages/paperclip-runner` failed `continues its own turn on a resumed warm session once when runnerd is restored mid-execution` with `NativeProviderTerminalFailure` (`code: native_provider_terminal_failed`, `providerCode: provider_turn_lost_on_restore`). This is the same failure as on staging. - Green after the fix: the same file has 303 passed, 1 failed. The one failure is `settles a governed wait through real Runner shutdown after a missing Codex terminal`. It needs the Rust `paperclip-runnerd` binary (`ENOENT runner/target/debug/paperclip-runnerd`). It fails the same way before and after the change. - The existing `requires the control plane restart claim` and `does not continue a lost turn this execution never started` cases still pass. - To prove that the disposition-only guard is necessary, I removed it locally. The new disposition-only case then failed (the run continued). I put the guard back. - `native-session-runtime.provider-turn.test.ts`: 36/36 passed. The Codex driver provider-turn, recovery, stale-generation and terminal-replay suites and `terminal-call-freshness.test.ts` passed. - Not run locally: `src/live/runnerd-restart-continuation.test.ts`. It needs the real Rust `runnerd` binary, and this sandbox has no Rust toolchain. CI builds the runner and runs these tests. - Live acceptance: after the automatic staging deploy, a chat follow-up whose `runnerd` restarts mid-turn must show one continuation turn in the same sandbox instead of a failed run. ## Risks - **Second executor.** The continuation starts only for the turn id that this execution received from its own `startTurn` call. A turn from an earlier process, or a turn that the execution adopted, is not this execution's turn. That path still needs the control-plane restart claim. The continuation runs at most once per run (`restartContinuationStarted`). - **Disk still full.** If the continuation fills the disk again, the second lost turn finds `restartContinuationStarted` set. The run then fails as `sandbox_disk_full`, and automatic retries stay off (#22239). The worst case is the current behavior. - **Disposition-only turns.** A recovered session that only asks for the disposition does not change. This change does not add a continuation for that turn. - Low risk otherwise. The change is 1 condition and 1 assignment in the runner runtime. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, Paperclip API and Sentry connectors). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (except one test that needs the Rust runner binary; see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no doc change is necessary: the runtime comment states the rule) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>