mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 07:42:38 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents in a managed sandbox run under the Paperclip runner (`runnerd`). When `runnerd` exits mid-run, the transport relaunches it in the same execution, and the runner restores the same conversation > - An agent can fill the sandbox disk with its own work. `runnerd` then cannot write its state and exits. #22068 added a disk reserve so the relaunch succeeds. Its stated goal was that "the agent's turn can continue and see the full disk" > - The relaunch now works, but the turn still dies. The runner reports the in-flight turn as `provider_turn_lost_on_restore`. The runtime sends its single same-conversation continuation only for a session recovered at startup under a control-plane restart claim. A relaunch inside the live execution has neither, so the run fails as `native_provider_terminal_failed` > - The generic transient retry then starts on a fresh sandbox, redoes the same install, fills the disk again, and ends in board-owned "Recovery needed" > - This pull request lets the execution that started a turn send the existing one-time continuation when the runner reports that same turn lost on restore > - The benefit is that the agent keeps its conversation and workspace and sees the full disk in its own tools, instead of the run dying and retrying into the same wall ## Linked Issues or Issue Description Refs #22068 **What happened?** On staging, two agent tasks on the Paperclip monorepo each failed on every attempt (5 failed runs in total). Each run used a fresh managed Daytona sandbox with 10 GiB of disk. The agent's own `df` showed `/` at 10G of 10G with 236K free. `node_modules` used 3.2G and `.paperclip-runtime` used 6.1G: about 3.0G of provider pack and a 2.8G pnpm store under the session data directory. `runnerd` exited with `failed to create private durable state temporary file: No space left on device (os error 28)`. The transport relaunched it in about 20 s, and runnerd restored its durable session. Then the runner reported `turn.failed {code: provider_turn_lost_on_restore, recoverable: true}`, and the run failed as `native_provider_terminal_failed`. Retries on fresh sandboxes hit the same failure. The tasks ended in "Recovery needed" with `provider_failure_meaning_unverified`. **Steps to reproduce** 1. Run an ACPX or Codex agent through `paperclip_runner` in a managed sandbox. 2. Have the agent fill the disk during a turn, or end `runnerd` in any other way mid-turn while the server keeps running. 3. The transport relaunches `runnerd`, and the runner reports the turn as lost on restore. 4. The run fails instead of continuing the turn. **Expected behavior** The execution that started the turn sends one continuation turn in the restored conversation, exactly as it does after a server restart. The agent continues from its recorded progress and sees the full disk in its tools. **Environment** Managed Daytona sandbox (fleet snapshot, 4 CPU, 8 GiB memory, 10 GiB disk), `paperclip_runner` native runtime with Claude through ACPX. ## What Changed - `packages/paperclip-runner/src/native-session-runtime.ts`: the execution records the turn id of the turn it starts. `resumeInterruptedTurn` now accepts two owners of a turn lost on restore: - a session recovered at startup with the control-plane restart claim, for the checkpointed active turn (unchanged), or - a session this execution opened itself, for the turn this execution started (new). - All other guards stay: one continuation per run, the provider allowlist, no compaction or goal runs, no pending runtime requests, and the snapshot must show that same turn as restart-interrupted. - `native-session-runtime.test.ts`: a new regression for an ACPX session that loses its own turn to a mid-execution `runnerd` restore, plus a negative case for a lost turn this execution never started. ## Verification - Red before the fix: `npx vitest run src/native-session-runtime.test.ts` failed `continues its own turn once when runnerd is restored mid-execution` with `NativeProviderTerminalFailure: Provider turn failed`, the same failure seen on staging. - Green after the fix: `native-session-runtime.test.ts` and `native-session-runtime.provider-turn.test.ts` pass (337/337). The existing `requires the control plane restart claim` case still refuses a recovered session without the claim. - `pnpm verify` (affected selection): see the CI results on this PR. - Live acceptance: after deploy, retry the affected staging tasks and confirm that a mid-turn `runnerd` restore leads to one continuation turn in the same sandbox instead of `native_provider_terminal_failed`. ## Risks - **Second executor.** The continuation can start only from the execution that opened the session and started the turn. It is not used for a recovered or adopted session without a claim. It runs at most once per run (`restartContinuationStarted`). The runner emits this code only after it proves the old provider process ended. - **Run starts.** This starts no new runs. It adds at most one turn to a run that would otherwise fail and schedule a fresh-sandbox retry. During a crash loop, `MAX_UNSTABLE_RUNNER_RESTORES` in the transport still fails the transport. - **Disk still full.** If the continuation fills the disk again, the second lost turn finds `restartContinuationStarted` set, and the run fails as it did before. The worst case is the current behavior. - **Not in this PR.** Paperclip's own footprint (about 3G of provider pack per sandbox, and a per-session pnpm store) and classifying a disk-full failure so it does not retry into the same wall. Both are follow-ups. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser, and a code-tracing subagent). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>