Files
PaperClipAI/packages
DottaandClaude Opus 5.5 06750cb18e fix(runner): continue a turn lost to a mid-run runnerd restore (#22235)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a managed sandbox run under the Paperclip runner
(`runnerd`). When `runnerd` exits mid-run, the transport relaunches it
in the same execution, and the runner restores the same conversation
> - An agent can fill the sandbox disk with its own work. `runnerd` then
cannot write its state and exits. #22068 added a disk reserve so the
relaunch succeeds. Its stated goal was that "the agent's turn can
continue and see the full disk"
> - The relaunch now works, but the turn still dies. The runner reports
the in-flight turn as `provider_turn_lost_on_restore`. The runtime sends
its single same-conversation continuation only for a session recovered
at startup under a control-plane restart claim. A relaunch inside the
live execution has neither, so the run fails as
`native_provider_terminal_failed`
> - The generic transient retry then starts on a fresh sandbox, redoes
the same install, fills the disk again, and ends in board-owned
"Recovery needed"
> - This pull request lets the execution that started a turn send the
existing one-time continuation when the runner reports that same turn
lost on restore
> - The benefit is that the agent keeps its conversation and workspace
and sees the full disk in its own tools, instead of the run dying and
retrying into the same wall

## Linked Issues or Issue Description

Refs #22068

**What happened?**
On staging, two agent tasks on the Paperclip monorepo each failed on
every attempt (5 failed runs in total). Each run used a fresh managed
Daytona sandbox with 10 GiB of disk. The agent's own `df` showed `/` at
10G of 10G with 236K free. `node_modules` used 3.2G and
`.paperclip-runtime` used 6.1G: about 3.0G of provider pack and a 2.8G
pnpm store under the session data directory. `runnerd` exited with
`failed to create private durable state temporary file: No space left on
device (os error 28)`. The transport relaunched it in about 20 s, and
runnerd restored its durable session. Then the runner reported
`turn.failed {code: provider_turn_lost_on_restore, recoverable: true}`,
and the run failed as `native_provider_terminal_failed`. Retries on
fresh sandboxes hit the same failure. The tasks ended in "Recovery
needed" with `provider_failure_meaning_unverified`.

**Steps to reproduce**
1. Run an ACPX or Codex agent through `paperclip_runner` in a managed
sandbox.
2. Have the agent fill the disk during a turn, or end `runnerd` in any
other way mid-turn while the server keeps running.
3. The transport relaunches `runnerd`, and the runner reports the turn
as lost on restore.
4. The run fails instead of continuing the turn.

**Expected behavior**
The execution that started the turn sends one continuation turn in the
restored conversation, exactly as it does after a server restart. The
agent continues from its recorded progress and sees the full disk in its
tools.

**Environment**
Managed Daytona sandbox (fleet snapshot, 4 CPU, 8 GiB memory, 10 GiB
disk), `paperclip_runner` native runtime with Claude through ACPX.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`: the
execution records the turn id of the turn it starts.
`resumeInterruptedTurn` now accepts two owners of a turn lost on
restore:
- a session recovered at startup with the control-plane restart claim,
for the checkpointed active turn (unchanged), or
- a session this execution opened itself, for the turn this execution
started (new).
- All other guards stay: one continuation per run, the provider
allowlist, no compaction or goal runs, no pending runtime requests, and
the snapshot must show that same turn as restart-interrupted.
- `native-session-runtime.test.ts`: a new regression for an ACPX session
that loses its own turn to a mid-execution `runnerd` restore, plus a
negative case for a lost turn this execution never started.

## Verification

- Red before the fix: `npx vitest run
src/native-session-runtime.test.ts` failed `continues its own turn once
when runnerd is restored mid-execution` with
`NativeProviderTerminalFailure: Provider turn failed`, the same failure
seen on staging.
- Green after the fix: `native-session-runtime.test.ts` and
`native-session-runtime.provider-turn.test.ts` pass (337/337). The
existing `requires the control plane restart claim` case still refuses a
recovered session without the claim.
- `pnpm verify` (affected selection): see the CI results on this PR.
- Live acceptance: after deploy, retry the affected staging tasks and
confirm that a mid-turn `runnerd` restore leads to one continuation turn
in the same sandbox instead of `native_provider_terminal_failed`.

## Risks

- **Second executor.** The continuation can start only from the
execution that opened the session and started the turn. It is not used
for a recovered or adopted session without a claim. It runs at most once
per run (`restartContinuationStarted`). The runner emits this code only
after it proves the old provider process ended.
- **Run starts.** This starts no new runs. It adds at most one turn to a
run that would otherwise fail and schedule a fresh-sandbox retry. During
a crash loop, `MAX_UNSTABLE_RUNNER_RESTORES` in the transport still
fails the transport.
- **Disk still full.** If the continuation fills the disk again, the
second lost turn finds `restartContinuationStarted` set, and the run
fails as it did before. The worst case is the current behavior.
- **Not in this PR.** Paperclip's own footprint (about 3G of provider
pack per sandbox, and a per-session pnpm store) and classifying a
disk-full failure so it does not retry into the same wall. Both are
follow-ups.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, and a code-tracing subagent).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:55:16 -06:00
..