Files
PaperClipAI/doc
DottaandClaude Opus 5.5 eb54831d28 fix(native): relaunch a remote Runner that exited for a fresh bootstrap (#21374)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs often use a remote Runner in a sandbox. After a restart,
a deploy or a same-run retry, the server reattaches to that Runner or
starts a new Runner over its saved state
> - A Runner that no controller reaches before its connection lease
expires stops its provider, saves its state as `recoverable_failure`
with `lease_expired_requires_bootstrap`, and exits. The Runner protocol
says that only a new process with a fresh bootstrap ticket can
reconnect, and that the new process continues from the same cursor
> - Reattachment started a new Runner over saved state only for a
settled `suspended` exit (or an exit before `session.open`). So it
refused this exit, and the run ended with
`runner_remote_process_identity_unavailable`
(`native_continuation_requires_reconciliation`)
> - This pull request lets reattachment launch a Runner over the exact
saved state of a Runner that exited to ask for a fresh bootstrap
> - The benefit is that the run continues by itself after a long
controller gap, with no Retry and no manual sandbox repair

## Linked Issues or Issue Description

Refs #21329, #21362, #20853

**What happened?**
On hosted staging (build `2b69cdfcd6`, read-only trace), Codex runs on
remote Daytona Runners ended with
`native_continuation_requires_reconciliation` and the error
`runner_remote_process_identity_unavailable`, about 5 per hour, after
#21329 was deployed. All five sample runs (c6fabe9a, 31d54b02, 995b63df,
dd595934, 50c6e6e6) ended the same way. The last
`native.remote_runner_recovery_failed` event recorded
`process_verification` / `process_not_running`, `runnerLifecycle:
recoverable_failure`, `runnerRecoverableFailure:
lease_expired_requires_bootstrap`, and the Runner diagnostic "connection
lease expired; a fresh bootstrap is required". Since 10:30 UTC, 8 of the
15 runs that ended with this error had this exact exit.

Timeline of run c6fabe9a:
- 10:46 to 10:54: `runner.session.startup` took 476 s in a loaded
sandbox.
- 10:54:02: the controller sent `turn.start`. At 10:54:47 the command
timed out: "PRP command turn.start timed out (runner connected; pending
ahead 0; last runner event seq 2 91s ago)". The run resumed in the same
run and the turn then ran (tools completed from 11:14).
- 11:39: a graceful server shutdown suspended the controller. At 11:43
the next controller started the reattachment, but it did not reconnect
to the Runner until the server restarted again at 11:58. In that gap the
Runner's lease expired and the Runner exited.
- 11:58:53: the reattachment proved the process gone and refused the
`recoverable_failure` state. The run failed at 11:59.

Why `turn.start` timed out while the Runner was connected: the Runner
answers `turn.start` only after Codex answers its `turn/start` JSON-RPC
request. While it waits, the Runner buffers Codex notifications and
commits no events, so the controller sees a quiet connected Runner with
nothing pending ahead. The controller's bound for the `turn.start`
command result is a fixed 30 s. #21329 changed only the later wait for
`turn/started`. In a sandbox that needed 8 minutes to start, Codex
answered later than 30 s. That timeout is recoverable, and the run did
recover from it. This pull request does not change it (see Risks). The
failure that ended the runs is the refused reattachment.

**Expected behavior**
When a remote Runner exits because its lease expired (or its reconnect
grace or deadline ended) and its saved state is intact, reattachment
starts a new Runner from that state and the run continues.

**Steps to reproduce**
1. Start a native Codex run on a remote sandbox Runner.
2. Keep every controller from reaching the Runner for longer than its
connection lease (on staging: a deploy that stalled the reattachment for
about 15 minutes).
3. The Runner saves `recoverable_failure` /
`lease_expired_requires_bootstrap` and exits.
4. The next reattachment fails with
`runner_remote_process_identity_unavailable`, and the run ends.

**Agent adapter(s) involved**
`paperclip_runner` with Codex on a Daytona remote sandbox. The rule is
not provider-specific.

## What Changed

- `remote-runner-recovery-diagnostics.ts`: new
`exitedRunnerRequestedFreshBootstrap` (diagnostic) and
`runnerStateRequestsFreshBootstrap` (saved state). They accept lifecycle
`recoverable_failure` with `lease_expired_requires_bootstrap`,
`transport_reconnect_grace_exceeded` or
`transport_reconnect_deadline_exceeded`. These are the three codes the
Rust Runner records after `shutdown_preserving_cleanup`. The state check
also requires no pending terminal delivery and no warm transition.
- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): when
the exact Runner process is proven gone and its diagnostic names one of
these exits, the server reads the saved state again. If the state has
the same six identity fields and still names that exit, the reattachment
launches a Runner from it instead of failing. The outbox does not need
to be empty and a turn may be active: the new Runner replays from the
same cursor. The attempt bound is the one the reattachment already
admits (3, or 4 for an unanswered-Runner replacement claim). Other exits
are unchanged.
- `native-session-executor.ts` (host preparation in
`createRunnerdBackend`): a resumed sandbox for a reattachment that did
not adopt a process re-reads the Runner state. When it still has the
same six identity fields and names the fresh-bootstrap exit, preparation
keeps it in place, as for an adopted live Runner. It no longer fails
with `runner_harness_state_mismatch: resumed_not_suspended` (found by
Greptile).
- `doc/execution-semantics.md`: one paragraph describes the rule and the
staging evidence.

## Verification

- Red on `origin/master`: the 5 new "launches a new Runner over the
saved state after ..." cases failed (the reattachment threw
`runner_remote_process_identity_unavailable`). The 7 new fail-closed
cases passed.
- Green: `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
controller restart adoption"` passes 118 tests.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
1712 passed, 21 skipped, 1 failed. The failure ("verifies a live warm
owner before refreshing run authority", warm-session supervision, not
touched here) passes when run alone. It is a timing failure under
full-file load.
- Launch path: "keeps the saved mid-turn state of a remote Runner that
exited for lease_expired_requires_bootstrap when the reattachment
prepares its sandbox" runs `createRunnerdBackend` and
`prepareExternalRunnerState` with a mid-turn Codex checkpoint. Red
before the host-preparation change (`resumed_not_suspended`), green
after. An unknown recoverable failure stays fail-closed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`
on the final revision: 1494 passed, 7 skipped.
- `cd server && node scripts/typecheck-layers.mjs` (includes test
programs): passes.
- Fail-closed cases covered: an unknown recoverable failure, a warm
transition, a pending terminal delivery, a state that changed after the
exit, a different Runner instance, attempt 4 on an ordinary claim, and
attempt 5 on a replacement claim.
- Staging read (read-only): the 5 sample runs and 69 runs with this
error in the last 9 hours were classified from their
`native.remote_runner_recovery_failed` events.

## Risks

- Medium. Reattachment now starts a new Runner over a saved state that
can be mid-turn. The Runner itself stops its provider before it saves
this state. The new Runner restores the session. It continues the turn,
or it reports the lost turn as recoverable only after it proves that the
old provider process ended. A recoverable lost turn on a remote
reattachment ends the run as `native_provider_terminal_failed` until
#21362 merges; the replacement lane then continues the task. That is
still better than the current reconciliation hold.
- The attempt bound does not grow. A Runner that exits this way again on
a later reattachment is bounded by the same provider attempt budget.
- This does not change the 30 s `turn.start` command bound or the deploy
gaps that let the lease expire. Those are separate follow-ups. 57 older
runs with this error had a different exit (`ready`, not settled). This
pull request keeps that case fail-closed; #20853 covers the
settled-`ready` part.
- No schema, API, UI or Runner image change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution, read-only browser
reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:53:38 -06:00
..
…
…
…
…