mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through a Runner (`paperclip-runnerd`) in a remote sandbox. When a Runner exits before `session.open`, the run restarts its bootstrap on the same run. > - #20972 lets host preparation restart that bootstrap when the Runner wrote `runner-state.json` but no provider state. It needs a proof that this run's own interrupted Runner is stopped or gone. > - That proof is recorded only when the server still has the run's local Runner controller root. A volume-free server, or a root that held no identity, skips it. The live chaos wave after #20972 still failed with `runner_harness_state_mismatch: resumed_unavailable`. > - This pull request lets host preparation establish the same proof from durable sandbox evidence when no local proof exists. > - The benefit is that the restart also works on volume-free servers, which is how Cloud runs. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261112T002811-s6402` ran #20972 (Claude through ACPX, Daytona warm, `--ephemeral-server`, K1/K2/K15 with `--k15-fresh-home`, seed 6402, `--runner-bind-fault`). 5 of 6 tasks needed Retry, with 6 `resumed_unavailable` failures. Run `fbb1666d` in t01: the sandbox held `runner-state.json` for this run with `lifecycle: connecting`, the same Runner instance and session, `lastConnectionProtocolVersion: null`, and no provider state. That is exactly the state #20972 restarts, but its proof was absent. **Root cause** `recordRemoteBootstrapRestartProof` is called only inside the `existsSync(scoped)` block of the restart path in `server/src/services/native-runtime/native-session-executor.ts`, after the local controller identity is read. With no local root (volume-free, a fresh home, or an emptied root), `provenOwnBootstrapRestart` stays false, so `inspectOwnUnpreparedBootstrap` from #20972 returns nothing and the resumed branch throws `resumed_unavailable`. **Expected behavior** A run with a `bootstrap_incomplete` recovery claim whose sandbox Runner state proves it never connected restarts its bootstrap without Retry. ## What Changed - `inspectOwnUnpreparedBootstrap` now runs for any `bootstrap_incomplete` claim of this run. When no proof was recorded, it records one only if the sandbox Runner state proves it never connected (`remoteRunnerStateNeverConnected`: `lastConnectionProtocolVersion: null`, `lastControllerCommandSeq: 0`, `reconnectCount: 0`, empty outbox and processed commands, no pending terminal delivery). The exact Runner process must also be stopped or proven absent by the existing `stopStaleRemoteRunnerBeforeBootstrap`, which uses the Runner state's own identity. All earlier conditions from #20972 still apply: same run, Runner instance and session, a lifecycle other than `suspended`, no failover backup and no archived rotation snapshot. ## Verification - Red on master (#20972 merged): `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"`. The new `interrupted_bootstrap_unprepared_unrecorded` cells failed: resumed lease `Error: runner_harness_state_mismatch: resumed_unavailable`, created lease no restart. - Green: those cells pass. `_connected` (a Runner state that connected, no local proof) and `_unproven` (the stop is not proven) keep the refusal. The full file passes (1409 of 1409). Server `tsc --noEmit` passes. - Chaos, same cell with `--runner-bind-fault`. Wave `w-20261112T002811-s6402` (#20972 only): 1 of 6 passed, 6 `resumed_unavailable`. Wave `w-20261112T010800-s6402` (this change): 3 of 6 passed, 2 `resumed_unavailable`. In this wave, run `0a080e83` restarted its bootstrap three times on the same sandbox: three `.bootstrap-restart-*` roots were moved aside. The baseline wave on master before #20972 (`w-20261111T230537-s6402`, no injected fault) passed 2 of 6. ## Risks - The new path runs only for a `bootstrap_incomplete` claim of this run when the full harness check fails, so the normal path is unchanged and adds no wait before `turn.start`. - A Runner state that ever connected, received a command or holds outbox data keeps today's refusal. The moved-aside root is kept for inspection. - Not fixed here, and seen in both waves: every Runner on a sandbox listens on one fixed port (43127 in all captures). A live Runner of another run or session on the same sandbox therefore makes each relaunch fail with `runner_ingress_bind_conflict`, until the bounded retry is exhausted (`native_session_retry_exhausted`). A later attempt can still end with `resumed_unavailable` or `resumed_not_suspended`. That needs its own fix in predecessor retirement or port selection. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20972, #20928, #20953, #20863. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>