mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent turns run in a remote Runner (`paperclip-runnerd`) in a Daytona sandbox. A release swap stops the old server and a new server recovers each run. > - When the old server stops during a run's bootstrap, the new server restarts that bootstrap on the same run (`bootstrap_incomplete`). The claim is granted only when the Postgres controller checkpoint proves that no provider turn was sent. > - The old server can stop after it staged launch files (Codex home, config) into the sandbox session folder, but before it launched the Runner. Its detach fence stops it there. > - The new server then finds no Runner state in the sandbox. A fresh bootstrap must create the session folder itself, and the folder already exists, so the run failed with `runner_harness_state_mismatch: resumed_unavailable`. The failure is an ambiguous session, so the task waits for a person. > - This pull request lets host preparation prove that no Runner was ever launched in that folder, move the folder aside, and restart the bootstrap. > - The benefit is that a release swap during this part of a bootstrap no longer blocks the task. No provider work can be lost or replayed, because no Runner ever ran. ## Linked Issues or Issue Description Refs #21120 (same release-swap class: the Runner was launched and connected before the old server stopped; that pull request stops it in host preparation. This pull request covers the case where no Runner was launched.) Refs #21079 (launch-time fence for a stale Runner on the same state folder.) **What happened?** On hosted staging, during a release swap on 2026-11-12, a Codex run on Daytona failed with `native_session_interrupted` and `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_unavailable`. The old server was in `harness_state.launch_material` when its shutdown drain detached the run. It recorded no Runner process identity. The new server chose "Restarting an incomplete native runner bootstrap on the same heartbeat run". Its host preparation found the sandbox resumed and no Runner state. Then the claim of the session folder failed, because the old server had already created that folder. The run needed a person. **Expected behavior** The run restarts its bootstrap on the same sandbox and continues without a person. A provider turn is never replayed and two Runners never share the run. **Steps to reproduce** 1. Start a Codex run on a Daytona sandbox. 2. Stop the server after the bootstrap stages launch files into the sandbox session folder and before it launches the Runner. 3. Let a new server recover the run with a `bootstrap_incomplete` claim. The new server has no local controller root for the run. 4. Host preparation fails with `runner_harness_state_mismatch: resumed_unavailable`. **Paperclip version or commit** Staging release `704b3f4d63`. The code is the same on master at `ef2b53b675`. **Deployment mode** Hosted staging, remote sandbox execution on Daytona. **Agent adapter(s) involved** Paperclip Runner with Codex (`codex_app_server`). ## What Changed - `native-session-executor.ts`: under a same-run `bootstrap_incomplete` claim with no failover backup, when host preparation finds no Runner state, a new check asks the sandbox in one command whether the session folder exists and holds no Runner trace. A Runner trace is the live `runner/runner-state.json`, the `runner/runner-process.identity` launch marker (the launch script writes it before the Runner starts), an `authority-epochs` archive, or any other `runner-state.json` or launch marker anywhere under the folder. If the folder holds no Runner trace, no Runner was launched there, so host preparation records the restart proof (`absent_confirmed`). It then uses the existing path: move the folder aside (it is kept for inspection), claim a new folder, and bootstrap again. If the folder is missing, the existing fresh-claim path runs unchanged. If any Runner trace exists, the run stays fail-closed as before. An archived session is therefore never discarded. - `native-session-executor.test.ts`: new `interrupted_bootstrap_unlaunched` cases. Resumed and created sandboxes restart the bootstrap. A launch marker, an archived authority epoch, Runner state elsewhere in the folder, or a failover backup keeps the run fail-closed with `resumed_unavailable`. ## Verification - Red: with the source change reverted, `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session" --maxWorkers=2` fails on both `interrupted_bootstrap_unlaunched` restart cases with `runner_harness_state_mismatch: resumed_unavailable`, which is the staging error. - Red: without the archived-state guard, the `_archived` and `_nested_state` cases fail (the folder is moved aside). With the guard they pass. - Green: the same command passes (64 tests). - `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts --maxWorkers=2`: full file, 1422 passed before the guard commit. The focused 64 pass on the final revision. - `npx tsc --noEmit -p server/tsconfig.json`: passes. - `git merge-tree` against #21120's head (`0d18f18928`) merges without conflict. The new check runs only when no Runner state exists, so #21120's new call site (it runs only when a Runner state exists) gets the same results as before. ## Risks - Low. The new path runs only in host preparation, under a `bootstrap_incomplete` claim for the same run id, with no failover backup. That claim is granted only after the previous controller's lease has ended. With no Runner state or launch marker anywhere in the folder, no Runner and no provider session existed there. The old folder is moved aside, not deleted. - A controller that is still launching at the same moment would have to hold a lease that the claim already took over. The detach fence stops such a launch, and #21079 adds a launch-time fence for a leftover Runner on the same state folder. - No added wait before a normal turn starts: the check runs only on this failure path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser for read-only run evidence, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>