mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Restart recovery decides whether an interrupted native run can resume, bootstrap again, or must wait for a person. > - A remote run is blocked as "ambiguous" when recovery cannot prove what reached the sandbox runner. > - In a Daytona chaos wave (#19947), two runs were blocked even though the controller had recorded that the relaunched runner exited before any provider work. > - This pull request accepts that recorded runner exit as proof and bootstraps the run again, under the existing attempt budget. > - The benefit is that a restart no longer turns a run with no provider work into a failed, blocked run. ## Linked Issues or Issue Description Refs #19947 (Daytona data), #19831 (durable-ownership plan), #19934 (the stuck claimed wake from the same runs). **What happened?** Wave `w-20261108T012041-s9301`, runs `450f277d…` and `9b93d40c…`, went through this sequence: 1. Restart recovery proved the remote bootstrap incomplete and relaunched it. 2. The new runner exited: "runnerd exited while waiting for run.prepare". The executor settled `bootstrap_retry` with `providerSessionEstablished: false`, `providerEventsExist: false` and `checkpointExists: false`. 3. The next restart recovery saw the recorded runner PID and blocked the run as `ambiguous_remote_recovery_evidence`. **What did you expect?** Bootstrap again, because the runner is proven gone and no provider work happened. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "relaunched remote runner"` fails on master with `blocked / ambiguous_remote_recovery_evidence`. ## What Changed - `native-restart-recovery.ts`: new proof `remoteRunnerExitedBeforeProviderWork`. It applies to a remote sandbox with a prepared workspace and a settled `bootstrap_retry` failure whose original code is `native_runner_process_exited`, with no provider session, events or checkpoint and no lease or successor. Such a run is classified as `bootstrap_incomplete` (`remote_runner_exited_before_provider_work`). It passes the same retry admission check as `remote_runner_never_launched`, and the same three-attempt budget. - Tests: the s9301-shaped retry plus five counter-cases. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "exited remote runner|relaunched remote runner|remote bootstrap retry|never requested a runner|exact-checkpoint retry"`: 14 pass. - Red/green: the new bootstrap test and the budget counter-case failed on master (both returned `blocked / ambiguous_remote_recovery_evidence`, the same as the s9301 rows) and pass here. The other four counter-cases (not a runner exit, session established, provider events, checkpoint) block on both. - Loop: focused vitest with one embedded Postgres, 26–50 s. - Not run locally: the server typecheck and the full suite (memory limits on this host). Required CI covers them. ## Risks - Fail-open risk: the proof needs the executor's own settled record of a runner exit with no provider work, plus a fresh check that no checkpoint or provider evidence exists. A recorded PID without the exit record still blocks, as before. - Previous-release story (rules/upgrade-safe-run-state.mdc): no persisted shape changes. Rows without these failure fields keep today's outcome. - Nothing runs before `turn.start`. This is restart recovery only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>