mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a recovery read to a remote Runner goes unanswered, the next attempt of the same run carries `replaceUnansweredRunner` > - That attempt verified the Runner was alive and then refused to adopt it anyway. On a resumed sandbox lease the reuse check then failed with `resumed_not_suspended` and left the Runner running. Otherwise a second Runner was launched, and its launch fence killed the busy Runner, which lost the provider turn. > - After a late reconnect a Runner is often slow, not dead: it is still replaying the events it saved with no controller attached > - This pull request makes the attempt check progress first. It adopts a live Runner that is making progress and replaces only a Runner that made no progress. > - The benefit is that a slow but working Runner keeps its turn, and the retry no longer causes the failure it was meant to fix. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13. The retry after a first-attempt timeout failed with these codes (counts from the traced sample of 136 failures): - `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_not_suspended` (part of 40 `ingress` failures) - `runner_remote_process_identity_unavailable` (25): the Runner was gone by the time the retry checked, because the launch fence or a previous retry had killed it - "restored the conversation after process loss, but…" / `provider_turn_lost_on_restore` (10): the replacement Runner restored from the old state and the provider turn was gone Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9 failed. Over 420 s: 14 of 15 failed. The process ID of the Runner changed inside the same sandbox in the failed runs, which shows the replacement. Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit` verifies the Runner is alive and then returns `undefined` because of that flag, so `adoptedProcess` is `undefined`. **Expected behavior** A live Runner that is making progress is adopted. A Runner that made no progress is stopped with proof and then restored from its own journal. **Steps to reproduce** 1. Reattach a remote run whose previous attempt left a recovery read unanswered (`replaceUnansweredRunner: true`), with the Runner alive and its saved event sequence past the controller's last acknowledgement. 2. On master the attempt launches a second Runner, or fails with `resumed_not_suspended` on a resumed sandbox lease. 3. The new tests in `native-session-executor.test.ts` and `native-runner-restart-recovery.integration.test.ts` do this. **Paperclip version or commit** `master` at426e8a35dc. ## What Changed - `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): with `replaceUnansweredRunner`, the attempt adopts the live Runner, launching nothing and signalling nothing, when: - the Runner's saved event sequence is past the controller's last acknowledgement, or - its saved state (event and command sequences, generation, lifecycle) changes within a bounded wait of about 10 s. - The baseline is the first successful state read; every successful read is also compared against the controller's acknowledgement. A Runner whose state cannot be read at all during the wait is adopted, not replaced: missing evidence is not proof of a wedge. A Runner whose identity changes during the wait fails closed. - A settled exit during the wait keeps the attempt allowance that admitted the replacement, so it also restores on the extra fourth attempt. - Only a Runner that made no progress is replaced. The relaunch's existing launch fence stops it and its process lineage with proof, and the new Runner restores its own journal. A resumed sandbox lease no longer refuses that relaunch with `resumed_not_suspended`. - A Runner that exits during the wait follows the existing exited-Runner rules: a settled suspension restores, and a mid-turn exit stays fail-closed. - The wait runs only on this recovery path, never on a normal run start. - `doc/execution-semantics.md`: describes adopt versus replace. - The claim flag `replaceUnansweredRunner` is set in claim logic that another session owns. This PR does not change it; it changes only what the executor does with it. ## Verification Red on master, green on this branch: - `native-session-executor.test.ts` - `adopts a live Runner that emitted events past the controller's last acknowledgement` - `adopts a live Runner whose saved state advances within the bounded wait` - `hands a live Runner that made no progress to the relaunch, whose launch fence stops it, without signalling it here` - `restores from the journal of a Runner that suspended and exited during the wait` - `stays fail-closed for a Runner that exited mid-turn during the wait` - `native-runner-restart-recovery.integration.test.ts`: `reattaches a live %s Runner whose recovery read went unanswered on a resumed sandbox lease` (progressing, wedged) - Review fixes, `native-runner-restart-recovery.integration.test.ts` (the first four were red before the fix): - `adopts a live Runner whose first state read failed and whose later reads advance` - `adopts a live Runner whose first readable state is already past the controller's acknowledgement` - `does not replace a live Runner whose state could not be read at all during the wait` - `restores from the journal of a Runner that suspended and exited during the wait on the admitted fourth attempt` - `fails closed when the Runner's identity changes during the wait` (pins existing behavior) - `native-remote-retry-admission.test.ts`: unchanged, still passes. Local runs on this branch: `tsc --noEmit` is clean for `server` on this branch (masterc6b497d831, which contains #21442). `native-runner-restart-recovery.integration.test.ts`, the adopt-or-replace block: 12 of 12 pass. The full `native-session-executor.test.ts` and `native-remote-retry-admission.test.ts` runs did not finish on this shared machine (load 20 to 96); CI ran them on this head and they passed. Conflict resolved: master added a relaunch branch for a Runner that exited for a fresh bootstrap (#21374) at the same place. Both branches are kept; the cheaper check for an unanswered Runner that made no progress comes first. Daytona results that exist. They ran on a combined branch with this PR, #21450, #21442 (merged) and #21362 (merged), so they cannot be credited to this PR alone: - G5 baseline (master d1d71a7755; 480 s delay injected on the first identity write per run): one Claude run reproduced the staging chain: authentication 419 ms, identity +423.5 s, attempt 1 `native session provider recovery timed out after 900000ms`, then the retry failed in 6 s with `runner_harness_state_mismatch: resumed_not_suspended`. - G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished hands-off. The chain did not recur. All runs that crossed swap 2 were adopted on the same Runner process ID. - Caveat: the adopt-or-replace retry path that this PR changes was never exercised in a cell. In G5c no first attempt failed, because all identity writes started together 4 to 5 s after the claim. G5c also ran before the phase timing fix in #21450. ## Risks - Not proven on Daytona: the adopt-or-replace path itself (see the caveat above). - Without #21442 (merged) an adopted Runner can still time out on its first command while it replays a backlog. This PR stops the replacement; #21442 (merged) makes the adopted Runner answer. - A Runner whose state cannot be read during the whole wait is adopted. If it is in fact wedged, the adopted attempt times out again and the attempt limit bounds it. - A wedged Runner that still emits a few events counts as progressing and is adopted. The existing attempt limit bounds this. - The progress check adds a wait of up to about 10 s on this recovery path only. A normal run start has no added wait. - The claim construction (`native-restart-recovery.ts`) is owned by another session and is unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>