mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs execute in a remote Runner, and a same-run retry reattaches that Runner after an interruption > - #21272 made the next attempt replace a Runner that left its recovery read unanswered, instead of adopting it again > - On staging, three runs after #21272 went live still failed: their first unanswered read came on the third and last attempt, so no next attempt existed > - This pull request gives such a run one more attempt, used only for that replacement > - The benefit is that the replacement from #21272 can run when a restart used up the middle attempt ## Linked Issues or Issue Description Refs #21272, Refs #21161 **What happened?** After #21272 (497ccba8f1) went live on staging at 05:34Z, three Codex runs still failed with `native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq … 55–84s ago)`. The run events of all three show the same sequence: 1. Attempt 1 fails at 06:24 with `runnerd did not report the provider turn identity`. 2. Attempt 2 starts at about 06:25. The 06:24–06:25 server restart ends it with `local_runtime_api_listener_unavailable`. This attempt sends no recovery read. 3. Attempt 3 adopts the same Runner. Its first recovery read gets no answer. Attempt 3 is the last of three, so the run ends. `adoptedRunnerUnanswered` was recorded on attempt 3, but no next attempt read it. The replacement from #21272 never ran. **Expected behavior** A Runner whose first unanswered read comes on the last attempt is still replaced once by a fresh Runner from the saved journal. The provider turn is not sent again. The run does not end only because a restart used the middle attempt. **Steps to reproduce** 1. Start a Codex run in a remote Daytona Runner. Let attempt 1 fail at turn start, so the Runner stays alive. 2. Restart the server during attempt 2, so that attempt fails without a recovery read. 3. Let the Runner leave its recovery read unanswered on attempt 3. 4. See the run end with `session.snapshot timed out` and no replacement. **Paperclip version or commit** Staging with #21272 (497ccba8f1). Fixed on `master` after `bbafa78fe2`. **Deployment mode** Hosted (staging), remote Daytona sandbox Runner. **Agent adapter(s) involved** Codex (native Runner, `paperclip-runnerd`). ## What Changed - `nativeUnansweredRunnerReplacementDisposition` (executor): when attempt 3 fails because its adopted live Runner left a recovery read unanswered, and that attempt was not itself a replacement, the failure stays `retryable_failure` (`native_session_interrupted`, retry in 30 s) instead of `native_session_retry_exhausted`. - `claimNativeRestartRecoveries`: a failed remote retry at attempt 3 with `adoptedRunnerUnanswered` is not treated as an exhausted budget. The claim is `reattach_remote_runner` at provider attempt 4 with `replaceUnansweredRunner`. - `nativeAdmittedProviderAttemptLimit` (executor admission gate): a `reattach_remote_runner` claim with `replaceUnansweredRunner` is admitted up to provider attempt 4. Every other claim keeps the limit of 3. - `verifyRemoteRunnerOrSettledExit`: the replacement of a live Runner is allowed up to attempt 4. An exited Runner keeps the fail-closed path. - Bound: the extra attempt is a replacement (`replaceUnansweredRunner`), so it never gets another one. Attempt 4 is exhausted by the existing rule. - `doc/execution-semantics.md` describes the extra attempt. ## Verification - End to end (embedded Postgres): `native-remote-retry-admission.test.ts` › "executes the one replacement attempt after the last attempt's first unanswered recovery read, and no further one". It seeds attempt 3 with the staging failure, claims it, and executes the claim through `executePaperclipNativeSession`. The provider is reached once, the run then ends at attempt 4, and no further claim is made. With the old `nextAttempt > 3` gate this test fails before the provider is reached. - Red on `master`, green with the fix (7 tests, including "admits provider attempt 4 at execution only for the replacement claim"): - `native-runner-restart-recovery.integration.test.ts` › "gives a Runner whose first unanswered read came on the last attempt one replacement attempt: true" (embedded Postgres; the coordinator is seeded at attempt 3 with the staging message). On `master` no reattach claim was made. - `native-session-executor.test.ts` › "bounds the replacement after …" (4 cases: the last attempt's first unanswered read is retryable; a replacement that also went unanswered, another failure, and attempt 4 stay terminal) and "replaces the live Runner on the extra replacement attempt". - Focused suites: `native-session-executor`, `native-restart-recovery`, `native-runner-restart-recovery.integration`, `runner-lost-mid-turn-continuation`: 1737 passed, 21 skipped. - `pnpm typecheck` in `server` passes. - Not run locally: the container-gated remote Runner suites and the full suite (CI covers it). Why the remote Runner stops answering is a separate investigation and is not changed here. ## Risks - Low to medium. This adds at most one provider attempt per run, and only after an unanswered recovery read on a live adopted Runner. The attempt replaces the Runner through the launch fence, which proves the old Runner and its lineage gone or fails closed. No provider turn is sent again. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run events), extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>