mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
DottaandClaude
16c30ac32b
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through `paperclip-runnerd`. After a restart, a new runner relaunches the provider (for example the Codex app server). A durable startup fence stops a second runner from launching a provider while an unadmitted one may still run. > - #19784 lets a new runner retire a fence whose recorded provider process and group have provably exited. That rule covers a launch that a kill cut off. It deliberately keeps a failed initialization closed, because that can be a real provider failure. > - A relaunch whose reply never came within the runner's 30 s request deadline is saved as the same failed initialization. A stalled or frozen host causes this. The restart-chaos fault TM2 SIGSTOPs the runner and Codex, and their monotonic clocks keep running while they are stopped. > - Every later runner then refused to restore with "provider startup ownership remains unadmitted", and the user had to click Retry. The owner's rule is that a run interrupted by a restart or sleep must resume by itself. > - This pull request records whether initialization failed only because the provider gave no reply, and lets a later runner retire that fence once the provider is provably gone. > - The benefit is that a stall during a provider relaunch continues automatically, while a real provider rejection still fails as before. ## Linked Issues or Issue Description Refs #19784 **What happened?** Restart-chaos wave `w-20261110T091152-s426`, task t03 (Codex, B5), run `c0bbce18-836b-43af-8195-92ebe3e21cca`. A TM3 restart relaunched the Codex provider. A TM2 freeze stopped the tree 1 s later for 45 s, and a TM3 kill followed the thaw. The next three runners each failed `runner.drain` with "provider startup ownership remains unadmitted". Recovery stopped with `native_session_retry_exhausted`, and the task needed Retry. **What did you expect?** The new runner relaunches the provider and the run continues. **Steps to reproduce** The new test `restart_relaunches_a_provider_whose_relaunch_timed_out_without_a_reply` (runner-core `tests/codex_provider.rs`). The fake Codex stalls its first `thread/resume` reply past the 30 s deadline. On `origin/master`, the saved attempt is `initialization_failed` / `thread_open`, and the next runner fails with "provider startup ownership remains unadmitted". ## What Changed Design: - The deadline that passes is the runner's own 30 s request deadline. When a startup request (`initialize`, `thread/start`/`thread/resume`, `thread/read`) reaches that deadline with the provider still running and silent, the runner sets `noReply: true` on the durable `initialization_failed` startup record. - Not `noReply`: a provider error response, a provider that exited before it replied, or closed stdout. - `retire_orphaned_startup_attempt` retires an `initialization_failed` fence only when `noReply` is set and the recorded process and its process group are both gone (`ESRCH`). This is the same proof as `spawned` in #19784. A rejection, an exit, a live process or group, and spawn or admission failures stay fenced. - Validation accepts `noReply` only on `initialization_failed` at the `initialize`, `thread_open` or `thread_read` stage. - Bounded: a runner restores once (`restore_checked` / `restore_error`), so the loop is one relaunch per recovery attempt. Recovery's existing resume budget (`native_session_retry_exhausted`, seen after three attempts in the wave) stops a provider that never replies. - Upgrade: `noReply` is `#[serde(default, skip_serializing_if = "Not::not")]`. A new runner reads state saved by master as `noReply: false`, so that state keeps today's behaviour and stays fenced. - Rollback: an old runner (`deny_unknown_fields`) rejects state that holds a `noReply` fence and fails closed. That run needs Retry, which is today's outcome for the same fence. The field is written only while a timed-out fence is present, and admission clears it, so all other state is unchanged for old runners. - Normal path: nothing is added before `turn.start`. The cost is one in-memory flag per request; no extra I/O. Files: - `runner-core/src/codex_provider.rs`: `last_request_got_no_reply` and the `no_reply` field on `ProviderStartupObservation::Failed`. - `runner-core/src/provider_backend.rs`: the `noReply` field, its validation, and the retirement rule, with unit tests: - a no-reply fence is retired at `initialize`, `thread_open` and `thread_read`; - it is kept while the provider may still run; - a rejection or a state saved without the field stays fenced. - `runner-core/src/bin/fake-codex-app-server.rs`: the test-only switch `--stall-first-thread-resume-ms`. - `runner-core/tests/codex_provider.rs`: the end-to-end red/green test. - `tests/restart-chaos/README.md` and `time-faults.ts`: correct the TM2 claim. A stopped process's monotonic clock keeps running, so TM2 is also a long process stall. ## Verification - `cargo test -p paperclip-runner-core --test codex_provider restart_relaunches_a_provider_whose_relaunch_timed_out` is red on `origin/master` (the next runner refuses restore) and green here, in about 31 s. - `cargo test -p paperclip-runner-core`: 1080 passed, 0 failed. This includes `failed_autonomous_restore_and_rollover_keep_their_exact_startup_origin`, where a real "no rollout found" rejection still never restarts. - `cargo fmt -p paperclip-runner-core -- --check` is clean. - A restart-chaos proof wave on seed 426 will follow in a comment. ## Risks - A provider that hangs for good is relaunched once per recovery attempt until the existing resume budget stops it. Before, it failed on the first attempt. This costs at most a few extra 30 s attempts, and they are bounded. - Rollback to an older runner while a `noReply` fence exists fails closed for that one run, which needs Retry, as it does today. - No database, server or API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, cargo and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>