mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 00:26:28 +01:00
DottaandClaude
e719d42557
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run talks to a Runner process (runnerd). The controller in the server sends commands. The Runner journals each command before it acts and replies with a result. > - `runner.suspend` and `runner.shutdown` are terminal commands. The Runner waits for a `command_result_ack` before it exits. It compares that acknowledgement with the receipt that it journaled. > - A Runner that is killed inside `runner.suspend` journals the command but not its result. The next Runner replays the command as `indeterminate`. It sends it on the wire as `failed` with `execution_indeterminate`, and it still treats it as terminal. > - The controller acknowledged the `failed` status that it received. The Runner compared that with its journal status `indeterminate` and exited with "terminal command result acknowledgement changed its durable identity". The controller also lifted its terminal fence for a failed suspend and sent the next command to a Runner that was about to exit. > - This pull request makes both sides agree on the status of that one receipt, and keeps the fence. > - The benefit is that a freeze or restart during a suspend no longer burns the same-run retries and leaves the task at a manual Retry. ## Linked Issues or Issue Description **What happened?** A restart-chaos task (local Runner, Claude, a 28-second freeze of the server during a live turn) needed a manual Retry. Retry worked. 1. The turn completed. The server then sent `runner.suspend`. The server froze before the Runner reported the result. 2. The close found the Runner not durably suspended (`commandStatus: pending`, `runnerLifecycle: ready`). The run failed with `native_runner_checkpoint_owner_lost`, and same-run recovery stopped the old runnerd. The suspend was journaled but not finished. 3. Attempt 2 started a new runnerd on the same state. The welcome delivered the stale suspend. The Runner replayed it as `indeterminate` and sent `failed`. The controller acknowledged `failed`. The Runner exited with code 1: "terminal command result acknowledgement changed its durable identity". The controller had recorded the suspend as `failed`. 4. Attempt 3 found the fence for a suspend that the controller no longer listed as pending. The Runner reconciled it and exited with code 0. The server reported "runnerd exited unexpectedly with code 0", and `native_session_retry_exhausted` stopped automatic recovery. **Expected behavior** The controller acknowledges exactly the receipt that the Runner sent. The Runner accepts that acknowledgement, records `suspended`, and the next attempt starts from a clean suspended Runner. **Steps to reproduce** 1. Start a native run with a local Runner. Let the turn complete so that the server sends `runner.suspend`. 2. Freeze the server (SIGSTOP) for about 30 seconds before the suspend result arrives, so that recovery stops the Runner inside the suspend. 3. Let same-run recovery start a new Runner. It exits with "terminal command result acknowledgement changed its durable identity". The unit tests in this PR reproduce the same sequence without a live run. **Paperclip version or commit** `master` at200c6e40e2. The failing wave ran on9296c74e8dplus #20656. **Root cause** `command_result_envelope` changes the journal status `indeterminate` to `failed` on the wire. `wait_for_terminal_result_ack` compared the acknowledgement with the journal status, not with the status that it sent. The controller also treated a `failed` suspend as a refused suspend (the Runner keeps serving), but the Runner treats an indeterminate suspend as terminal. ## What Changed - Runner (`runner.rs`): a new `wire_command_status` gives the status that goes on the wire. `wait_for_terminal_result_ack` accepts the acknowledgement when its status is the wire status (`failed`) or the journal status (`indeterminate`). Command id, type, and controller sequence must still match exactly. - Controller (`durable-prp-control-plane.ts`): for a result with `execution_indeterminate`, the `command_result_ack` names `indeterminate`. Runners that shipped before this change accept only that status. The controller keeps the terminal-lifecycle fence for an indeterminate suspend, so it does not send the next command to a Runner that is exiting. The command stays `failed` in the controller journal, as before. - Runner Rust test `indeterminate_suspend_replay_accepts_the_acknowledged_wire_status`: it seeds a journaled suspend, starts the Runner against a fake controller that acknowledges `failed` and then `indeterminate`, and checks that the Runner finishes as `suspended` with no fence. Before the fix it failed with the production message. - Controller test: it checks the `indeterminate` acknowledgement and that the next queued command is not delivered on that connection. - `docs/durable-recovery.md`: it describes this case. ## Verification - Red, then green: `cargo test -p paperclip-runner-core --lib indeterminate_suspend_replay` failed with `DurableRunnerError("terminal command result acknowledgement changed its durable identity")` before the Runner change and passes after it. - Red, then green: `npx vitest run src/control-plane/durable-prp-control-plane.test.ts -t "indeterminate suspension"` failed (`status: "failed"`, expected `"indeterminate"`) before the controller change and passes after it. - `cargo test -p paperclip-runner-core --lib durable`: 152 passed. - `npx vitest run src/control-plane/durable-prp-control-plane.test.ts`: 215 passed. - `npx vitest run src/live/runnerd-codex-transport-settlement.test.ts src/live/runnerd-codex-transport-suspension.test.ts`: 30 passed. These use the real runnerd binary built from `master`, so they also cover the new controller with an old Runner. - `npx tsc --noEmit -p packages/paperclip-runner`: clean. - `pnpm verify` returned exit 2 (broad input). Required CI supplies the broad coverage. - Not run: a chaos wave. The unit tests reproduce the exact sequence at the Runner and controller boundary. ## Risks - Version mix. Old Runner with new server: the acknowledgement says `indeterminate`, which is the old Runner's journal status, so it is accepted. New Runner with old server: the acknowledgement says `failed`, which is the new Runner's wire status, so it is accepted. New with new: accepted. - No command runs again. The indeterminate suspend is never executed again; only its acknowledgement changes. The Runner still refuses a different command id, type, or controller sequence. - The Runner change takes effect only after a new Runner image. The controller change alone removes the failure for Runners that already shipped. - Remaining gap: the attempt that delivers the stale suspend still ends when that Runner exits as suspended. The next same-run attempt then starts from a clean suspended Runner. A later change can restart the Runner inside the same attempt. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>