mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
DottaandClaude Opus 5.5
71313207eb
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex and ACPX runs survive control-plane restarts: the runner restores the provider conversation, and the control plane continues the task from its checkpoint > - When the runner restores a conversation after process loss, it reports the turn that was active as lost on restore (`provider_turn_lost_on_restore`) > - A controller can die after the provider accepted a turn but before that turn reached the checkpoint. Then the next recovery does not know the turn that the runner reports as lost > - In a restart chaos wave with two hard kills about 20 s apart, this made the second recovery fail with "Provider turn terminal did not name the active turn". The session close then became unrecoverable > - This pull request lets recovery continue: the transport no longer stalls a new turn start behind that report, and the Codex session records the report as restored history > - The benefit is that a task continues automatically after back-to-back restarts, instead of failing and needing an operator ## Linked Issues or Issue Description **What happened?** A Codex task was resumed after a hard restart. A second hard restart came about 20 s later. The second recovery failed at once with `NativeProviderTerminalFailure: Provider turn terminal did not name the active turn.`, and automatic recovery stopped. The session close could not settle, which quarantined later work (that blast radius is fixed separately in #19532). **Steps to reproduce** 1. Start a native Codex task. Kill the control plane (SIGKILL) shortly after the provider accepts a turn and before the next checkpoint is saved. 2. Let restart recovery resume the run. Kill the control plane again while the resumed run is active. 3. Observe the second recovery. Durable evidence from the wave (runner journal of the run): - Restart 1 (17:32:34.568): the provider accepted continuation turn T2. The controller was killed 170 ms later, before T2 reached the checkpoint. - Recovery 1 started a new turn T3. The runner command result named T3 (seq 15). The restored runner then reported T2 as lost on restore (seq 20) before `turn.started` for T3 (seq 23). The transport held the T2 terminal while the T3 start was pending, and stopped pumping at it. So T3 ran in the provider (items, a tool call) but the control plane never accepted or checkpointed it. - Restart 2: recovery found no active turn. The restored runner reported T3 as lost on restore. The Codex session had never bound T3, so it failed the protocol (`turn_binding_mismatch`). **Expected behavior** Recovery continues the task after each restart (R1). **Actual behavior** The run failed with a protocol error. Its close could not prove settlement, so cleanup needed an operator. ## What Changed - `runnerd-codex-transport.ts`: while a turn start is pending, the event pump holds a terminal only when the terminal can belong to that turn. When the durable command result names the new provider turn and the terminal names a different turn, the pump delivers it and continues to the new `turn.started`. A terminal that names no turn, or any terminal before the command result, is still held. - `codex-session-notifications.ts`: a `turn/completed` with status `failed` and a recoverable `provider_turn_lost_on_restore` error, for a turn that this session never bound (not active and not terminal), is recorded as restored history with a `harness.diagnostic` (`unbound_restored_turn_lost`). The session keeps its own turn. The runner reports this cause only after it proved that the old provider process ended and restored the conversation, so that turn can produce no further work or result. - Other terminals for an unbound turn (a completion, an ordinary failure) still fail with `turn_binding_mismatch`. ## Verification - Red before the fix, real `paperclip-runnerd` and fake Codex app server: `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-restart-continuation.test.ts -t "never bound"` failed with `NativeProviderTerminalFailure: Provider turn terminal did not name the active turn.` at `codex-harness-session.ts:207` via `native-session-runtime.ts` (the same stack as the wave). - Green after the fix: that test completes the run on `provider-turn-2` with no `session.failed` and no `turn.failed`. - New driver tests in `codex-app-server-driver.terminal-replay.test.ts`: the restore-lost unbound turn is recorded (also on replay) and the session's own turn completes. A completion or ordinary failure for an unbound turn still fails the protocol. - New transport unit test for the hold rule (`does not hold a pending turn start behind another turn's restore-lost terminal`). - Existing suites pass: `src/drivers/codex/`, `src/native-session-runtime.test.ts`, `src/live/runnerd-restart-continuation.test.ts`, `src/live/runnerd-controller-outage.test.ts`, `src/live/runnerd-codex-transport-warm-attachment.test.ts`, `src/live/runnerd-codex-transport.test.ts` (turn-start cases). Three tests timed out once at host load 51 and passed when rerun alone. - `pnpm --filter @paperclipai/paperclip-runner typecheck:typescript` passes. - Live replay of the wave task: results are added in a comment below. ## Risks - The transport change applies only when the command result already named the new turn and the terminal names a different turn. A terminal for the new turn keeps the existing fence that puts `turn.accepted` first. - The session change accepts only the runner's restart-interruption cause. A lost turn's work is already in the restored conversation. The continuation turn tells the agent to reconcile unfinished work. - No profile, schema or pack contract changes. ACPX sessions that use the same Codex harness session get the same behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>