mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 14:19:00 +01:00
DottaandClaude
4158799d1b
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos`) tests that agent work survives faults. > - The harness has two fault families: control-plane restarts (K*) and storage faults (ST*). > - On 2026-11-09, two production failures came from a third family. The provider was alive, but a protocol message was late or missing (#20330, #20336). > - The harness could not produce that family, so it did not catch these failures. > - This pull request adds provider message faults (PV1 delay, PV2 drop, PV3 duplicate) on the real path between the runner and the Codex app-server. It also adds a saved-question scenario (Q1) that uses the faults. > - The benefit: the harness can now find this class of failure, and it fails against both pre-fix commits. ## Linked Issues or Issue Description Refs #20330 Refs #20336 **What happened?** A Codex interruption terminal that arrived after 18 s failed a governed wait (#20330). A dropped terminal after a saved handoff failed shutdown settlement with "Couldn't finish this attempt" (#20336). The chaos harness could not inject either fault. **Expected behavior** The harness can delay, drop or duplicate a chosen provider protocol message for one task. A scenario fails on the pre-fix product code and passes on master. **Steps to reproduce** Run the wave commands in Verification. Run them on master and in a worktree at the pre-fix commit with `tests/restart-chaos` copied from this branch. ## What Changed - `provider-faults.ts` and `provider-shim/`: the PV1–PV3 fault family. - A resolve hook goes into the wave's server through `NODE_OPTIONS`. It answers only the `@openai/codex/package.json` lookup that product code uses to find the local Codex. It returns a stand-in package in the instance temp root, and then removes itself from `NODE_OPTIONS`. - The stand-in `codex` bin is a stdio shim. It starts the real Codex with the same arguments and relays JSON lines unchanged. It changes only the message that the task's rule names: the `turn/completed` after a `turn/interrupt` (`terminal`) or the response to `turn/interrupt` (`interrupt_ack`). Each rule applies once per task. The task nonce binds the rule to the task. - Every decision goes to `<wave>/provider-faults.jsonl`. A PV task fails `provider_fault_applied` when its fault did not happen. - There is no product change. Thus the same harness runs against older product commits. - `question-handoff.ts`: Q1 workload. The agent saves one question with `request_human_input`. After the asking run settles, the board answers through the respond route. The oracle checks that the question was asked once, the asking run succeeded, no run failed, nothing ran before the answer, and the brief has the answer. - PV faults run only with `--env local --adapters codex`, only with workloads that interrupt a turn (Q1, S1, G1), and not in the same wave as restart faults. A PV1 delay passes only when the held message was released. - Registry edits: `--faults` accepts PV1–PV3. New options are `--provider-fault-message` and `--provider-fault-delays` (a sweep, cycled across tasks). The edits are in `types.ts`, `config.ts`, `scheduler.ts`, `workloads.ts`, `observer.ts` (Q1 board step) and `main.ts` (install the seam and add the check). - README section with commands, the seam, and what is not covered. ## Verification - Unit tests: `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`. 23 files and 279 tests pass on the final head. The new tests drive the real shim against a stand-in provider for each kind (delay, drop, duplicate, interrupt ack, once per task, held message discarded on exit, a multibyte character split across writes). They also run the resolve hook in a child Node process. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (real Runner, real Codex 0.160.0, gpt-6.1-sol, macOS arm64): Pre-fix runs use a worktree at the merge parent with `tests/restart-chaos` copied from this branch. Common arguments: `--env local --adapters codex --workloads Q1`. | Scenario | Product code | Wave | Result | | --- | --- | --- | --- | | 1. PV1, terminal 18 s late (`--faults PV1 --provider-fault-delays 18 --tasks 1 --seed 101`) | master (`8004ad069e`) | `w-20261110T000333-s101` | pass: terminal held 18 s and released, asking run `succeeded`, one question, brief has the answer | | 1. same | `34c1e6469c` (parent of #20330 merge) | `w-20261110T014734-s101` | **fail**: Codex stopped at 14.9 s with the terminal still held, asking run `failed native_provider_terminal_failed`, banner "Couldn’t finish this attempt. Your message is saved.", Retry needed (R3) | | 1. sweep 5/14/25/29 s (`--faults PV1 --provider-fault-delays 5,14,25,29 --concurrency 2 --tasks 4 --seed 102`) | master | `w-20261110T015426-s102` | 4/4 pass | | 1. same sweep | `34c1e6469c` | `w-20261110T015708-s102` | 5 s and 14 s pass; **25 s and 29 s fail** (`native_provider_terminal_failed`, Retry needed): the old 15 s drain | | 2. PV2, terminal dropped (`--faults PV2 --tasks 1 --seed 103`) | master | `w-20261110T015239-s103` | pass: terminal dropped, asking run `succeeded` through verified shutdown, one question, nothing ran before the answer | | 2. same | `a67b936885` (parent of #20336 merge) | `w-20261110T015010-s103` | **fail**: asking run `failed native_provider_terminal_failed`, banner "Couldn’t finish this attempt", Retry needed (R3) | The first master wave exited 1 for one cold `agent_boot` speed sample (8.6 s against a 7 s p95 budget), before any fault, while a preflight build ran in parallel. Its task passed. No new product failure was found on master. ## Risks - Test-only. No product code changes. - The shim adds one Node process in front of Codex for PV waves only. Other waves do not change. - The nonce binds a rule to a Codex process. If one Codex process serves two tasks, the second task's traffic can bind it too. Q1 tasks use separate sessions. - Claude is not covered. Claude runs through the ACPX sidecar against a hash-verified provider pack, and its ACP protocol has no interrupt acknowledgement. A Claude shim needs a different seam. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use, code edits and local execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>