mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
DottaandClaude Opus 5.5
a65626a2a2
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (tests/restart-chaos) kills and restarts the control plane during real agent tasks, and grades every task on reliability rules R1–R6 plus a task oracle > - Dotta's bar for today's campaign: restarts must continue on their own (Retry is only a fallback), the system must stay fast (speed budgets in `native-speed-budgets.ts`), and realistic scenarios must be covered > - The harness graded a task that needed a user Retry as a pass, graded speed only in crash-point mode, and counted a K4 controller hold as slow TTFT. It also had no multi-agent scenario > - This pull request closes those four gaps > - The benefit is that every chaos wave now fails on a needed Retry and on a real speed regression, but not on the fault's own outage ## Linked Issues or Issue Description Refs #19831 (durable ownership plan) and the restart chaos plan (`doc/plans/2026-11-06-control-plane-restart-chaos.md`). Used by today's campaign waves: - w-20261108T134237-s9401 re-graded to 104/107 under the Retry rule; - w-20261108T153612-s9401: 100/101, with a Speed section showing 6 breaches after outage removal. ## What Changed - **R3 Retry rule** (`grader.ts`): a task that needed a Retry fails R3 as `retry_needed (<failed runs' error codes>)`, and a second click fails as `retry_needed_<n>_clicks`. The harness still clicks Retry so the task can finish and be graded. - **Per-wave speed grade** (new `wave-speed.ts`, wired in `main.ts`): - It reads every task run's run log through the API and reports TTFT (from run created) and agent boot p50/p95 per adapter and start kind, restart recovery (ready → the run's next provider event), and single-run `run.performance.budget_exceeded` events, against the local budgets. - A breach fails the wave. - Output goes to `speed.json` and a Speed section in `SUMMARY.md`. Remote environments report numbers with no budget. - **Outage removal**: each performance span is shortened by its overlap with fault outage windows (fault start to replacement server ready) before it is judged. The summary reports how many spans and how much time were removed. - **M1 workload** (multi-agent delegation): - The parent task creates two subtasks, part 1 delegated to the other Chaos Lab agent. - Each subtask saves a marked document, and the parent writes a summary of both before it completes. - The observer collects the subtasks and their documents, and the oracle checks the parent links, assignees, statuses and markers. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 14 files, 163 tests pass. New tests cover the M1 grade, the R3 Retry rule (one click and two clicks), the wave speed grade (budget judgement, restart recovery, remote no-budget, fault→run attribution) and outage removal. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Real waves: - The R3 rule re-graded w-20261108T134237-s9401 to 104/107. - w-20261108T153612-s9401 produced the Speed section: 923 spans adjusted, 2,452 s of outage removed, 6 breaches. - M1 has not yet run in a live wave. It's next in the campaign rotation. ## Risks - Waves that passed before can now fail: on a needed Retry, or on a speed breach. That is the intended bar. - Speed is judged at wave concurrency on a shared host, which is noisier than the idle baseline the budgets came from. Breaches are reported per row, so a reviewer can see the margin. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>