mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness runs agent tasks across server swaps and keeps evidence for each failure > - Wave `w-20261112T181948-s7101` had two "run started after the task was done" cases that could not be traced > - The wave deleted its database before anyone saved which wake started those runs, which server created them, or how the task status changed > - The evidence bundle also kept only the last 200 events of a run, so it dropped the first 291 events of a Retry run, including its wake payload > - The staging-mix wording made one Codex agent refuse to redo an interrupted round, and the task blocked on a harness artefact > - This pull request saves that evidence for every task, keeps a run's first events, and rewords the round rules without changing the grade ## Linked Issues or Issue Description Refs #21125, #21168. **What existing behavior does this improve?** Restart chaos evidence capture and the staging-mix task wording. **Subsystem affected** Test harness only: `tests/restart-chaos/`. No product code changes. **Current behavior** - The harness does not save a task's wake rows, run context snapshots or status history before the wave database is deleted. - `evidence.ts` keeps only the last 200 events per run. - The task tells the agent "do not skip, repeat or shorten any" round, and the follow-up says "do not restart any". An agent read that as "never re-run an interrupted round". **Proposed behavior** - Save, for every task, the wake rows, run snapshots, status changes and serving server generations. - Keep a run's first 400 and last 200 events, with a marker between them. - Say plainly that every round must appear in the ledger exactly once, that an interrupted round must be run again, and that a round already in the ledger must never be run again. **Reason and benefit** Wake and run origins can be traced after the wave, and the wording no longer creates failures that come only from the harness. ## What Changed - New `wake-evidence.ts`, called once at the end of each wave while the database still exists. It writes `wake-evidence/<task>.json` for every task. Each file holds: - for each run: its `context_snapshot`, invocation source, trigger and wakeup request id; - the `agent_wakeup_requests` rows for the issue and for each run: reason, source, trigger detail, requesting actor, idempotency key, payload and times; - the issue's status changes from `activity_log`. Each wake and run names the server generations alive when it was created (`servingAtCreate`): pid, boot id, epoch and process space from `controller_instances`, plus the harness's primary or secondary role during an overlap. Each run also names its owning controller (`owner`). `wake-evidence/servers.json` lists every controller and harness server window. Known secret values are redacted. - `evidence.ts` keeps all of a run's events, or the first 400 and the last 200 with an `omittedEvents` marker between them. - `staging-mix.ts` only changes the wording; the grading is unchanged. The new text: - every round must appear in the ledger file exactly once, with the hash the command printed; - before a round, read the ledger; a round whose line is there is finished and must never run again; - a round interrupted before its line was written must be run again. The follow-up comment says the same. - `main.ts` gains one call; the file stays at 1,000 lines. README updated. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 59 files, 711 tests pass. That includes a real embedded PostgreSQL test that writes a wake, a run, a status change and a controller row, then checks the evidence file. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Unit tests cover the serving-generation join during an overlap, the per-issue join, the first 400 and last 200 events of a 1,291-event run, and the new wording. - Not run: a live wave with this change. ## Risks - Low risk. Test harness only. - Evidence capture is one set of reads per wave. A failure is recorded as `wake_evidence_failed` and does not fail the wave. - Known master CI failures, not touched here: the storage S3 multipart throttle test and the Runner native-distribution-integrity test. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>