mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 23:08:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness grades each task after deploy-like restarts, and the `completion_once` rule finds tasks that an agent completed twice > - A follow-up comment can arrive while a task runs. The run that answers it can complete the task again, and the grader exempts that run > - The grader did not exempt a recovery continuation of that answering run when the answering run failed before it committed a completion > - So the grader reported a duplicate completion for correct product behavior, and the wave headline counted a false duplicated side effect and an extra turn > - This pull request makes a continuation (a run with `retryOfRunId`) of an answering run that never committed a completion count as the answering run > - The benefit is that wave results show real duplicates only, and a real second completion after a committed one still fails ## Linked Issues or Issue Description **What happened?** In a restart-chaos staging-mix wave, task t128 (opencode, fault K13) graded `completion_once` as failed with the detail "duplicate completion: runs 38a8772f, 833314ed each reported completion". The wake evidence shows this sequence: - Run 38a8772f completed the task (accepted disposition `done`) and posted the DONE comment. - The follow-up comment arrived before that completion. Its first two runs (5707e93a and 2c1d364e) failed before a provider turn, so they did not receive it. - Run c5f9a5a4 received the follow-up and started a provider turn. It then failed with `native_session_retry_exhausted` and did not commit a completion. - Execution recovery started run 833314ed with `retryOfRunId` = c5f9a5a4 (wake reason `issue_recovery_action_restored`). That run applied the follow-up and committed `done`. The grader marked only c5f9a5a4 as the answering run. Run 833314ed was a continuation of it, but the grader counted it as a second completion and as an extra turn. **Expected behavior** A continuation of the answering run must count as the answering run when the answering run did not commit a completion. A continuation that completes again after the answering run committed its completion must still fail `completion_once`. **Steps to reproduce** 1. Use the t128 wake evidence shape: a first run that completes, two follow-up runs that fail before a provider turn, an answering run that fails without a completion, and a recovery run with `retryOfRunId` set to the answering run that completes. 2. Grade it with `gradeStagingMix` through `evidenceCompletion` or `completionEvidence`. 3. Before this change, `completion_once` fails with "duplicate completion: runs 38a8772f, 833314ed each reported completion". ## What Changed - `tests/restart-chaos/follow-up-delivery.ts`: add `continuedRunId` (reads `retryOfRunId` from the run context or the wake payload) and `followUpAnsweringRuns`. The new function returns the runs that answered a follow-up, plus each continuation of an answering run that did not commit a `done` result. - `tests/restart-chaos/staging-mix-retry.ts`: `evidenceCompletion` and `runsAfterDone` use `followUpAnsweringRuns` to set `answersFollowUp` and to separate answered runs from extra turns. `followUpsAnsweredLater` and the wait statistics are not changed, so one follow-up still counts as one answer. - `tests/restart-chaos/wake-evidence.ts`: `completionEvidence` (the record that the wave grade uses) uses the same function. - `tests/restart-chaos/staging-mix-retry.test.ts`: add a t128 test group from the real wave evidence. One test shows the continuation passes. One test shows a continuation after a committed answering completion still fails. ## Verification - Red before the fix: `passes when the continuation of an answering run that never committed completes the follow-up` failed (`completionRunIds` returned both 38a8772f and 833314ed). - Green after the fix: both new tests pass, and all existing truth-table and follow-up tests pass. - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 71 files, 844 tests passed. - `tsc -p tests/restart-chaos/tsconfig.json --noEmit`: passed. - I regraded the real t128 wake evidence and comments with the fixed functions. The result is one completer (38a8772f), no extra turns, and answered runs c5f9a5a4 and 833314ed. - `pnpm verify` exited 2 because `follow-up-delivery.ts` is a global CI input. Required CI supplies the broad coverage. ## Risks - Low risk. The change is in the restart-chaos test harness only. It does not change product code. - The exemption needs an explicit `retryOfRunId` link to the answering run. A recovery run with no link (for example the t005 case) still fails, as before. - If the answering run committed `done`, its continuation is not exempt, so a real second completion still fails. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>