mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) grades each task of a wave against rules R1–R6 > - R1 allows the workload budget plus outage time, but it ignores the time the first run waits for a free agent slot > - In a staging-mix wave, 32 tasks arrive in about 2.5 minutes and each agent has 4 slots, so the last arrivals wait 21–25 minutes even with perfect FIFO dispatch > - One task (t031) is the tightest in every wave. Its margin went from +165 s and +142 s to −87 s, and the wave failed R1 for a fair wait > - This pull request credits the first run's initial queue wait in R1 only when the harness can prove the wait was fair > - The benefit is that a fair queue no longer fails R1, and a real dispatch-fairness bug fails with its own reason ## Linked Issues or Issue Description **What happened?** In wave `w-20261120T190100-s7106`, task t031 failed `R1:no_rest_within_budget` by 87 s. Its first assignment run sat queued from 19:04:31Z to 19:32:43Z. Its opencode agent was at its 4-run limit for the whole wait, and the run started in the same second that a slot freed. The R1 budget (`1.5 × turn + 20 min`) does not include queue wait. **Expected behavior** A task whose first run waited fairly for agent capacity is not marked late because of that wait. A task whose run was overtaken by a newer run, or that waited while a slot was idle, still fails R1. **Steps to reproduce** 1. Run a staging-mix wave with 32 arrivals and 4 slots per agent (for example the plan in `w-20261120T190100-s7106`). 2. Look at the last task assigned to the busiest agent. 3. See that its first run queues for more than 20 minutes and that R1 fails although every slot was busy. **Paperclip version or commit** `f7db7312b4` (master) ## What Changed - `agent-slots.ts`: `agentSlotsFromRows` records each run's `createdAt`, `invocationSource` and `retryOfRunId`. New `queueWaitFairnessFrom` decides if a run's queue wait was fair. It returns `unfair` when a newer fresh assignment run of the same agent started during the wait, or when the agent had an idle slot for more than 2 s (`QUEUE_HANDOFF_TOLERANCE_MS`) outside a server outage. Newer follow-ups and retries rank higher in the dispatcher by design, so they do not count as overtaking. It returns `unproven` when slot evidence, `createdAt` or the invocation source is missing, or when the only idle slot overlapped an outage. - `grader.ts`: new `firstRun`, `firstRunQueuedMs` and `firstRunQueueWait`. `gradeR1` always checks fairness (it defaults to "no slot evidence"). When the task is over its base budget, R1 adds the first run's initial wait minus any fault overlap if the wait is fair. A provably unfair wait fails as `unfair_queue_wait`. An unprovable wait (including a first run that never started) is not credited and the detail says why. Waits of retries and later runs are never credited. Tasks inside the base budget grade exactly as before. - `observer.ts`: the observer does not time a task out while its first run is still queued (up to one more budget), and it extends the deadline once by that run's queue wait when it starts, so it watches long enough for the grader to apply the credit. - `wave-grade-inputs.ts`: passes the fairness check to `gradeTask`. - `README.md`: documents the R1 queue-wait credit. - No product code changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 71 files, 865 tests pass. - `pnpm exec tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean. - New tests in `grader.test.ts` (`R1 fair queue wait credit`): - credits a fair 1,509 s FIFO queue wait that left the task 87 s over the old budget - fails as an unfair queue wait when a newer run started ahead of the queued run - fails as an unfair queue wait when the agent had an idle slot during the wait - credits the wait when a newer follow-up or retry that the dispatcher ranks higher started ahead - does not call an idle slot during a server outage unfair - says so when the first run never started - allows a same-second slot hand-off within the tolerance - does not credit a wait it cannot prove fair - leaves a short queue wait unchanged - credits only the first run's initial wait, not outage time inside it or later retries - New tests in `observer.test.ts`: extends the deadline once by the first run's queue wait when it starts; does not time out while the first run is still queued, up to one more budget. - Offline re-grade of the saved wave `w-20261120T190100-s7106` with the new `gradeR1` (all 32 saved R1 verdicts reproduce first): - No verdict changes for the other 31 tasks. - t031 with the wave's saved evidence: `done_too_late 2291s > 2183s (queue wait not credited: no createdAt for 4 runs …)`. This wave's `agent-slots.json` predates `createdAt`, and 4 of the 10 runs that started during the wait are subtask runs that the per-task wake evidence does not cover. The grader cannot prove FIFO from the saved files, so it does not credit the wait. - t031 with those 4 runs taken as created earlier (as the earlier DB diagnosis found): `pass — done after 2291s (allowed 3701s incl. 1518s fair queue wait)`. - New waves record `createdAt` for every run, so the check is complete from the next wave. - `--replay` re-executes a task on a live server. It is not a re-grade, so I did not use it. ## Risks - Low risk. The change is in the test harness only. - Tasks inside their base budget grade exactly as before. Only a task that is over budget can change verdict. An over-budget R1 detail now also says why a queue wait was not credited; the failure signature is unchanged. - The fairness rule models the dispatcher coarsely: only a newer fresh assignment run counts as overtaking. Readiness and priority are not compared, and all harness tasks use the same priority. - An observer can watch a task for longer, up to the first run's queue wait. This can make a wave a little longer when queues are long. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>