Files
PaperClipAI/tests
DottaandClaude Opus 5.5 104f323bee test(restart-chaos): credit a provably fair first-run queue wait in R1 (#22446)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos`) grades each task
of a wave against rules R1–R6
> - R1 allows the workload budget plus outage time, but it ignores the
time the first run waits for a free agent slot
> - In a staging-mix wave, 32 tasks arrive in about 2.5 minutes and each
agent has 4 slots, so the last arrivals wait 21–25 minutes even with
perfect FIFO dispatch
> - One task (t031) is the tightest in every wave. Its margin went from
+165 s and +142 s to −87 s, and the wave failed R1 for a fair wait
> - This pull request credits the first run's initial queue wait in R1
only when the harness can prove the wait was fair
> - The benefit is that a fair queue no longer fails R1, and a real
dispatch-fairness bug fails with its own reason

## Linked Issues or Issue Description

**What happened?**

In wave `w-20261120T190100-s7106`, task t031 failed
`R1:no_rest_within_budget` by 87 s. Its first assignment run sat queued
from 19:04:31Z to 19:32:43Z. Its opencode agent was at its 4-run limit
for the whole wait, and the run started in the same second that a slot
freed. The R1 budget (`1.5 × turn + 20 min`) does not include queue
wait.

**Expected behavior**

A task whose first run waited fairly for agent capacity is not marked
late because of that wait. A task whose run was overtaken by a newer
run, or that waited while a slot was idle, still fails R1.

**Steps to reproduce**

1. Run a staging-mix wave with 32 arrivals and 4 slots per agent (for
example the plan in `w-20261120T190100-s7106`).
2. Look at the last task assigned to the busiest agent.
3. See that its first run queues for more than 20 minutes and that R1
fails although every slot was busy.

**Paperclip version or commit**

`f7db7312b4` (master)

## What Changed

- `agent-slots.ts`: `agentSlotsFromRows` records each run's `createdAt`,
`invocationSource` and `retryOfRunId`. New `queueWaitFairnessFrom`
decides if a run's queue wait was fair. It returns `unfair` when a newer
fresh assignment run of the same agent started during the wait, or when
the agent had an idle slot for more than 2 s
(`QUEUE_HANDOFF_TOLERANCE_MS`) outside a server outage. Newer follow-ups
and retries rank higher in the dispatcher by design, so they do not
count as overtaking. It returns `unproven` when slot evidence,
`createdAt` or the invocation source is missing, or when the only idle
slot overlapped an outage.
- `grader.ts`: new `firstRun`, `firstRunQueuedMs` and
`firstRunQueueWait`. `gradeR1` always checks fairness (it defaults to
"no slot evidence"). When the task is over its base budget, R1 adds the
first run's initial wait minus any fault overlap if the wait is fair. A
provably unfair wait fails as `unfair_queue_wait`. An unprovable wait
(including a first run that never started) is not credited and the
detail says why. Waits of retries and later runs are never credited.
Tasks inside the base budget grade exactly as before.
- `observer.ts`: the observer does not time a task out while its first
run is still queued (up to one more budget), and it extends the deadline
once by that run's queue wait when it starts, so it watches long enough
for the grader to apply the credit.
- `wave-grade-inputs.ts`: passes the fairness check to `gradeTask`.
- `README.md`: documents the R1 queue-wait credit.
- No product code changes.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
71 files, 865 tests pass.
- `pnpm exec tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean.
- New tests in `grader.test.ts` (`R1 fair queue wait credit`):
- credits a fair 1,509 s FIFO queue wait that left the task 87 s over
the old budget
- fails as an unfair queue wait when a newer run started ahead of the
queued run
- fails as an unfair queue wait when the agent had an idle slot during
the wait
- credits the wait when a newer follow-up or retry that the dispatcher
ranks higher started ahead
  - does not call an idle slot during a server outage unfair
  - says so when the first run never started
  - allows a same-second slot hand-off within the tolerance
  - does not credit a wait it cannot prove fair
  - leaves a short queue wait unchanged
- credits only the first run's initial wait, not outage time inside it
or later retries
- New tests in `observer.test.ts`: extends the deadline once by the
first run's queue wait when it starts; does not time out while the first
run is still queued, up to one more budget.
- Offline re-grade of the saved wave `w-20261120T190100-s7106` with the
new `gradeR1` (all 32 saved R1 verdicts reproduce first):
  - No verdict changes for the other 31 tasks.
- t031 with the wave's saved evidence: `done_too_late 2291s > 2183s
(queue wait not credited: no createdAt for 4 runs …)`. This wave's
`agent-slots.json` predates `createdAt`, and 4 of the 10 runs that
started during the wait are subtask runs that the per-task wake evidence
does not cover. The grader cannot prove FIFO from the saved files, so it
does not credit the wait.
- t031 with those 4 runs taken as created earlier (as the earlier DB
diagnosis found): `pass — done after 2291s (allowed 3701s incl. 1518s
fair queue wait)`.
- New waves record `createdAt` for every run, so the check is complete
from the next wave.
- `--replay` re-executes a task on a live server. It is not a re-grade,
so I did not use it.

## Risks

- Low risk. The change is in the test harness only.
- Tasks inside their base budget grade exactly as before. Only a task
that is over budget can change verdict. An over-budget R1 detail now
also says why a queue wait was not credited; the failure signature is
unchanged.
- The fairness rule models the dispatcher coarsely: only a newer fresh
assignment run counts as overtaking. Readiness and priority are not
compared, and all harness tasks use the same priority.
- An observer can watch a task for longer, up to the first run's queue
wait. This can make a wave a little longer when queues are long.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 16:44:42 -06:00
..