Files
PaperClipAI/tests
DottaandClaude Opus 5.5 e64872fcdb test(restart-chaos): credit a provably fair follow-up queue wait in R1 (#22452)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos`) grades each task
of a wave against rules R1–R6
> - #22446 lets R1 credit the first run's queue wait when the harness
can prove that wait was fair
> - The staging-mix shape gives each agent about 21 assignments on 4
slots, all within about 2.5 minutes. A follow-up comment that arrives
later also waits 20–26 minutes in a fair FIFO queue
> - The R1 budget (`1.5 × turn + 20 min`) cannot absorb that wait.
Recent grade summaries show a follow-up queued-wait p50 of about
450–1,100 s, so the problem is structural
> - This pull request extends the same fairness proof to the queue wait
of a run that a board follow-up comment started
> - The benefit is that a fair capacity queue no longer fails R1, while
deploy-caused waits and real dispatch-fairness bugs still fail

## Linked Issues or Issue Description

Refs #22446

**What happened?**

In wave `w-20261120T224738-s7106`, task t007 failed R1 as
`no_rest_within_budget (last status in_progress)`. Its first assignment
run ran from 22:49:15Z to 23:00:21Z and succeeded. The harness then
posted a follow-up comment, which created run `dde4d929` (invocation
source `automation`) at 23:00:21Z. That run sat queued for 1,558 s,
until 23:26:19Z, behind its agent's older assignments. It succeeded at
23:27:45Z, 17 s after the observer's deadline, so the observer stopped
watching. All 32 tasks finished hands-off, with 0 Retry banners, 0
duplicates and clean swaps.

**Expected behavior**

A task whose follow-up run waited fairly for agent capacity is not
marked late because of that wait. A follow-up run that a newer fresh
assignment overtook, or that waited while a slot was idle, still fails
R1. Waits of retries and recovery runs still count against the budget.

**Steps to reproduce**

1. Run a staging-mix wave with 32 arrivals and 4 slots per agent (for
example the plan in `w-20261120T224738-s7106`).
2. Find a task whose follow-up comment is posted while its first run
works.
3. See that the follow-up run queues for more than 20 minutes after the
first run ends, and that R1 fails although every slot was busy.

**Paperclip version or commit**

`104f323bee` (master, after #22446)

## What Changed

- `grader.ts`: `runSelection` chooses the runs whose queue wait R1 may
credit. The rules, in order:
1. If any run on the task has no parseable `createdAt`, nothing is
credited (`no createdAt for <ids>, so runs cannot be ordered`).
2. The first run (earliest `createdAt`) is a candidate unless it is a
retry.
3. For each harness follow-up comment (`followUp`,
`stagingMix.followUp`, `stagingMix.afterEndFollowUps`): without a send
time, nothing is credited for it (`no send time for follow-up comment
<id>`). Otherwise the candidate is the earliest run that carries the
comment, was created at or after the send time (`followUp.sentAt ?? at`;
the staging-mix `at` is taken before the request), and was not cancelled
before it executed (cancelled with no start, or a pre-dispatch
cancellation such as `continuation_task_ownership_changed`).
4. That run counts only if it is not a retry, the previous executed run
succeeded and finished at or before its `createdAt`, and the wake
evidence shows a comment wake reason (`issue_commented`,
`issue_comment_mentioned`, `issue_reopened_via_comment`) and no
continued or interrupted run. Without wake evidence nothing is credited
(`no wake evidence for follow-up run <id>`).
5. Each candidate's wait (`createdAt` to `startedAt`, clipped to the
task's `createdAt..restAt`, minus outage time) is credited only when
`queueWaitFairnessFrom` proves it fair. Any provably unfair wait fails
as `unfair_queue_wait` and names the run. An unproven wait is not
credited, and the detail says why.
- `grader.ts`: `gradeR1` adds the credited waits to the allowed time. A
candidate that never started gives `<kind> <id> never started`.
- `wave-grade-inputs.ts`, `types.ts`: `prepare` now adds `runWakes`
(each run's wake reason and continued run ids, from
`wake-evidence/<task>.json`) to the record before grading.
- `observer.ts`: the observer holds the timeout while any credited run
is queued (up to one more budget). It extends the deadline once per
credited run by that run's wait when the run starts.
- `README.md`: documents the follow-up credit.
- No product code changes.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
71 files, 884 tests pass.
- `pnpm exec tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean.
- New tests in `grader.test.ts` (`R1 follow-up queue wait credit`):
- credits a fair 1,558 s follow-up queue wait behind the agent's older
assignments
- fails as unfair when a newer fresh assignment started ahead of the
follow-up
  - fails as unfair when a slot sat idle while the follow-up was queued
  - does not credit the wait of a retry or a recovery run
- does not credit a follow-up run created while the previous run was
still running
- credits a fair wait of a run woken by an after-end follow-up, but not
after a recovery
  - refuses follow-up credit when an earlier run has no createdAt
- does not credit a recovery run that carries the comment after a
success
- reads each run's wake reason and continued runs from the wake evidence
- does not credit a run created before its follow-up comment was posted
- credits a run created after the comment request was sent but before it
returned
  - credits nothing when the earliest run has no createdAt
- credits the first executed carrier, never a run cancelled before it
executed
  - refuses follow-up credit without wake evidence
  - credits only the part of a wait inside the task's own lifetime
  - does not credit outage time inside the follow-up's wait
  - sums the fair waits of the first run and a follow-up
- New tests in `observer.test.ts`:
- waits while a follow-up run is queued and extends the deadline by its
queue wait once it starts (fails before the change)
  - does not hold the deadline for a queued run that a recovery created
- Offline re-grade of the saved wave `w-20261120T224738-s7106` with a
scratch script (not committed). The old `gradeR1` reproduces all 32
saved R1 details first. Records go through the new `prepare`, so
`runWakes` come from the saved wake evidence (t007: `dde4d929` has wake
reason `issue_commented` and no continued run).
  - The other 31 tasks: no R1 verdict changes (31 pass).
- t007 as saved: still `no_rest_within_budget`, because the old observer
timed out 17 s before the task finished.
- t007 as the new observer records it (deadline extended by the 1,558 s
wait to 23:53:26Z, task done at 23:27:45Z): `pass — done after 2310s
(allowed 3851s incl. 1558s fair queue wait)`. The wave's
`agent-slots.json` has `createdAt` for every run, so the proof is
complete.
- The same t007 record without slot evidence: `done_too_late 2310s >
2293s (queue wait not credited: no agent slot evidence …)`.

## Risks

- Low risk. The change is in the test harness only.
- Tasks inside their base budget grade exactly as before.
- A follow-up wait is credited only with wake evidence. Waves without
`wake-evidence/<task>.json` get no follow-up credit, and the R1 detail
says so.
- The observer cannot see wake reasons while it runs, so it may watch a
little longer than the grader later credits. The grader decides the
verdict.
- An observer can watch a task for longer, up to each credited run's
queue wait. This can make a wave a little longer when queues are long.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 18:41:34 -06:00
..