Files
PaperClipAI/tests
DottaandClaude Opus 5.5 9ebc276b42 test(restart-chaos): let R5 wait out a slow model request and stop calling held wakes undispatched (#21730)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos`) grades each task
of a chaos wave with rules R1–R6
> - R5 fails a run that stays `running` with no new evidence for more
than 3 minutes (`silent_dead_run`)
> - In wave w-20261115T044649-s7106, task t028 failed R5. Its run was
quiet for 303.6 s (05:22:13.005 to 05:27:16.647) because one DeepSeek
request through OpenRouter was slow. Every sample showed an active
provider turn and no tool running. The run then finished normally.
> - A slow model request is not a dead run. False failures hide real
failures and make the wave number wrong.
> - This pull request lets R5 wait up to 10 minutes while a provider
turn is active and no tool runs. It also stops the release-swap grade
from calling queued wakes "never dispatched" while a live run holds
them.
> - The benefit is a wave grade that counts only real dead runs and real
lost wakes.

## Linked Issues or Issue Description

**Bug description**
R5 reports `silent_dead_run` for a run that waits on one slow model
request. The samples show `providerTurnActive=true` and no tool in
flight for the whole gap. The run then finishes normally. Separately,
the release-swap grade detail says "2 queued wake(s) never dispatched"
for a task whose run was still running. Those wakes wait behind the
active run by design.

**Steps to reproduce**
Grade the t028 samples of wave w-20261115T044649-s7106 with `gradeR5`.
The result is `fail: silent_dead_run 7b51b352… 182s`. Grade a
release-swap task that has a stale running run and queued wakes. The
detail contains "never dispatched".

**Expected behavior**
R5 passes a quiet run while a provider turn is active and no tool runs,
up to 10 minutes. R5 still fails a quiet run with no active provider
turn after 3 minutes, and a provider turn quiet for more than 10
minutes. The grade does not call wakes undispatched while a queued or
running run holds them.

## What Changed

- `grader.ts`: add `PROVIDER_TURN_LIMIT_MS` (10 minutes).
`deadRunningRun` uses this limit instead of the 3-minute
`STALL_LIMIT_MS` when the run sample has `providerTurnActive`. A run
with a tool in flight or a live process is still skipped as before.
- Why 10 minutes: the observed slow request took 303.6 s. Common
provider HTTP clients (the OpenAI and Anthropic SDKs) time out a request
at 10 minutes. After 10 minutes the provider client has given up, so
continued quiet means the run is dead, not slow. A turn whose process
died also keeps `providerTurnActive` set, so the bound still catches it.
- `release-swap-grade.ts`: omit the "N queued wake(s) never dispatched"
note while the task has a queued, running or scheduled-retry run. The
stale-run note stays.
- New fixture `fixtures/t028-slow-provider-turn.json`: the real t028
samples from 05:21:30 to the run's end (116 samples).
- `README.md`: document the provider-turn bound for R5 and the
queued-wake rule.

## Verification

- Red on `origin/master`, green with this change:
- `triggers.test.ts` › R5 live evidence › "does not flag a slow model
request while the provider turn is active with no tool running
(w-20261115T044649-s7106/t028)"
- `triggers.test.ts` › R5 live evidence › "still fails a silent run with
no active provider turn, and a provider turn silent past the
provider-latency bound" (guards: 4 min quiet with no provider turn
fails; 9.5 min quiet with a provider turn passes; 10.5 min quiet with a
provider turn fails)
- `release-swap-grade.test.ts` › "does not report queued wakes as never
dispatched while a run holds them (w-20261115T044649-s7106/t019)" (also
checks that wakes with no live run still get the note)
- `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 71
files, 826 tests pass.
- `npx tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean.
- No wave was run.

## Risks

- Low risk. The change touches only the test harness grader.
- A run whose provider turn is active but whose process is truly dead
now fails R5 after 10 minutes instead of 3. The fault is still caught,
but later.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 01:02:31 -06:00
..