mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) grades each task of a chaos wave with rules R1–R6 > - R5 fails a run that stays `running` with no new evidence for more than 3 minutes (`silent_dead_run`) > - In wave w-20261115T044649-s7106, task t028 failed R5. Its run was quiet for 303.6 s (05:22:13.005 to 05:27:16.647) because one DeepSeek request through OpenRouter was slow. Every sample showed an active provider turn and no tool running. The run then finished normally. > - A slow model request is not a dead run. False failures hide real failures and make the wave number wrong. > - This pull request lets R5 wait up to 10 minutes while a provider turn is active and no tool runs. It also stops the release-swap grade from calling queued wakes "never dispatched" while a live run holds them. > - The benefit is a wave grade that counts only real dead runs and real lost wakes. ## Linked Issues or Issue Description **Bug description** R5 reports `silent_dead_run` for a run that waits on one slow model request. The samples show `providerTurnActive=true` and no tool in flight for the whole gap. The run then finishes normally. Separately, the release-swap grade detail says "2 queued wake(s) never dispatched" for a task whose run was still running. Those wakes wait behind the active run by design. **Steps to reproduce** Grade the t028 samples of wave w-20261115T044649-s7106 with `gradeR5`. The result is `fail: silent_dead_run 7b51b352… 182s`. Grade a release-swap task that has a stale running run and queued wakes. The detail contains "never dispatched". **Expected behavior** R5 passes a quiet run while a provider turn is active and no tool runs, up to 10 minutes. R5 still fails a quiet run with no active provider turn after 3 minutes, and a provider turn quiet for more than 10 minutes. The grade does not call wakes undispatched while a queued or running run holds them. ## What Changed - `grader.ts`: add `PROVIDER_TURN_LIMIT_MS` (10 minutes). `deadRunningRun` uses this limit instead of the 3-minute `STALL_LIMIT_MS` when the run sample has `providerTurnActive`. A run with a tool in flight or a live process is still skipped as before. - Why 10 minutes: the observed slow request took 303.6 s. Common provider HTTP clients (the OpenAI and Anthropic SDKs) time out a request at 10 minutes. After 10 minutes the provider client has given up, so continued quiet means the run is dead, not slow. A turn whose process died also keeps `providerTurnActive` set, so the bound still catches it. - `release-swap-grade.ts`: omit the "N queued wake(s) never dispatched" note while the task has a queued, running or scheduled-retry run. The stale-run note stays. - New fixture `fixtures/t028-slow-provider-turn.json`: the real t028 samples from 05:21:30 to the run's end (116 samples). - `README.md`: document the provider-turn bound for R5 and the queued-wake rule. ## Verification - Red on `origin/master`, green with this change: - `triggers.test.ts` › R5 live evidence › "does not flag a slow model request while the provider turn is active with no tool running (w-20261115T044649-s7106/t028)" - `triggers.test.ts` › R5 live evidence › "still fails a silent run with no active provider turn, and a provider turn silent past the provider-latency bound" (guards: 4 min quiet with no provider turn fails; 9.5 min quiet with a provider turn passes; 10.5 min quiet with a provider turn fails) - `release-swap-grade.test.ts` › "does not report queued wakes as never dispatched while a run holds them (w-20261115T044649-s7106/t019)" (also checks that wakes with no live run still get the note) - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 71 files, 826 tests pass. - `npx tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean. - No wave was run. ## Risks - Low risk. The change touches only the test harness grader. - A run whose provider turn is active but whose process is truly dead now fails R5 after 10 minutes instead of 3. The fault is still caught, but later. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>