mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
test(restart-chaos): let R5 wait out a slow model request and stop calling held wakes undispatched (#21730)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) grades each task of a chaos wave with rules R1–R6 > - R5 fails a run that stays `running` with no new evidence for more than 3 minutes (`silent_dead_run`) > - In wave w-20261115T044649-s7106, task t028 failed R5. Its run was quiet for 303.6 s (05:22:13.005 to 05:27:16.647) because one DeepSeek request through OpenRouter was slow. Every sample showed an active provider turn and no tool running. The run then finished normally. > - A slow model request is not a dead run. False failures hide real failures and make the wave number wrong. > - This pull request lets R5 wait up to 10 minutes while a provider turn is active and no tool runs. It also stops the release-swap grade from calling queued wakes "never dispatched" while a live run holds them. > - The benefit is a wave grade that counts only real dead runs and real lost wakes. ## Linked Issues or Issue Description **Bug description** R5 reports `silent_dead_run` for a run that waits on one slow model request. The samples show `providerTurnActive=true` and no tool in flight for the whole gap. The run then finishes normally. Separately, the release-swap grade detail says "2 queued wake(s) never dispatched" for a task whose run was still running. Those wakes wait behind the active run by design. **Steps to reproduce** Grade the t028 samples of wave w-20261115T044649-s7106 with `gradeR5`. The result is `fail: silent_dead_run 7b51b352… 182s`. Grade a release-swap task that has a stale running run and queued wakes. The detail contains "never dispatched". **Expected behavior** R5 passes a quiet run while a provider turn is active and no tool runs, up to 10 minutes. R5 still fails a quiet run with no active provider turn after 3 minutes, and a provider turn quiet for more than 10 minutes. The grade does not call wakes undispatched while a queued or running run holds them. ## What Changed - `grader.ts`: add `PROVIDER_TURN_LIMIT_MS` (10 minutes). `deadRunningRun` uses this limit instead of the 3-minute `STALL_LIMIT_MS` when the run sample has `providerTurnActive`. A run with a tool in flight or a live process is still skipped as before. - Why 10 minutes: the observed slow request took 303.6 s. Common provider HTTP clients (the OpenAI and Anthropic SDKs) time out a request at 10 minutes. After 10 minutes the provider client has given up, so continued quiet means the run is dead, not slow. A turn whose process died also keeps `providerTurnActive` set, so the bound still catches it. - `release-swap-grade.ts`: omit the "N queued wake(s) never dispatched" note while the task has a queued, running or scheduled-retry run. The stale-run note stays. - New fixture `fixtures/t028-slow-provider-turn.json`: the real t028 samples from 05:21:30 to the run's end (116 samples). - `README.md`: document the provider-turn bound for R5 and the queued-wake rule. ## Verification - Red on `origin/master`, green with this change: - `triggers.test.ts` › R5 live evidence › "does not flag a slow model request while the provider turn is active with no tool running (w-20261115T044649-s7106/t028)" - `triggers.test.ts` › R5 live evidence › "still fails a silent run with no active provider turn, and a provider turn silent past the provider-latency bound" (guards: 4 min quiet with no provider turn fails; 9.5 min quiet with a provider turn passes; 10.5 min quiet with a provider turn fails) - `release-swap-grade.test.ts` › "does not report queued wakes as never dispatched while a run holds them (w-20261115T044649-s7106/t019)" (also checks that wakes with no live run still get the note) - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 71 files, 826 tests pass. - `npx tsc -p tests/restart-chaos/tsconfig.json --noEmit`: clean. - No wave was run. ## Risks - Low risk. The change touches only the test harness grader. - A run whose provider turn is active but whose process is truly dead now fails R5 after 10 minutes instead of 3. The fault is still caught, but later. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6400698a8e
commit
9ebc276b42
6 files changed
+46
-4
No files matched your search
@@ -661,7 +661,7 @@ cap. The quiet period lets a continuation the product wakes after a run ends (fo
|
||||
subtasks finish) start before the read, so the task is graded by that continuation, not as `nothing pending`. A task still running at the cap is not counted in the number; the headline
|
||||
reports how many with the age of their last activity, and the wave fails. The grade uses the same 5-minute threshold:
|
||||
after the drain, a run with no event for 5 minutes, a queued wake, a scheduled retry or a working notice with no
|
||||
progressing run behind it is not pending work, so the task is `nothing pending`. The single read then covers every task of the wave's company that had a run (probes and agent-created subtasks included), the
|
||||
progressing run behind it is not pending work, so the task is `nothing pending`. Queued wakes behind a queued or running run wait for that run by design, so the detail does not call them never dispatched. The single read then covers every task of the wave's company that had a run (probes and agent-created subtasks included), the
|
||||
way the board shows it:
|
||||
|
||||
| Verdict | When |
|
||||
@@ -2155,7 +2155,7 @@ re-arms the task's own triggered faults instead of using their offsets.
|
||||
temporary instance is deleted.
|
||||
- Faults are serialized; each one hits every task running at that moment.
|
||||
- A `todo` or `in_progress` task with no live run and no named wait is not a rest. The observer keeps watching it until it resumes or passes the R5 stall limit.
|
||||
- R5 also fails a run that stays `running` with no live evidence (no new run event, `lastUsefulActionAt` or `execution.lastConfirmedActivityAt`, and no start) for more than 3 minutes after the server is ready. A `running` status alone is not proof of life. A quiet run is still live while a tool it started has not ended (for example a 4-minute `bench.sh`) or while its recorded local runner process still runs with the same PID and start time.
|
||||
- R5 also fails a run that stays `running` with no live evidence (no new run event, `lastUsefulActionAt` or `execution.lastConfirmedActivityAt`, and no start) for more than 3 minutes after the server is ready. A `running` status alone is not proof of life. A quiet run is still live while a tool it started has not ended (for example a 4-minute `bench.sh`) or while its recorded local runner process still runs with the same PID and start time. While a provider turn is active and no tool runs, one model request is outstanding: R5 allows up to 10 minutes of quiet before it counts a dead run. A slow OpenRouter request took 303.6 s in w-20261115T044649-s7106/t028, and common provider clients time out a request at 10 minutes.
|
||||
- The wave fingerprints the checkout and `packages/paperclip-runner/dist` at start and after every fault. A build, install or branch switch during a wave would make later restarts run different code, so it fails the wave as a harness error (`worktree_changed_mid_wave`). The baseline is taken after the harness's own setup (preflight rebuilds provider entrypoints) and covers the tracked diff, untracked file contents, the runner `dist` and the runnerd binary. Build before the wave, never during it.
|
||||
- `faults.json` and `task-faults.json` (each task's launch time and the faults that crossed it) are written after every fault, so `--replay` of a task in an interrupted or wedged wave, or a storm task, re-fires the faults that crossed it at the same offsets.
|
||||
- A fault that fails to complete fails the wave as a harness error (exit 2), even when recovery brings the server back.
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
{"source":"w-20261115T044649-s7106/t028: one model request outstanding for 303.6 s with no tool running; the run then finished normally","samples":[{"at":1794720090451,"reachable":true,"serverReady":true,"issueStatus":"in_progress","interruption":{"state":"working","canRetry":false,"retryWait":false,"workspaceFailure":null,"observedRunId":"7b51b352-85f1-4b6c-9224-c6170b8d4231","userAction":null},"activeRecoveryAction":false,"recoveryOwner":null,"executionBlocker":false,"scheduledRetry":false,"runs":[{"id":"7b51b352-85f1-4b6c-9224-c6170b8d4231","status":"running","createdAt":"2026-11-15T04:50:01.510Z","startedAt":"2026-11-15T05:09:16.731Z","finishedAt":null,"errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":null,"lastEvidenceAt":"2026-11-15T05:20:42.672Z","toolInFlight":true,"providerTurnActive":true}]},{"at":1794720093681,"reachable":true,"serverReady":true,"issueStatus":"in_progress","interruption":{"state":"working","canRetry":false,"retryWait":false,"workspaceFailure":null,"observedRunId":"7b51b352-85f1-4b6c-9224-c6170b8d4231","userAction":null},"activeRecoveryAction":false,"recoveryOwner":null,"executionBlocker":false,"scheduledRetry":false,"runs":[{"id":"7b51b352-85f1-4b6c-9224-c6170b8d4231","status":"running","createdAt":"2026-11-15T04:50:01.510Z","startedAt":"2026-11-15T05:09:16.731Z","finishedAt":null,"errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":null,"lastEvidenceAt":"2026-11-15T05:20:42.672Z","toolInFlight":true,"providerTurnActive":true}]},{"at":1794720096928,"reachable":true,"serverReady":true,"issueStatus":"in_progress","interruption":{"state":"working","canRetry":false,"retryWait":false,"workspaceFailure":null,"observedRunId":"7b51b352-85f1-4b6c-9224-c6170b8d4231","userAction":null},"activeRecoveryAction":false,"recoveryOwner":null,"executionBlocker":false,"scheduledRetry":false,"runs":[{"id":"7b51b352-85f1-4b6c-9224-c6170b8d4231","status":"running","createdAt":"2026-11-15T04:50:01.510Z","startedAt":"2026-11-15T05:09:16.731Z","finishedAt":null,"errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":null,"lastEvidenceAt":"2026-11-15T05:20:42.672Z","toolInFlight":true,"providerTurnActive":true}]},{"at":1794720100201,"reachable":true,"serverReady":true,"issueStatus":"in_progress","interruption":{"state":"working","canRetry":false,"retryWait":false,"workspaceFailure":null,"observedRunId":"7b51b352-85f1-4b6c-9224-c6170b8d4231","userAction":null},"activeRecoveryAction":false,"recoveryOwner":null,"executionBlocker":false,"scheduledRetry":false,"runs":[{"id":"7b51b352-85f1-4b6c-9224-c6170b8d4231","status":"running","createdAt":"2026-11-15T04:50:01.510Z","startedAt":"2026-11-15T05:09:16.731Z","finishedAt":null,"errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":null,"lastEvidenceAt":"2026-11-15T05:20:42.672Z","toolInFlight":true,"providerTurnActive":true}]},{"at":1794720103431,"reachable":true,"serverReady":true,"issueStatus":"in_progress","interruption":{"state":"working","canRetry":false,"retryWait":false,"workspaceFailure":null,"observedRunId":"7b51b352-85f1-4b6c-9224-c6170b8d4231","userAction":null},"activeRecoveryAction":false,"recoveryOwner":null,"executionBlocker":false,"scheduledRetry":false,"runs":[{"id":"7b51b352-85f1-4b6c-9224-c6170b8d4231","status":"running","createdAt":"2026-11-15T04:50:01.510Z","startedAt":"2026-11-15T05:09:16.731Z","finishedAt":null,"errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":null,"lastEvidenceAt":"2026-11-15T05:20:42.672Z","toolInFlight":true,"providerTurnActive":true}]},{"at":1794720106662,"reachable":true,"serverReady":true,"issueStatus":"in_progress","interruption":{"state":"working","canRetry":false,"retryWait":false,"workspaceFailure":null,"observedRunId":"7b51b352-85f1-4b6c-9224-c6170b8d4231","userAction":null},"activeRecoveryAction":false,"recoveryOwner":null,"executionBlocker":false,"scheduledRetry":false,"runs":[{"id":"7b51b352-85f1-4b6c-9224-c6170b8d4231","status":"running","createdAt":"2026-11-15T04:50:01.510Z","startedAt":"2026-11-15T05:09:16.731Z","finishedAt":null,"errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":null,"lastEvidenceAt":"2026-11-15T05:20:42.672Z","toolInFlight":true,"providerTurnActive":true}]},{"at":1794720109887,"reachable":true,"serverReady":true,"issueStatus":"in_progress","interruption":{"state":"working","canRetry":false,"retryWait":false,"workspaceFailure":null,"observedRunId":"7b51b352-85f1-4b6c-9224-c6170b8d4231","userAction":null},"activeRecoveryAction":false,"recoveryOwner":null,"executionBlocker":false,"scheduledRetry":false,"runs":[{"id":"7b51b352-85f1-4b6c-9224-c6170b8d4231","status":"running","createdAt":"2026-11-15T04:50:01.510Z","startedAt":"2026-11-15T05:09:16.731Z","finishedAt":null,"errorCode":null,"retryOfRunId":null,"wakeCommentIds":[],"livenessState":null,"lastEvidenceAt":"2026-11-15T05:20:42.672Z","toolInFlight":true,"providerTurnActive":true}]},{"at":1794720113084,"reachable":truLine truncated
|
||||
@@ -7,6 +7,7 @@ import { enforcedUniversalFailures, gradeUniversal, type SlotFree, type Universa
|
||||
|
||||
export const ACTIVE_RUN_STATUSES = new Set(["queued", "running", "scheduled_retry"]);
|
||||
export const STALL_LIMIT_MS = 3 * 60_000;
|
||||
export const PROVIDER_TURN_LIMIT_MS = 10 * 60_000;
|
||||
const REST_STATUSES_WITH_WORK = new Set(["todo", "in_progress"]);
|
||||
|
||||
const ms = (iso: string | null | undefined) => (iso ? Date.parse(iso) : NaN);
|
||||
@@ -210,8 +211,9 @@ export function deadRunningRun(sample: TaskSample, readySince: number): { id: st
|
||||
if (run.status !== "running") continue;
|
||||
// Quiet is not dead while a tool is still in flight or the run's process is verifiably alive.
|
||||
if (run.toolInFlight || run.processAlive) continue;
|
||||
const limit = run.providerTurnActive ? PROVIDER_TURN_LIMIT_MS : STALL_LIMIT_MS;
|
||||
const evidence = Math.max(ms(run.lastEvidenceAt) || 0, ms(run.startedAt) || 0, readySince);
|
||||
if (sample.at - evidence > STALL_LIMIT_MS) return { id: run.id, idleMs: sample.at - Math.max(ms(run.lastEvidenceAt) || 0, ms(run.startedAt) || 0) };
|
||||
if (sample.at - evidence > limit) return { id: run.id, idleMs: sample.at - Math.max(ms(run.lastEvidenceAt) || 0, ms(run.startedAt) || 0) };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
@@ -128,6 +128,20 @@ describe("release swap task grade", () => {
|
||||
expect(gradeReleaseSwapTask(task({ status: "todo", runs: [failed], queuedWakes: 1 })).verdict).toBe("still_working");
|
||||
});
|
||||
|
||||
it("does not report queued wakes as never dispatched while a run holds them (w-20261115T044649-s7106/t019)", () => {
|
||||
const now = 10_000_000;
|
||||
const held = gradeReleaseSwapTask(task({ status: "in_progress", now, staleAfterMs: 60_000, queuedWakes: 2,
|
||||
interruption: interruption({ state: "working" }),
|
||||
runs: [{ runId: "9df2283c", status: "running", errorCode: null, createdAt: 0, startedAt: 1, finishedAt: null, lastActivityAt: now - 70_000 }] }));
|
||||
expect(held.detail).toContain("run 9df2283c running with no activity for 1 min");
|
||||
expect(held.detail).not.toContain("never dispatched");
|
||||
const queued = gradeReleaseSwapTask(task({ status: "todo", now, staleAfterMs: 60_000, queuedWakes: 1,
|
||||
runs: [{ runId: "q1", status: "queued", errorCode: null, createdAt: now - 120_000, startedAt: null, finishedAt: null, lastActivityAt: null }] }));
|
||||
expect(queued.detail).not.toContain("never dispatched");
|
||||
const orphan = gradeReleaseSwapTask(task({ status: "todo", now, staleAfterMs: 60_000, runs: [failed], queuedWakes: 2 }));
|
||||
expect(orphan.detail).toContain("2 queued wake(s) never dispatched");
|
||||
});
|
||||
|
||||
it("keeps a task still running past the drain cap out of the number and reports it with its last activity", () => {
|
||||
const running = gradeReleaseSwapTask(task({ status: "in_progress", now: 1_000_000, staleAfterMs: 900_000,
|
||||
runs: [{ runId: "r5", status: "running", errorCode: null, createdAt: 0, startedAt: 1, finishedAt: null, lastActivityAt: 940_000 }] }));
|
||||
|
||||
@@ -143,9 +143,10 @@ export function gradeReleaseSwapTask(task: GradeTaskInput): TaskGrade {
|
||||
const lastActivityAgeMs = task.now !== undefined && liveActivity.length ? task.now - Math.max(...liveActivity) : null;
|
||||
const verdict = (value: TaskVerdict, detail: string): TaskGrade => ({ ...base, verdict: value, detail, label: label(value), lastActivityAgeMs });
|
||||
const stale = staleRuns(task);
|
||||
const heldByRun = task.runs.some(run => LIVE_RUN.has(run.status));
|
||||
const idle = task.now === undefined ? [] : [
|
||||
...stale.map(run => `run ${run.runId} ${run.status} with no activity for ${Math.round((task.now! - (run.lastActivityAt ?? run.createdAt ?? 0)) / 60_000)} min`),
|
||||
...(task.queuedWakes ? [`${task.queuedWakes} queued wake(s) never dispatched`] : []), ...(task.scheduledRetry ? ["a scheduled retry that never ran"] : []),
|
||||
...(task.queuedWakes && !heldByRun ? [`${task.queuedWakes} queued wake(s) never dispatched`] : []), ...(task.scheduledRetry ? ["a scheduled retry that never ran"] : []),
|
||||
...(task.interruption && (task.interruption.state === "working" || task.interruption.state === "retry_scheduled" || task.interruption.retryWait)
|
||||
? [`the board shows ${task.interruption.retryWait ? "a Retry waiting on the system" : task.interruption.state}`] : []),
|
||||
...(task.activeRecoveryOwner && task.activeRecoveryOwner !== "board" ? [`a ${task.activeRecoveryOwner} recovery action that made no progress`] : [])];
|
||||
|
||||
@@ -416,6 +416,30 @@ describe("R5 live evidence", () => {
|
||||
expect(gradeR5(quietB5({}))).toMatchObject({ verdict: "fail", detail: expect.stringMatching(/^silent_dead_run r1/) });
|
||||
});
|
||||
|
||||
it("does not flag a slow model request while the provider turn is active with no tool running (w-20261115T044649-s7106/t028)", () => {
|
||||
const slow = JSON.parse(readFileSync(path.join(import.meta.dirname, "fixtures/t028-slow-provider-turn.json"), "utf8")) as { samples: TaskSample[] };
|
||||
const gap = slow.samples.filter(sample => sample.runs?.[0]?.lastEvidenceAt === "2026-11-15T05:22:13.005Z");
|
||||
expect(gap.at(-1)!.at - gap[0]!.at).toBeGreaterThan(300_000);
|
||||
expect(gap.every(sample => sample.runs![0]!.providerTurnActive && !sample.runs![0]!.toolInFlight)).toBe(true);
|
||||
expect(gradeR5(record(slow.samples)).verdict).toBe("pass");
|
||||
});
|
||||
|
||||
const quietTurn = (live: { providerTurnActive?: boolean }, quietMs: number) => {
|
||||
const start = Date.parse("2026-11-15T05:00:00Z");
|
||||
const run = { id: "r1", status: "running", createdAt: null, startedAt: new Date(start - 600_000).toISOString(), finishedAt: null,
|
||||
errorCode: null, retryOfRunId: null, wakeCommentIds: [], livenessState: null, lastEvidenceAt: new Date(start).toISOString(), ...live };
|
||||
const offsets = Array.from({ length: Math.floor(quietMs / 30_000) + 1 }, (_, index) => index * 30_000);
|
||||
return record(offsets.map(offset => ({ at: start + offset, reachable: true, serverReady: true, issueStatus: "in_progress", runs: [run] })));
|
||||
};
|
||||
|
||||
it("still fails a silent run with no active provider turn, and a provider turn silent past the provider-latency bound", () => {
|
||||
expect(gradeR5(quietTurn({}, 240_000))).toMatchObject({ verdict: "fail", detail: expect.stringMatching(/^silent_dead_run r1/) });
|
||||
expect(gradeR5(quietTurn({ providerTurnActive: false }, 240_000))).toMatchObject({ verdict: "fail", detail: expect.stringMatching(/^silent_dead_run r1/) });
|
||||
expect(gradeR5(quietTurn({ providerTurnActive: true }, 570_000)).verdict).toBe("pass");
|
||||
expect(gradeR5(quietTurn({ providerTurnActive: true }, 630_000))).toMatchObject({ verdict: "fail",
|
||||
detail: expect.stringMatching(/^silent_dead_run r1 .* for 630s/) });
|
||||
});
|
||||
|
||||
it("derives tool-in-flight and process liveness from run events", async () => {
|
||||
const startedAt = new Date().toISOString();
|
||||
const identity = { processPid: 4242, processGroupId: 4242, processStartedAt: "2026-11-06T10:00:00.000Z", targetKind: "local" };
|
||||
|
||||
Reference in new issue
Block a user