test(ci): stop the live-run census hanging server shards, and name any stalled test file (#21941)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - PR CI splits the server tests across 12 shards; `ci / verify` fails
when any shard fails
> - Since 2026-11-16 the server shards were cancelled at their 20-minute
job limit with no output (shard 2/12 five times, plus shards 1, 3 and
9), which blocked unrelated pull requests until reruns
> - Turbo prints a task's output only when it ends, so a cancelled shard
never names the stuck file
> - Recomputing each cancelled shard's file list at its own merge commit
put `issue-recovery-live-run-agent-census.test.ts` in 12 of 14 of them;
it is also the slowest file in passing shards (235-300 s)
> - That census started 9-14 real agent runs per file while `setTimeout`
was faked, waited on them with no bound, and had 30-minute test
timeouts, so a stuck wait could never fail inside the job
> - This pull request stops the census from starting real runs, bounds
every wait, and adds a CI reporter that fails a run naming any file that
stops making progress
> - The benefit is that this file cannot hang a shard, and the next
stuck file fails in minutes with its name and step in the log

## Linked Issues or Issue Description

Refs #21867 (the same fix for the sibling agent-row census), #21831 (the
behavior this census guards)

**What happened?**
`ci / General tests (server (2/12))` was cancelled at 20m16s in actions
runs 43865258195, 43926799428, 43932238377, 43934278368, 43944290677 and
43935722497 (both attempts), and other server shards hit the same limit
(for example 43901003914 shard 1, 43945768035 shard 3, 43845343831 shard
9). The hung jobs show CPU near 0 for the last 9 minutes and printed
nothing after `Running //#test:run:general`.

Shard membership changes with every merge commit. To compare the shards,
I recomputed each job's file list at its exact merge commit with
`GITHUB_WORKFLOW=PR` (the PR workflow removes the native-runner suites
from these shards, so a local dry run without it gives a different
list).
`server/src/__tests__/issue-recovery-live-run-agent-census.test.ts` was
in 12 of the 14 hung shards.

Why that file:
- During its 10-minute fake-timer warm-up, the "live" task was
`in_progress` with no run. That is real assigned work, so recovery woke
its agent and a real `process`-adapter run started while `setTimeout`
was faked. The run lost its lease ("Legacy controller lease lost"),
recovery retried, and a local run of the file showed 13 real run
executions.
- No wait in the file had a bound, and each test had a 30-minute
timeout, so a delivery pass waiting on a faked timer or a run holding a
lock never failed inside the 20-minute job.
- Under local CPU contention, 3 of 4 copies of the old file failed the
census assertion (39 settled checks, caused by the stray runs), and each
copy took about 270 s with 9-14 real runs.

Fixing the runs exposed a second problem in the fixture: the seeded
tasks were stamped "now". The first 5-minute backstop after the live
run's first hint re-checks rows changed within 5 minutes before the last
full pass, so it re-checked all 860 seeded tasks. The old file passed
only because the stray runs happened to trigger that backstop during the
warm-up.

**Expected behavior**
The census never starts a real run, every wait has a bound that names
what it waited for, and a CI test file that stops making progress fails
the job quickly with its name.

## What Changed

- `issue-recovery-live-run-agent-census.test.ts`
- The live task is a settled review until its run starts; `startLiveRun`
now writes the checkout (`in_progress` + `executionRunId`) the way
execution start does.
- Population agents and the held agent take no on-demand wakes
(`wakeOnDemand: false`), so a recovery check cannot dispatch a run. The
census counts queue checks, so this does not change what it measures.
- The seeded population is backdated 2 hours ("settled long before the
window", as the file's comment already said).
- Each step (seed, worker start, run start, run-event tick, fake-timer
advance, pass settlement, direct writes) has a real 20-second bound
captured before the timers are faked, with a message naming the step.
Test timeout is 4 minutes instead of 30.
- New `scripts/ci/stalled-file-reporter.mjs`, enabled in GitHub Actions
from the root `vitest.config.ts`: when a running file starts no test or
hook and reports no result for 8 minutes
(`PAPERCLIP_VITEST_FILE_STALL_MS` overrides), it prints a `::error
file=...` annotation naming the file and the step it was in, cancels the
run and exits 1. The slowest healthy server file runs about 5 minutes in
total, so 8 minutes without any progress means a stuck file. Tests:
`scripts/ci/stalled-file-reporter.test.mjs` (added to
`test:release-registry`).

## Verification

- Census, red: the old file runs 13 real agent runs locally (`heartbeat
execution setup failed`, `LegacyControllerLeaseLostError`); with 4
copies under CPU contention, 3 failed with 39 settled checks at about
270 s each.
- Census, green: 4/4 tests pass, 0 real runs, about 80 s locally.
- Census still guards #21831: with the `ag.status <> 'running'` line
removed from `changed-tasks.ts`, the first test fails with 534 settled
checks and 540 backstop checks, the numbers #21831 recorded.
- Reporter, red/green: the real-Vitest case (a file awaiting a promise
that never settles, with a 10-minute test timeout) is still running when
the 20-second test bound ends without the reporter; with the reporter it
exits 1 in about 8 s and prints the annotation naming the file and the
test.
- Reporter on a real server file: `PAPERCLIP_VITEST_FILE_STALL_MS=3000`
on `issue-recovery-recorded-decision-idle.test.ts` prints the annotation
naming the file and step and exits 1.
- `node --test scripts/ci/stalled-file-reporter.test.mjs`: 5/5 pass.

## Risks

- Test and CI-tooling change only. No product code changed.
- The census's agents no longer take on-demand wakes. The census
measures recovery queue checks, not dispatch, and still fails without
the #21831 fix.
- The reporter stops the whole CI run when one file stalls for 8
minutes. A legitimate single test or hook that runs more than 8 minutes
with no other progress in its file would now fail; none exists today
(the slowest whole file is about 5 minutes). The limit is configurable.
- Other fake-timer recovery files still have long test timeouts and
unbounded pass waits (for example `issue-recovery-activity-hints`, 30
minutes). The reporter now caps any of them at 8 minutes without
progress and names the file; bounding them is follow-up work.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-16 19:50:54 -06:00
1 parent d8a5db36e6
commit d258f478fc
6 files changed
+325 -28

No files matched your search

+146
View File
@@ -0,0 +1,146 @@
import path from "node:path";
import { fileURLToPath } from "node:url";
// A test file that stops making progress otherwise holds its CI job silently
// until the job's time limit cancels it. Turbo prints a task's output only
// when the task ends, so a cancelled job never names the stuck file (server
// shard 2/12 on 2026-11-16: five cancellations at 20m16s, no file named).
//
// This reporter watches every running file. When one starts no test, hook or
// result for the stall limit, it prints a GitHub error annotation naming the
// file and the step it was in, then ends the run with a failure, so the job
// fails minutes earlier with the file and step in its log.
export const stallEnvName = "PAPERCLIP_VITEST_FILE_STALL_MS";
// The slowest healthy server files ran about 5 minutes in total
// (issue-recovery-live-run-agent-census, 235-300 s in PR shards), and no
// single test or hook in a healthy run goes this long without progress.
export const defaultStallMs = 8 * 60_000;
// How long a cancelled run may take to stop before the process exits anyway.
export const exitGraceMs = 30_000;
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
export function stallLimitMs(env = process.env) {
const configured = env[stallEnvName]?.trim();
if (!configured) return defaultStallMs;
const parsed = Number(configured);
if (!Number.isInteger(parsed) || parsed <= 0) {
throw new Error(`${stallEnvName} must be a positive integer of milliseconds. Received "${configured}".`);
}
return parsed;
}
// Watching is for CI, where a stuck file costs a whole job; a local run can be
// stopped by hand. The variable turns it on anywhere, for example to reproduce.
export function stalledFileReporterEnabled(env = process.env) {
return env.GITHUB_ACTIONS === "true" || Boolean(env[stallEnvName]?.trim());
}
function escapeData(value) {
return String(value).replaceAll("%", "%25").replaceAll("\r", "%0D").replaceAll("\n", "%0A");
}
function escapeProperty(value) {
return escapeData(value).replaceAll(":", "%3A").replaceAll(",", "%2C");
}
export function stallAnnotation(entry, now) {
const minutes = ((now - entry.progressAt) / 60_000).toFixed(1);
const properties = [`file=${escapeProperty(entry.file)}`, `title=${escapeProperty("Test file stopped making progress")}`];
return `::error ${properties.join(",")}::${escapeData(
`${entry.file} made no progress for ${minutes} min while in: ${entry.step}. `
+ "The run was stopped so this job fails with the file's name instead of reaching its time limit.")}`;
}
export default class StalledFileReporter {
constructor(options = {}) {
this.limitMs = options.limitMs ?? stallLimitMs();
this.now = options.now ?? Date.now;
this.exit = options.exit ?? (code => process.exit(code));
this.write = options.write ?? (line => process.stderr.write(`${line}\n`));
this.root = options.root ?? repoRoot;
this.graceMs = options.graceMs ?? exitGraceMs;
this.intervalMs = options.intervalMs ?? Math.min(15_000, Math.max(250, Math.floor(this.limitMs / 4)));
this.files = new Map();
this.timer = null;
this.stopped = false;
}
onInit(vitest) {
this.vitest = vitest;
}
onTestRunStart() {
this.stopped = false;
if (this.timer) return;
// The reporter runs in the main Vitest process, whose timers tests never
// fake. The interval must not keep a finished run alive.
this.timer = setInterval(() => this.check(), this.intervalMs);
this.timer.unref?.();
}
progress(moduleId, step) {
const entry = this.files.get(moduleId);
if (!entry) return;
entry.progressAt = this.now();
entry.step = step;
}
// Queued fires before the file is imported, so a file that never finishes
// loading (a top-level await or a blocking setup import) is watched too.
onTestModuleQueued(testModule) {
const file = path.relative(this.root, testModule.moduleId).split(path.sep).join("/");
this.files.set(testModule.moduleId, { file, progressAt: this.now(), step: "loading the file" });
}
onTestModuleStart(testModule) {
if (!this.files.has(testModule.moduleId)) this.onTestModuleQueued(testModule);
this.progress(testModule.moduleId, "starting the file's tests");
}
onHookStart(hook) {
const moduleId = hook.entity?.module?.moduleId ?? hook.entity?.moduleId;
const owner = hook.entity?.fullName ?? hook.entity?.name;
this.progress(moduleId, owner ? `${hook.name} of ${owner}` : hook.name);
}
onHookEnd(hook) {
this.onHookStart({ ...hook, name: `after ${hook.name}` });
}
onTestCaseReady(testCase) {
this.progress(testCase.module.moduleId, `test "${testCase.fullName}"`);
}
onTestCaseResult(testCase) {
this.progress(testCase.module.moduleId, `after test "${testCase.fullName}"`);
}
onTestModuleEnd(testModule) {
this.files.delete(testModule.moduleId);
}
onTestRunEnd() {
this.files.clear();
if (this.timer) clearInterval(this.timer);
this.timer = null;
// A cancelled run can report its finished files as passed. It still failed:
// exit once the other reporters have printed their summaries.
if (this.stopped) setTimeout(() => this.exit(1), 1000);
}
check() {
if (this.stopped) return;
const now = this.now();
const stalled = [...this.files.values()].filter(entry => now - entry.progressAt >= this.limitMs);
if (stalled.length === 0) return;
this.stopped = true;
for (const entry of stalled) this.write(stallAnnotation(entry, now));
const running = [...this.files.values()].map(entry => entry.file).sort();
this.write(`[stalled-file-reporter] stopping the run; files still running: ${running.join(", ")}`);
process.exitCode = 1;
// A stalled worker may never answer a cancel, so exit after a grace period
// either way. This timer is kept referenced on purpose.
setTimeout(() => this.exit(1), this.graceMs);
void Promise.resolve(this.vitest?.cancelCurrentRun("stalled-file")).catch(() => undefined);
}
}
+114
View File
@@ -0,0 +1,114 @@
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import test from "node:test";
import StalledFileReporter, { defaultStallMs, stallAnnotation, stalledFileReporterEnabled, stallLimitMs }
from "./stalled-file-reporter.mjs";
const root = fileURLToPath(new URL("../../", import.meta.url));
const reporterPath = fileURLToPath(new URL("./stalled-file-reporter.mjs", import.meta.url));
test("the stall limit defaults to eight minutes and accepts a positive override", () => {
assert.equal(stallLimitMs({}), defaultStallMs);
assert.equal(defaultStallMs, 8 * 60_000);
assert.equal(stallLimitMs({ PAPERCLIP_VITEST_FILE_STALL_MS: " 1500 " }), 1500);
assert.equal(stallLimitMs({ PAPERCLIP_VITEST_FILE_STALL_MS: "" }), defaultStallMs);
for (const value of ["0", "-5", "1.5", "soon"]) {
assert.throws(() => stallLimitMs({ PAPERCLIP_VITEST_FILE_STALL_MS: value }), /positive integer/, value);
}
});
test("the reporter watches in GitHub Actions, or wherever the limit is set", () => {
assert.equal(stalledFileReporterEnabled({}), false);
assert.equal(stalledFileReporterEnabled({ GITHUB_ACTIONS: "true" }), true);
assert.equal(stalledFileReporterEnabled({ PAPERCLIP_VITEST_FILE_STALL_MS: "1000" }), true);
});
test("a file stops the run only once it makes no progress for the whole limit", () => {
let now = 0;
const lines = [], exits = [];
const cancels = [];
const reporter = new StalledFileReporter({ limitMs: 1000, now: () => now, write: line => lines.push(line),
exit: code => exits.push(code), root: "/repo", graceMs: 0 });
reporter.onInit({ cancelCurrentRun: reason => { cancels.push(reason); return Promise.resolve(); } });
const moduleId = "/repo/server/src/stuck.test.ts";
const testCase = { fullName: "suite > waits forever", module: { moduleId } };
reporter.onTestModuleStart({ moduleId });
reporter.onTestModuleStart({ moduleId: "/repo/server/src/healthy.test.ts" });
now = 900; reporter.onTestCaseReady(testCase);
now = 1800; reporter.onTestCaseResult({ fullName: "ok", module: { moduleId: "/repo/server/src/healthy.test.ts" } });
reporter.check();
assert.deepEqual(lines, [], "progress 900 ms ago is not a stall");
now = 1900; reporter.check();
assert.equal(lines.length, 2);
assert.match(lines[0], /^::error file=server\/src\/stuck\.test\.ts,title=Test file stopped making progress::/);
assert.match(lines[0], /while in: test "suite > waits forever"/);
assert.equal(lines[1], "[stalled-file-reporter] stopping the run; files still running: server/src/healthy.test.ts, server/src/stuck.test.ts");
assert.deepEqual(cancels, ["stalled-file"]);
reporter.check();
assert.equal(lines.length, 2, "a stopped run reports once");
process.exitCode = 0;
});
test("finished files and hooks count as progress", () => {
let now = 0;
const lines = [];
const reporter = new StalledFileReporter({ limitMs: 1000, now: () => now, write: line => lines.push(line), exit: () => {}, root: "/repo" });
const moduleId = "/repo/a.test.ts";
reporter.onTestModuleStart({ moduleId });
now = 800; reporter.onHookStart({ name: "beforeAll", entity: { moduleId } });
now = 1600; reporter.check();
assert.deepEqual(lines, []);
reporter.onTestModuleEnd({ moduleId });
now = 10_000; reporter.check();
assert.deepEqual(lines, []);
assert.equal(stallAnnotation({ file: "a,b:c.test.ts", progressAt: 0, step: "x\ny" }, 90_000),
"::error file=a%2Cb%3Ac.test.ts,title=Test file stopped making progress::a,b:c.test.ts made no progress for 1.5 min while in: x%0Ay."
+ " The run was stopped so this job fails with the file's name instead of reaching its time limit.");
});
test("a file that never finishes loading is named from the moment it is queued", () => {
let now = 0;
const lines = [];
const reporter = new StalledFileReporter({ limitMs: 1000, now: () => now, write: line => lines.push(line), exit: () => {}, root: "/repo" });
reporter.onTestModuleQueued({ moduleId: "/repo/server/src/top-level-await.test.ts" });
now = 1000; reporter.check();
assert.match(lines[0] ?? "", /^::error file=server\/src\/top-level-await\.test\.ts,.*while in: loading the file\./);
process.exitCode = 0;
});
// The failure this guards: a test awaiting a promise that never settles, with
// a test timeout longer than the CI job, holds Vitest until the job is
// cancelled and never prints the file's name.
for (const [label, body, step] of [
["a test that never settles", 'import { test } from "vitest";\ntest("waits forever", () => new Promise(() => {}), 600_000);\n', 'test "waits forever"'],
["a top-level await that never settles", 'import { test } from "vitest";\nawait new Promise(() => {});\ntest("never collected", () => {});\n', "loading the file"],
]) test(`a real Vitest run with ${label} fails fast and names the file`, () => {
const directory = realpathSync(mkdtempSync(path.join(tmpdir(), "stalled-file-reporter-")));
try {
writeFileSync(path.join(directory, "stuck.test.mjs"), body);
writeFileSync(path.join(directory, "quick.test.mjs"), `import { test } from "vitest";
test("passes", () => {});
`);
const configFile = path.join(directory, "vitest.config.mjs");
writeFileSync(configFile, `export default ${JSON.stringify({ test: { include: ["*.test.mjs"],
reporters: ["default", [reporterPath, { limitMs: 1500, graceMs: 5000, root: directory }]] } })};`);
const started = Date.now();
const result = spawnSync(process.execPath, [path.join(root, "node_modules/vitest/vitest.mjs"), "run",
"--config", configFile, "--root", directory], {
cwd: directory, encoding: "utf8", timeout: 60_000,
env: { ...process.env, GITHUB_ACTIONS: "", PAPERCLIP_VITEST_FILE_STALL_MS: "" },
});
const output = `${result.stdout}\n${result.stderr}`;
assert.equal(result.signal, null, `Vitest was still running after 60 s:\n${output}`);
assert.notEqual(result.status, 0, output);
assert.ok(Date.now() - started < 30_000, `took ${Date.now() - started} ms:\n${output}`);
assert.match(output, /::error file=stuck\.test\.mjs,title=Test file stopped making progress::stuck\.test\.mjs made no progress/);
assert.ok(output.includes(`while in: ${step}.`), output);
} finally {
rmSync(directory, { recursive: true, force: true });
}
});