test(ci): stop the live-run census hanging server shards, and name any stalled test file (#21941)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - PR CI splits the server tests across 12 shards; `ci / verify` fails
when any shard fails
> - Since 2026-11-16 the server shards were cancelled at their 20-minute
job limit with no output (shard 2/12 five times, plus shards 1, 3 and
9), which blocked unrelated pull requests until reruns
> - Turbo prints a task's output only when it ends, so a cancelled shard
never names the stuck file
> - Recomputing each cancelled shard's file list at its own merge commit
put `issue-recovery-live-run-agent-census.test.ts` in 12 of 14 of them;
it is also the slowest file in passing shards (235-300 s)
> - That census started 9-14 real agent runs per file while `setTimeout`
was faked, waited on them with no bound, and had 30-minute test
timeouts, so a stuck wait could never fail inside the job
> - This pull request stops the census from starting real runs, bounds
every wait, and adds a CI reporter that fails a run naming any file that
stops making progress
> - The benefit is that this file cannot hang a shard, and the next
stuck file fails in minutes with its name and step in the log

## Linked Issues or Issue Description

Refs #21867 (the same fix for the sibling agent-row census), #21831 (the
behavior this census guards)

**What happened?**
`ci / General tests (server (2/12))` was cancelled at 20m16s in actions
runs 43865258195, 43926799428, 43932238377, 43934278368, 43944290677 and
43935722497 (both attempts), and other server shards hit the same limit
(for example 43901003914 shard 1, 43945768035 shard 3, 43845343831 shard
9). The hung jobs show CPU near 0 for the last 9 minutes and printed
nothing after `Running //#test:run:general`.

Shard membership changes with every merge commit. To compare the shards,
I recomputed each job's file list at its exact merge commit with
`GITHUB_WORKFLOW=PR` (the PR workflow removes the native-runner suites
from these shards, so a local dry run without it gives a different
list).
`server/src/__tests__/issue-recovery-live-run-agent-census.test.ts` was
in 12 of the 14 hung shards.

Why that file:
- During its 10-minute fake-timer warm-up, the "live" task was
`in_progress` with no run. That is real assigned work, so recovery woke
its agent and a real `process`-adapter run started while `setTimeout`
was faked. The run lost its lease ("Legacy controller lease lost"),
recovery retried, and a local run of the file showed 13 real run
executions.
- No wait in the file had a bound, and each test had a 30-minute
timeout, so a delivery pass waiting on a faked timer or a run holding a
lock never failed inside the 20-minute job.
- Under local CPU contention, 3 of 4 copies of the old file failed the
census assertion (39 settled checks, caused by the stray runs), and each
copy took about 270 s with 9-14 real runs.

Fixing the runs exposed a second problem in the fixture: the seeded
tasks were stamped "now". The first 5-minute backstop after the live
run's first hint re-checks rows changed within 5 minutes before the last
full pass, so it re-checked all 860 seeded tasks. The old file passed
only because the stray runs happened to trigger that backstop during the
warm-up.

**Expected behavior**
The census never starts a real run, every wait has a bound that names
what it waited for, and a CI test file that stops making progress fails
the job quickly with its name.

## What Changed

- `issue-recovery-live-run-agent-census.test.ts`
- The live task is a settled review until its run starts; `startLiveRun`
now writes the checkout (`in_progress` + `executionRunId`) the way
execution start does.
- Population agents and the held agent take no on-demand wakes
(`wakeOnDemand: false`), so a recovery check cannot dispatch a run. The
census counts queue checks, so this does not change what it measures.
- The seeded population is backdated 2 hours ("settled long before the
window", as the file's comment already said).
- Each step (seed, worker start, run start, run-event tick, fake-timer
advance, pass settlement, direct writes) has a real 20-second bound
captured before the timers are faked, with a message naming the step.
Test timeout is 4 minutes instead of 30.
- New `scripts/ci/stalled-file-reporter.mjs`, enabled in GitHub Actions
from the root `vitest.config.ts`: when a running file starts no test or
hook and reports no result for 8 minutes
(`PAPERCLIP_VITEST_FILE_STALL_MS` overrides), it prints a `::error
file=...` annotation naming the file and the step it was in, cancels the
run and exits 1. The slowest healthy server file runs about 5 minutes in
total, so 8 minutes without any progress means a stuck file. Tests:
`scripts/ci/stalled-file-reporter.test.mjs` (added to
`test:release-registry`).

## Verification

- Census, red: the old file runs 13 real agent runs locally (`heartbeat
execution setup failed`, `LegacyControllerLeaseLostError`); with 4
copies under CPU contention, 3 failed with 39 settled checks at about
270 s each.
- Census, green: 4/4 tests pass, 0 real runs, about 80 s locally.
- Census still guards #21831: with the `ag.status <> 'running'` line
removed from `changed-tasks.ts`, the first test fails with 534 settled
checks and 540 backstop checks, the numbers #21831 recorded.
- Reporter, red/green: the real-Vitest case (a file awaiting a promise
that never settles, with a 10-minute test timeout) is still running when
the 20-second test bound ends without the reporter; with the reporter it
exits 1 in about 8 s and prints the annotation naming the file and the
test.
- Reporter on a real server file: `PAPERCLIP_VITEST_FILE_STALL_MS=3000`
on `issue-recovery-recorded-decision-idle.test.ts` prints the annotation
naming the file and step and exits 1.
- `node --test scripts/ci/stalled-file-reporter.test.mjs`: 5/5 pass.

## Risks

- Test and CI-tooling change only. No product code changed.
- The census's agents no longer take on-demand wakes. The census
measures recovery queue checks, not dispatch, and still fails without
the #21831 fix.
- The reporter stops the whole CI run when one file stalls for 8
minutes. A legitimate single test or hook that runs more than 8 minutes
with no other progress in its file would now fail; none exists today
(the slowest whole file is about 5 minutes). The limit is configurable.
- Other fake-timer recovery files still have long test timeouts and
unbounded pass waits (for example `issue-recovery-activity-hints`, 30
minutes). The reporter now caps any of them at 8 minutes without
progress and names the file; bounding them is follow-up work.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-16 19:50:54 -06:00
1 parent d8a5db36e6
commit d258f478fc
6 files changed
+325 -28

No files matched your search

@@ -11,9 +11,23 @@ import { seedRecoveryPopulation } from "./helpers/recovery-population.js";
vi.hoisted(() => { process.env.PAPERCLIP_DEBUG_WORK_SIGNALS = "stranded-assignments:1,assigned-work-invariant:1"; });
const MINUTE = 60_000;
const TEST_TIMEOUT_MS = 240_000;
const STEP_TIMEOUT_MS = 20_000;
const TOPICS = new Set(["stranded-assignments", "assigned-work-invariant"]);
const realSetTimeout = globalThis.setTimeout;
const realClearTimeout = globalThis.clearTimeout;
type Recorded = { at: number; topic: string; name: string; fields: Record<string, unknown> };
/** Bounds one step with the real clock, captured before the timers are faked,
* so a wait that cannot finish fails in seconds and names what it waited for. */
function bounded<T>(work: Promise<T>, waitingFor: () => string): Promise<T> {
let timer: ReturnType<typeof setTimeout> | undefined;
const expired = new Promise<never>((_, reject) => {
timer = realSetTimeout(() => reject(new Error(`Timed out after ${STEP_TIMEOUT_MS} ms waiting for ${waitingFor()}`)), STEP_TIMEOUT_MS);
});
return Promise.race([work, expired]).finally(() => realClearTimeout(timer));
}
/**
* Staging shape: about 860 settled assigned tasks, most of them held by the
* same few agents, and one agent whose native run stays live for the whole
@@ -27,9 +41,14 @@ describeIssueRecovery("stranded-task checks beside one long live run (census)",
const until = Date.now() + ms;
let nextTick = Date.now();
while (Date.now() < until) {
if (onTick && Date.now() >= nextTick) { await onTick(); nextTick = Date.now() + 10_000; }
await vi.advanceTimersByTimeAsync(Math.min(1000, until - Date.now()));
await Promise.allSettled([...state.passes]); await immediate();
const at = new Date(Date.now()).toISOString();
if (onTick && Date.now() >= nextTick) {
await bounded(onTick(), () => `the run event at simulated ${at} to commit`);
nextTick = Date.now() + 10_000;
}
await bounded(vi.advanceTimersByTimeAsync(Math.min(1000, until - Date.now())), () => `simulated time to advance past ${at}`);
await bounded(Promise.allSettled([...state.passes]), () => `${state.passes.size} delivery passes started by simulated ${at} to settle`);
await immediate();
}
}
@@ -56,38 +75,52 @@ describeIssueRecovery("stranded-task checks beside one long live run (census)",
async function seed() {
const f = await state.fixture();
// 12 agents leave 3 workers, so each holds about 266 of the settled reviews.
const population = await seedRecoveryPopulation(state.db, f.companyId, { agents: 12, inReview: 800, inProgress: 30, todo: 30 });
// No agent takes on-demand wakes: a check that repairs a task would start a
// real run while setTimeout is faked, and that run can wait forever. The
// census counts queue checks, not runs.
const population = await seedRecoveryPopulation(state.db, f.companyId,
{ agents: 12, inReview: 800, inProgress: 30, todo: 30, wakeOnDemand: false });
// Settled long before the window, as on staging. Rows stamped now count as
// changed for the first backstop after the warm-up, which would re-check
// every seeded task once.
const settledAt = new Date(Date.now() - 2 * 60 * MINUTE);
await state.db.update(issues).set({ createdAt: settledAt, updatedAt: settledAt }).where(eq(issues.companyId, f.companyId));
await state.db.update(heartbeatRuns).set({ updatedAt: settledAt }).where(eq(heartbeatRuns.companyId, f.companyId));
await state.db.update(agents).set({ createdAt: settledAt, updatedAt: settledAt }).where(eq(agents.companyId, f.companyId));
const liveAgentId = population.agentIds[7]!;
const [liveTask] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Live native task", status: "in_progress",
// A settled review until its run starts. An in-progress task with no run
// is real assigned work, so recovery would start a real run for it while
// setTimeout is faked, and that run can wait forever.
const [liveTask] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Live native task", status: "in_review",
assigneeAgentId: liveAgentId }).returning({ id: issues.id });
// A paused agent with one waiting task, both settled long before the window.
const old = new Date(Date.now() - 2 * 60 * MINUTE);
// It takes no on-demand wakes, so resuming it never starts a real run.
const [held] = await state.db.insert(agents).values({ companyId: f.companyId, name: "Held worker", role: "engineer", status: "paused",
lifecycleHolds: ["manual"], adapterType: "process", reportsTo: population.agentIds[1], createdAt: old, updatedAt: old,
lifecycleHolds: ["manual"], adapterType: "process", reportsTo: population.agentIds[1], createdAt: settledAt, updatedAt: settledAt,
adapterConfig: { command: process.execPath, args: ["-e", "0"], cwd: process.cwd() },
runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: true } } }).returning({ id: agents.id });
runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: false } } }).returning({ id: agents.id });
const [waiting] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Waits for its agent", status: "todo",
assigneeAgentId: held!.id, createdAt: old, updatedAt: old }).returning({ id: issues.id });
assigneeAgentId: held!.id, createdAt: settledAt, updatedAt: settledAt }).returning({ id: issues.id });
return { ...f, population, liveAgentId, liveTaskId: liveTask!.id, heldAgentId: held!.id, waitingTaskId: waiting!.id };
}
async function boot() {
const f = await seed();
const f = await bounded(seed(), () => "the fixture to be seeded");
const events = record();
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
await state.start({});
await bounded(state.start({}), () => "the recovery workers to start");
await advance(10 * MINUTE);
events.length = 0;
return { ...f, events };
}
/** What execution start writes (heartbeat.ts): the run, then the agent's
* status flip to running without a heartbeat time. */
/** What execution start writes (heartbeat.ts): the run, the task's checkout,
* then the agent's status flip to running without a heartbeat time. */
async function startLiveRun(f: Awaited<ReturnType<typeof boot>>) {
const [run] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.liveAgentId, status: "running",
runtimeMode: "native", startedAt: new Date(), contextSnapshot: { issueId: f.liveTaskId }, nativeIssueId: f.liveTaskId })
.returning({ id: heartbeatRuns.id });
await state.db.update(issues).set({ executionRunId: run!.id }).where(eq(issues.id, f.liveTaskId));
await state.db.update(issues).set({ status: "in_progress", executionRunId: run!.id }).where(eq(issues.id, f.liveTaskId));
await state.db.update(agents).set({ status: "running", updatedAt: new Date() }).where(eq(agents.id, f.liveAgentId));
const runId = run!.id;
return {
@@ -102,7 +135,7 @@ describeIssueRecovery("stranded-task checks beside one long live run (census)",
it("does not re-check the live agent's settled tasks while its run stays live", async () => {
const f = await boot();
const live = await startLiveRun(f);
const live = await bounded(startLiveRun(f), () => "the live run to start");
let ticks = 0;
const before = state.meter.read();
await advance(30 * MINUTE, () => { ticks += 1; return live.tick(); });
@@ -121,11 +154,11 @@ describeIssueRecovery("stranded-task checks beside one long live run (census)",
expect(result.strandedByWake.hint ?? 0).toBeLessThanOrEqual(ticks + 10);
expect(result.assignedWork).toBeLessThanOrEqual(10);
expect(result.signalsBeyondRunEvents).toBe(0);
}, 1_800_000);
}, TEST_TIMEOUT_MS);
it("still checks a task of the live agent within the backstop bound when its own rows change without a hint", async () => {
const f = await boot();
const live = await startLiveRun(f);
const live = await bounded(startLiveRun(f), () => "the live run to start");
await advance(6 * MINUTE, live.tick);
f.events.length = 0;
const target = f.population.inReview.find((_, index) => index % 3 === 0)!;
@@ -133,31 +166,32 @@ describeIssueRecovery("stranded-task checks beside one long live run (census)",
expect(current!.assignee).toBe(f.liveAgentId);
const changedAt = Date.now();
// A direct write that sends no hint (for example an edit by an older server).
await state.db.update(issues).set({ status: "todo", updatedAt: new Date() }).where(eq(issues.id, target));
await bounded(state.db.update(issues).set({ status: "todo", updatedAt: new Date() }).where(eq(issues.id, target)),
() => "the direct task write");
await advance(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE, live.tick);
const checked = f.events.find(event => event.topic === "stranded-assignments" && event.name === "check" && event.fields.issueId === target);
expect(checked).toBeDefined();
expect(checked!.at - changedAt).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE);
}, 1_800_000);
}, TEST_TIMEOUT_MS);
it("still checks a resumed agent's task within the backstop bound when the resume sends no hint", async () => {
const f = await boot();
const live = await startLiveRun(f);
const live = await bounded(startLiveRun(f), () => "the live run to start");
await advance(6 * MINUTE, live.tick);
f.events.length = 0;
const waiting = f.waitingTaskId;
const changedAt = Date.now();
await state.db.update(agents).set({ status: "idle", lifecycleHolds: [], updatedAt: new Date() })
.where(eq(agents.id, f.heldAgentId));
await bounded(state.db.update(agents).set({ status: "idle", lifecycleHolds: [], updatedAt: new Date() })
.where(eq(agents.id, f.heldAgentId)), () => "the agent resume write");
await advance(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE, live.tick);
const checked = f.events.find(event => event.topic === "stranded-assignments" && event.name === "check" && event.fields.issueId === waiting);
expect(checked).toBeDefined();
expect(checked!.at - changedAt).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE);
}, 1_800_000);
}, TEST_TIMEOUT_MS);
it("still re-checks every settled task in the hourly full pass", async () => {
const f = await boot();
const live = await startLiveRun(f);
const live = await bounded(startLiveRun(f), () => "the live run to start");
await advance(ISSUE_RECOVERY_FULL_BACKSTOP_MS + 5 * MINUTE, live.tick);
const checked = new Set(checksOf(f.events, "stranded-assignments").map(event => String(event.fields.issueId)));
const missing = f.population.inReview.filter(id => !checked.has(id));
@@ -165,5 +199,5 @@ describeIssueRecovery("stranded-task checks beside one long live run (census)",
const runs = await state.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, f.companyId), sql`${heartbeatRuns.status} = 'running'`));
expect(runs.map(run => run.id)).toContain(live.runId);
}, 1_800_000);
}, TEST_TIMEOUT_MS);
});