Files
PaperClipAI/server/src
DottaandClaude Opus 5.5 278660149c test(recovery): bound every wait in the agent-row census and stop it starting real runs (#21867)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The recovery census tests prove that recovery queues do not re-check
settled tasks without a reason
> - The agent-row census from #21861 stopped producing output in a
server test shard and ran until the 20-minute job limit
> - Its tests let real agent runs start while `setTimeout` was faked,
and no wait in the file had a bound (the test timeout was 30 minutes)
> - This pull request stops the census from starting real runs, bounds
every wait with a real timer, and makes the file much faster
> - The benefit is that the file cannot hang a shard, and a stuck step
fails in seconds with a message that names the step

## Linked Issues or Issue Description

Refs #21861

**What happened?**
In a 10-shard server test run,
`server/src/__tests__/issue-recovery-agent-row-write-census.test.ts`
finished its third test and then wrote no more output until the job
reached its 20-minute limit. The file had a 30-minute timeout per test,
so the hang never failed as a test. Several tests let real agent runs
start while `setTimeout` was faked:
- the idle-agent guard marked a live run succeeded while its task stayed
in progress, so recovery continued the task;
- the hold-release guard released an agent that had a `todo` task;
- the pause and terminate guards stopped agents with live native runs.

A run execution that waits on a faked timer, or holds a lock that a
delivery pass then waits on, does not end. A local run of the old file
showed 4 run executions ("heartbeat execution setup failed").

**Expected behavior**
The census never starts a real run. Every wait has a bound and a message
that names what it waited for. The file finishes well under 60 seconds.

**Steps to reproduce**
1. Run the old file and search its output for "heartbeat execution setup
failed": 4 run executions.
2. Run it in a slow, shared CI shard: the file can stop with no output
until the job limit.

**Paperclip version or commit**
`master` at 94cb7ad359.

## What Changed

- Fixture agents take no on-demand wakes (`wakeOnDemand: false`), so a
recovery check cannot dispatch a run. The census counts queue checks, so
this does not change what the tests measure.
- The idle-agent guard writes the configuration of an agent that never
ran, instead of ending the second agent's live run.
- The held agent's waiting task is a settled review, not a `todo` task.
- The pause and terminate guards use two idle agents without live runs.
- Each step (run-event tick, fake-timer advance, pass settlement,
writes, lifecycle commands) has a real 20-second bound that names what
it waited for. The bound uses the real `setTimeout`, captured before the
timers are faked. Each test has a 2-minute timeout instead of 30
minutes.
- The fixture is smaller (140 + 20 settled reviews instead of 700 + 80).
Warm-up is 6 simulated minutes instead of 12, and each write window is 6
minutes instead of 7.
- The full-pass guard uses a 15-minute full pass (`fullBackstopMs`)
instead of the hourly default. The fixture's `start` accepts
`fullBackstopMs` for this.

## Verification

- Old file, 4 copies at the same time (local CPU contention): all
passed, 190–211 s per file, 172 s of test time, 4 real run executions.
- New file, 4 copies at the same time: all passed, 48.5–49.1 s per file
(about 30 s of test time; each test 4–6 s), 0 run executions.
- The census still fails without the #21861 fix. With
`notifyRunDispatch` put back in `agent-configuration.ts`, the census
test fails with 480 settled checks (three whole-company reloads of 160)
and 519 `company` checks. With the fix, it has 0 settled checks.
- Server `tsc --noEmit` passed.

## Risks

- Test-only change. No product code changed.
- The guards now use idle agents and settled tasks rather than agents
with live runs, so they do not cover a pause or termination that must
stop a live run. Those flows have their own lifecycle tests. The
live-run census (`issue-recovery-live-run-agent-census.test.ts`) still
covers the hourly full pass with the real interval.
- The fixture is smaller, but the census still compares whole-company
reloads against zero.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 01:13:19 -06:00
..
…
…
…
…
…