Files
PaperClipAI/server
DottaandClaude Opus 5.5 d216e413f3 test(recovery): stop the idle-backstop suite starting real runs and bound every wait (#21924)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The task recovery suites drive real recovery workers against an
embedded database with a faked clock
> - `issue-recovery-idle-backstop.test.ts` seeds 50 agents that take
on-demand wakes, so recovery starts real `process` runs while timers are
faked. A run then waits on timers that only move when the test advances,
and the test waits for the run's passes with no time limit
> - On master's readiness run for b4fa1c7fb1 (run 43861616674, attempt
2), server shard 6/10 went silent inside this file and was cancelled at
the 20-minute job limit. Locally the same test timed out at 30 minutes
in larger runs, and then the next test's setup hook timed out at the
database truncate
> - This pull request stops the file's agents from starting runs, as
#21867 did for the agent-row census, and puts a time limit on every
wait, with a message that names what it waited for
> - The benefit is that this suite can no longer hang a CI shard
silently; a stuck step fails within 20 seconds and says why

## Linked Issues or Issue Description

Refs #21867, #21872, #21905

**What happened?**
Server shard 6/10 of the readiness run for master b4fa1c7fb1 (run
43861616674, attempt 2, job 133284921313) printed "running 199 of 1988
suites", logged a completion timing line for 197 files, and was then
silent from 12:15:05 until it was cancelled at 12:24:24.
`issue-recovery-idle-backstop.test.ts` started at 12:12:47 and never
logged its completion line. Its last output was at 12:13:34, in
"recovers a task whose hint was never sent within the backstop bound".

Locally, the same test hit its 30-minute test timeout in two larger
multi-file runs, and the following tests' `beforeEach` (`TRUNCATE
companies CASCADE`) hit the 30-second hook timeout. The logs of those
runs show real runs executing inside this test: a process run starting
("running without injected PAPERCLIP_API_KEY") and
`LegacyControllerLeaseLostError` raised from a faked-timer deadline in
`legacy-controller-lease.ts`.

Cause: `seedRecoveryPopulation` creates agents with `wakeOnDemand:
true`, and the held agent in this file too. When recovery finds a
stranded task it wakes the agent, and the wake starts a real run at once
(`startNextQueuedRunForAgent`). The run's legacy controller lease and
other waits use timers that the test fakes, while lease checks use the
database clock. So the run either loses its lease when the fake clock
jumps, or waits on a frozen timer. Meanwhile `advance()` waited for
every delivery pass with no time limit. Whether a pass ends up waiting
on such a run depends on timing, which is why the hang is rare.

**Expected behavior**
No fixture agent in this suite can start a run, and every wait fails
with a named message within a fixed time.

**Steps to reproduce**
Run `issue-recovery-idle-backstop.test.ts` and grep its output for
"running without injected PAPERCLIP_API_KEY" or "Legacy controller lease
lost": 11 to 13 occurrences per run on master. The hang itself is rare
(see Verification).

**Paperclip version or commit**
`master` at 9de343b3fe.

## What Changed

- `helpers/recovery-population.ts`: optional `wakeOnDemand` (default
`true`, so the other suites are unchanged).
- `issue-recovery-idle-backstop.test.ts`:
  - The population and the held agent take no on-demand wakes.
- The two tests that asserted a run was created now assert that recovery
asked for a wake of the task. The wake queue records each one as skipped
(`heartbeat.wakeOnDemand.disabled`), so the assertion still proves
recovery found the task and acted.
- Every wait has a 20-second real-time limit with a message naming the
step: seeding, worker start, each simulated second, each set of pending
delivery passes (listed by queue), each write and read. Per-test limit 2
minutes, down from 30.
- `issue-recovery.test-fixture.ts`: records the queue of each pending
pass (`pendingPassQueues()`), so a timed-out wait names the queues it
was still waiting for.

Not a product bug: no product code kept a transaction open after
shutdown. The truncate timeouts came after the previous test timed out
with real runs still in flight.

## Verification

- Real runs started during one run of the file: 11 to 13 on master (two
runs), 0 after.
- The file alone: 4 passed (98 s).
- Loop after the change: 15 runs (5 rounds of 3 in parallel): 15 passed,
0 hung or failed, 0 real run starts.
- Before the change, my local reproduction rate was low: 3 failing
multi-file runs out of about 10 large runs today (one silent hang, two
with this test's 30-minute timeout). 0 of 14 targeted runs (alone, alone
under load, and the 4-file combination) failed. So the loop shows the
change holds; it cannot show a before/after hang count. The
deterministic difference is the run starts above.
- Master shard 6/10 for b4fa1c7fb1: see "What happened?". My local
replay of the sharding script for that commit gave 201 of 2004 suites,
against CI's 199 of 1988. The CI environment excludes some suites, so I
identified the file from the job log, not from the replay.

## Risks

- The two tests now prove that a wake was asked for, not that a run was
created. Creating a run from a wake is covered by the wake-queue and
dispatch suites, and these tests are about recovery finding the task.
- Test-only change. No product code changed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 13:01:51 -06:00
..