mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The task recovery suites drive real recovery workers against an embedded database with a faked clock > - `issue-recovery-idle-backstop.test.ts` seeds 50 agents that take on-demand wakes, so recovery starts real `process` runs while timers are faked. A run then waits on timers that only move when the test advances, and the test waits for the run's passes with no time limit > - On master's readiness run forb4fa1c7fb1(run 43861616674, attempt 2), server shard 6/10 went silent inside this file and was cancelled at the 20-minute job limit. Locally the same test timed out at 30 minutes in larger runs, and then the next test's setup hook timed out at the database truncate > - This pull request stops the file's agents from starting runs, as #21867 did for the agent-row census, and puts a time limit on every wait, with a message that names what it waited for > - The benefit is that this suite can no longer hang a CI shard silently; a stuck step fails within 20 seconds and says why ## Linked Issues or Issue Description Refs #21867, #21872, #21905 **What happened?** Server shard 6/10 of the readiness run for masterb4fa1c7fb1(run 43861616674, attempt 2, job 133284921313) printed "running 199 of 1988 suites", logged a completion timing line for 197 files, and was then silent from 12:15:05 until it was cancelled at 12:24:24. `issue-recovery-idle-backstop.test.ts` started at 12:12:47 and never logged its completion line. Its last output was at 12:13:34, in "recovers a task whose hint was never sent within the backstop bound". Locally, the same test hit its 30-minute test timeout in two larger multi-file runs, and the following tests' `beforeEach` (`TRUNCATE companies CASCADE`) hit the 30-second hook timeout. The logs of those runs show real runs executing inside this test: a process run starting ("running without injected PAPERCLIP_API_KEY") and `LegacyControllerLeaseLostError` raised from a faked-timer deadline in `legacy-controller-lease.ts`. Cause: `seedRecoveryPopulation` creates agents with `wakeOnDemand: true`, and the held agent in this file too. When recovery finds a stranded task it wakes the agent, and the wake starts a real run at once (`startNextQueuedRunForAgent`). The run's legacy controller lease and other waits use timers that the test fakes, while lease checks use the database clock. So the run either loses its lease when the fake clock jumps, or waits on a frozen timer. Meanwhile `advance()` waited for every delivery pass with no time limit. Whether a pass ends up waiting on such a run depends on timing, which is why the hang is rare. **Expected behavior** No fixture agent in this suite can start a run, and every wait fails with a named message within a fixed time. **Steps to reproduce** Run `issue-recovery-idle-backstop.test.ts` and grep its output for "running without injected PAPERCLIP_API_KEY" or "Legacy controller lease lost": 11 to 13 occurrences per run on master. The hang itself is rare (see Verification). **Paperclip version or commit** `master` at9de343b3fe. ## What Changed - `helpers/recovery-population.ts`: optional `wakeOnDemand` (default `true`, so the other suites are unchanged). - `issue-recovery-idle-backstop.test.ts`: - The population and the held agent take no on-demand wakes. - The two tests that asserted a run was created now assert that recovery asked for a wake of the task. The wake queue records each one as skipped (`heartbeat.wakeOnDemand.disabled`), so the assertion still proves recovery found the task and acted. - Every wait has a 20-second real-time limit with a message naming the step: seeding, worker start, each simulated second, each set of pending delivery passes (listed by queue), each write and read. Per-test limit 2 minutes, down from 30. - `issue-recovery.test-fixture.ts`: records the queue of each pending pass (`pendingPassQueues()`), so a timed-out wait names the queues it was still waiting for. Not a product bug: no product code kept a transaction open after shutdown. The truncate timeouts came after the previous test timed out with real runs still in flight. ## Verification - Real runs started during one run of the file: 11 to 13 on master (two runs), 0 after. - The file alone: 4 passed (98 s). - Loop after the change: 15 runs (5 rounds of 3 in parallel): 15 passed, 0 hung or failed, 0 real run starts. - Before the change, my local reproduction rate was low: 3 failing multi-file runs out of about 10 large runs today (one silent hang, two with this test's 30-minute timeout). 0 of 14 targeted runs (alone, alone under load, and the 4-file combination) failed. So the loop shows the change holds; it cannot show a before/after hang count. The deterministic difference is the run starts above. - Master shard 6/10 forb4fa1c7fb1: see "What happened?". My local replay of the sharding script for that commit gave 201 of 2004 suites, against CI's 199 of 1988. The CI environment excludes some suites, so I identified the file from the job log, not from the replay. ## Risks - The two tests now prove that a wake was asked for, not that a run was created. Creating a run from a wake is covered by the wake-queue and dispatch suites, and these tests are about recovery finding the task. - Test-only change. No product code changed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>