mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The recovery system starts automatic runs: continuations, retries, and sweep replacements > - #21443 added a per-company automatic-run brake. It stops recovery from creating new automatic runs while a company's runs keep failing before they start > - The brake did not hold automatic runs that were already queued. The dispatcher still started them, so a burst of queued recovery runs kept failing in the same way > - This pull request adds the brake's start gate to the queued-run dispatch pass, and wakes a held company at its next try without a polling loop > - The benefit is that a failing company stops spending runs on automatic work, people's work still starts at once, and held work starts again on its own when the brake releases ## Linked Issues or Issue Description Refs #21443 Refs #21433 Refs #21427 Refs #21505 **What happened?** While the automatic-run brake was open for a company, its already-queued automatic runs (continuations and retries) still started. Each one failed before it started, for the same cause that opened the brake. **Expected behavior** A queued automatic run must stay queued while the brake is open. Only the brake's single probe, and then the half-open release rate (2, 4, 8 per window), can start one. A person's work (a Retry, or a continuation of a person's request) must never be held. Held runs must start again on their own, with nobody acting. **Steps to reproduce** 1. Make 20 automatic runs in one company fail before they start, so the brake opens. 2. Queue automatic continuations for that company. 3. Run a dispatch pass. Before this change, the continuations start. ## What Changed - `server/src/services/heartbeat/queue.ts`: the dispatch pass asks `reserveAutomaticRunStart` after the tried/backoff skip and the claim-attempt budget check, and before the budget increment and the full row read. A held run spends no claim attempt, does no row read, and does not enter the declined-claim backoff. If the brake grants the start (its probe or a half-open release start) but the claim then returns null, throws, or finds the row gone, the pass gives the start back, so another automatic run can use it in the same pass. The dispatcher's person-origin answer is passed as is, and it is `undefined` only when the dispatcher did not check origin (an agent with one queued run). Then the brake's own cached person check decides. - `server/src/services/heartbeat/queued-run-line.ts`: `queuedRunDispatchFacts` also returns `personOriginChecked`, so the queue knows when the dispatcher's negative answer is real. - `server/src/services/heartbeat/dispatch-work.ts`: the queued-run dispatch deadline worker adds companies whose next try is due to its scopes, and includes the next try time in its next wake. A held company wakes for its next try without a polling loop. If the brake's bounded wake list had to drop a company, the worker wakes all companies at the earliest dropped time. - `server/src/services/recovery/automatic-run-brake.ts`: when the brake holds a run, it records the company's next try time (probe time or the next release window) in a bounded map. A probe or half-open release grant returns a release that gives the start back with a compare-and-set on its own run or window. It exports `reserveAutomaticRunStart`, `takeDueAutomaticRunWakes`, and `nextAutomaticRunWakeAt`. The brake now uses `isRecoveryRun`, `isPersonOwnedFailureCode`, and `isRestartClassFailureCode` from `continuation-cause.ts` directly. The rules are the same as before; the lazy loader keeps only the imports that would make an import cycle. - `server/src/__tests__/automatic-run-start-gate.test.ts` (new, 11 tests through the real heartbeat queue): bounded queries for 250 held runs over 100 passes, and for 1,500 held runs (past the 1,024-run cache) over 3 passes; a declined probe claim gives the probe back and another automatic run starts in the same pass, and the probe stays with that run on a later pass; an unused half-open start goes back to its window; with 276 held companies (past the 256-company wake list) every company is still woken; a work run behind 250 held runs starts in the same pass; a person's Retry is never held; the only queued run is not held when it continues a person's Retry; exactly one probe starts across two server instances; the release gives 2, 6, 14 cumulative starts over three windows; held runs are not in the declined-claim backoff after the pause; the dispatch worker wakes the company at the next try; and an end-to-end test that a held automatic run stays queued while the brake is open, then starts and succeeds on its own at the next try. - Two small test updates in `automatic-run-brake.test.ts` and `automatic-run-brake-import.test.ts` for the new cache size field and the moved rule. - `server/src/__tests__/automatic-run-brake-recovery-passes.test.ts`: one test for the sweep path from #21505 (continue a blocked task behind a run refused at dispatch). The sweep only records the continuation (no wake, no new run), and `deliverReconciledExecutions`, which the brake already gates, creates the run only after the pause. No second gate is needed on that path. ## Verification - `npx vitest run src/__tests__/automatic-run-start-gate.test.ts`: 11 passed. - Red before for the review fixes: on the first PR head, the 1,500-run cost test made 3,848 queries (bound 130), the declined-probe test started 0 runs (expected 1), and the 276-company wake test missed 20 companies. - Red before: with the `queue.ts` gate line removed, 5 of the 8 tests fail (bounded pass, single probe, release rate, backoff, end-to-end). - Red before for the person case: with the dispatcher's `false` passed through as final, "never holds the only queued run when it continues a person's Retry" fails (0 started). It passes after the change. - `automatic-run-brake.test.ts` 19 passed, `automatic-run-brake-import.test.ts` 1 passed, `automatic-run-brake-recovery-passes.test.ts` 28 passed, `failed-continuation-continues-again.test.ts` 32 passed (after merging master with #21505). - Nearby queue tests (on the final head): `heartbeat-run-dispatch-work` 15, `queued-run-dispatch-stacking` 6, `heartbeat-queued-run-claim-isolation` 10, `assigned-work-stalled-queued-run` 12, all passed. - `server` `tsc --noEmit` is clean. ## Risks - Behavior change only while a company's brake is open or half-open. When the brake is closed, the gate returns from its cache with no query. - The first time the gate sees a non-person automatic run while the brake is on, the brake does one person-origin query for that run. The result is cached per run (bounded to 1,024 runs). - No migration. No API or UI change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>