Files
PaperClipAI/server
DottaandClaude Opus 5.5 c36b96ae00 fix(recovery): hold the start of queued automatic runs while a company's automatic-run brake is on (#21561)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The recovery system starts automatic runs: continuations, retries,
and sweep replacements
> - #21443 added a per-company automatic-run brake. It stops recovery
from creating new automatic runs while a company's runs keep failing
before they start
> - The brake did not hold automatic runs that were already queued. The
dispatcher still started them, so a burst of queued recovery runs kept
failing in the same way
> - This pull request adds the brake's start gate to the queued-run
dispatch pass, and wakes a held company at its next try without a
polling loop
> - The benefit is that a failing company stops spending runs on
automatic work, people's work still starts at once, and held work starts
again on its own when the brake releases

## Linked Issues or Issue Description

Refs #21443
Refs #21433
Refs #21427
Refs #21505

**What happened?**
While the automatic-run brake was open for a company, its already-queued
automatic runs (continuations and retries) still started. Each one
failed before it started, for the same cause that opened the brake.

**Expected behavior**
A queued automatic run must stay queued while the brake is open. Only
the brake's single probe, and then the half-open release rate (2, 4, 8
per window), can start one. A person's work (a Retry, or a continuation
of a person's request) must never be held. Held runs must start again on
their own, with nobody acting.

**Steps to reproduce**
1. Make 20 automatic runs in one company fail before they start, so the
brake opens.
2. Queue automatic continuations for that company.
3. Run a dispatch pass. Before this change, the continuations start.

## What Changed

- `server/src/services/heartbeat/queue.ts`: the dispatch pass asks
`reserveAutomaticRunStart` after the tried/backoff skip and the
claim-attempt budget check, and before the budget increment and the full
row read. A held run spends no claim attempt, does no row read, and does
not enter the declined-claim backoff. If the brake grants the start (its
probe or a half-open release start) but the claim then returns null,
throws, or finds the row gone, the pass gives the start back, so another
automatic run can use it in the same pass. The dispatcher's
person-origin answer is passed as is, and it is `undefined` only when
the dispatcher did not check origin (an agent with one queued run). Then
the brake's own cached person check decides.
- `server/src/services/heartbeat/queued-run-line.ts`:
`queuedRunDispatchFacts` also returns `personOriginChecked`, so the
queue knows when the dispatcher's negative answer is real.
- `server/src/services/heartbeat/dispatch-work.ts`: the queued-run
dispatch deadline worker adds companies whose next try is due to its
scopes, and includes the next try time in its next wake. A held company
wakes for its next try without a polling loop. If the brake's bounded
wake list had to drop a company, the worker wakes all companies at the
earliest dropped time.
- `server/src/services/recovery/automatic-run-brake.ts`: when the brake
holds a run, it records the company's next try time (probe time or the
next release window) in a bounded map. A probe or half-open release
grant returns a release that gives the start back with a compare-and-set
on its own run or window. It exports `reserveAutomaticRunStart`,
`takeDueAutomaticRunWakes`, and `nextAutomaticRunWakeAt`. The brake now
uses `isRecoveryRun`, `isPersonOwnedFailureCode`, and
`isRestartClassFailureCode` from `continuation-cause.ts` directly. The
rules are the same as before; the lazy loader keeps only the imports
that would make an import cycle.
- `server/src/__tests__/automatic-run-start-gate.test.ts` (new, 11 tests
through the real heartbeat queue): bounded queries for 250 held runs
over 100 passes, and for 1,500 held runs (past the 1,024-run cache) over
3 passes; a declined probe claim gives the probe back and another
automatic run starts in the same pass, and the probe stays with that run
on a later pass; an unused half-open start goes back to its window; with
276 held companies (past the 256-company wake list) every company is
still woken; a work run behind 250 held runs starts in the same pass; a
person's Retry is never held; the only queued run is not held when it
continues a person's Retry; exactly one probe starts across two server
instances; the release gives 2, 6, 14 cumulative starts over three
windows; held runs are not in the declined-claim backoff after the
pause; the dispatch worker wakes the company at the next try; and an
end-to-end test that a held automatic run stays queued while the brake
is open, then starts and succeeds on its own at the next try.
- Two small test updates in `automatic-run-brake.test.ts` and
`automatic-run-brake-import.test.ts` for the new cache size field and
the moved rule.

- `server/src/__tests__/automatic-run-brake-recovery-passes.test.ts`:
one test for the sweep path from #21505 (continue a blocked task behind
a run refused at dispatch). The sweep only records the continuation (no
wake, no new run), and `deliverReconciledExecutions`, which the brake
already gates, creates the run only after the pause. No second gate is
needed on that path.

## Verification

- `npx vitest run src/__tests__/automatic-run-start-gate.test.ts`: 11
passed.
- Red before for the review fixes: on the first PR head, the 1,500-run
cost test made 3,848 queries (bound 130), the declined-probe test
started 0 runs (expected 1), and the 276-company wake test missed 20
companies.
- Red before: with the `queue.ts` gate line removed, 5 of the 8 tests
fail (bounded pass, single probe, release rate, backoff, end-to-end).
- Red before for the person case: with the dispatcher's `false` passed
through as final, "never holds the only queued run when it continues a
person's Retry" fails (0 started). It passes after the change.
- `automatic-run-brake.test.ts` 19 passed,
`automatic-run-brake-import.test.ts` 1 passed,
`automatic-run-brake-recovery-passes.test.ts` 28 passed,
`failed-continuation-continues-again.test.ts` 32 passed (after merging
master with #21505).
- Nearby queue tests (on the final head): `heartbeat-run-dispatch-work`
15, `queued-run-dispatch-stacking` 6,
`heartbeat-queued-run-claim-isolation` 10,
`assigned-work-stalled-queued-run` 12, all passed.
- `server` `tsc --noEmit` is clean.

## Risks

- Behavior change only while a company's brake is open or half-open.
When the brake is closed, the gate returns from its cache with no query.
- The first time the gate sees a non-person automatic run while the
brake is on, the brake does one person-origin query for that run. The
result is cached per run (bounded to 1,024 runs).
- No migration. No API or UI change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:33:53 -06:00
..