Files
PaperClipAI/server
DottaandClaude Opus 5.5 ade7e0643f fix(recovery): pace recovery continuations and start person-initiated work first (#21234)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Execution recovery continues tasks that a restart or a lost run left
unfinished
> - Recent recovery changes restored many held tasks at the same time,
and each restored task got a queued run on its agent
> - The agent's queue starts runs oldest first, so a person's Retry or
chat message waited behind every older recovery run
> - On staging, one agent had 131 queued runs, and person Retries waited
more than 50 minutes
> - This pull request paces how many recovery continuations enter an
agent's queue, and it starts person-initiated work before system
recovery work
> - The benefit is that a burst of recovery cannot starve people, and
every recovery run still runs

## Linked Issues or Issue Description

Refs #21222, #21139, #21135

**What happened?**

Staging deployed #21139 and #21135 at 22:53Z. Within one hour, the
company had 201 queued runs and 78 running runs. 122 queued runs had the
recovery wake reason `issue_recovery_action_restored`. Each one was on a
different task. They all came from the reconciled-execution delivery
pass (`source: execution.reconciled`), at a rate of 5 to 15 per minute
from 22:57Z to 23:51Z. One agent had 131 queued runs. The oldest queued
run had waited since 22:53Z. A person's Retry at 23:14Z was still queued
4 minutes later. Six person Retries on that agent, queued from 23:08Z to
23:14Z, were still queued at 00:04Z, more than 50 minutes behind
recovery runs. At 00:04Z, that agent's newest start was a recovery run
created at 22:57Z. In the last 400 runs, 25 succeeded.

**Expected behavior**

A person's Retry, chat message, or assignment starts before system
recovery work on the same agent. Recovery work still drains at a steady
rate. All recovery runs run eventually, and none is dropped. A task
whose continuation waits shows that the task continues automatically. It
does not show that the task is idle, and it does not ask for Retry.

**Steps to reproduce**

1. Give an agent a concurrency limit of 1.
2. Queue many recovery continuations for the agent
(`issue_recovery_action_restored`), for example after a server restart
with many held tasks.
3. Click Retry on another task of the same agent.
4. See that the Retry stays queued until all older recovery runs start.

**Paperclip version or commit**

Staging build 2026.1106.0+2187.git.57aeda190

**Deployment mode**

Hosted cloud (staging)

**Agent adapter(s) involved**

paperclip_runner (native and legacy runtimes). The problem is in the
shared run queue, so it applies to all adapters.

## What Changed

- New module
`server/src/services/heartbeat/queued-run-dispatch-order.ts` decides the
order in which an agent's queued runs start. Person-initiated runs start
first: `task_retry`, external chat messages, and any wake that a user
requested. Other work comes next, together with recovery runs for
interruptions that ended in the last 30 minutes. Old-backlog recovery
runs come last. Inside each class, the existing order applies:
readiness, then priority, then age. Runs whose dependencies are not
ready stay last.
- Recovery runs get a bounded share. When the agent started 3 other runs
after the oldest queued recovery run arrived and none of them was
recovery work, the next slot goes to a recovery run. The share applies
across the whole batch when one dispatch fills many slots. Thus people
cannot starve recovery.
- `startNextQueuedRunForAgent` in
`server/src/services/heartbeat/queue.ts` uses the new order. It changes
only the ranking. It does not change `claimQueuedRun`, the admission
preflight, or the promoter.
- `deliverReconciledExecutions` in
`server/src/services/execution-recovery-resolution.ts` paces delivery.
It leaves a continuation pending when the agent already has a queued
recovery run, or when the agent's queue is at or above its concurrency
limit. One pass delivers at most 5 continuations per company and at most
1 per agent. The pass reads pending continuations in pages and leaves
out agents and companies that it already handled, so continuations for
busy agents cannot hide continuations for other agents. A continuation
that has waited 5 minutes or more is admitted even when ordinary work
keeps the queue full, and it goes ahead of fresher continuations,
longest wait first. Otherwise the most recent interruption goes first.
The existing 15-second continuation schedule delivers the remaining
items. This schedule runs only while pending continuations exist, so an
idle server does not poll.
- The assigned-work sweep
(`server/src/services/recovery/assigned-work-sweep.ts`) counts a pending
continuation as a pending wake for its task. Thus the invariant does not
see a held-back task as stranded, and it does not enqueue a second wake.

## Verification

- `pnpm exec vitest run
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/recovery-wake-herd.test.ts` (in `server/`): 17 tests pass.
- Red/green: with the source changes reverted to master, all tests in
`recovery-wake-herd.test.ts` fail. Each review fix also has a test that
fails on the head before that fix. With this change, they pass.
- `recovery-wake-herd.test.ts` covers these cases with embedded Postgres
and the real heartbeat queue:
- Six recovery wakes are queued, then a person's request. The first
dispatch starts the person's run. All six recovery runs start later.
- Four pending continuations for one agent: one pass delivers one. The
others stay pending and are delivered on later passes. The continuation
schedule stays armed while items are pending and returns null after all
items are delivered.
- A continuation stays pending while the agent's queue is at its
concurrency limit.
- The per-company cap applies, and the most recent interruption is
delivered first.
- A continuation that has waited past the bound is admitted while
ordinary work keeps the queue full.
- New interruptions arrive for the same agent on every 15-second pass,
and the older continuation is still admitted within the bound.
- Continuations for four busy agents (32 rows, more than one page) do
not hide an idle agent's continuation.
- A held-back continuation counts as a `queued_wake` pending intent for
the assigned-work invariant, and the task shows `working` with
`canRetry: false`.
- `queued-run-dispatch-order.test.ts` simulates dispatch: a Retry starts
ahead of 40 recovery wakes, and all 40 still run. With a person Retry
arriving at every step, every window of 4 starts includes 1 recovery
run. A fresh interruption's continuation is not put behind old-backlog
recovery runs. When one dispatch fills 8 slots, the order is PPPRPPPR (P
is a person run, R is a recovery run).
- Related suites pass: fresh-continuation fallback, issue recovery
actions, completed-turn cleanup hold, issue monitor scheduler, workspace
recovery fallback, assigned-work invariant, execution control work,
execution recovery resolution, and the heartbeat queue.
- `node scripts/typecheck-layers.mjs` (in `server/`) passes.

## Risks

- Behavior change: when a person's run waits behind recovery runs, it
now starts first. Recovery runs start a little later, but they still get
at least 1 of every 4 starts while people send work.
- Delivery of continuations is slower during a burst: at most 1 per
agent and 5 per company every 15 seconds. This is intentional. A
continuation that waits shows as continuing automatically, and the
invariant does not repair it twice.
- Ranking adds up to three small indexed reads when an agent has more
than one queued run. The reads run only when the queue has recovery runs
or wake requests to classify.
- No schema or migration changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, and read-only staging
API requests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:30:56 -06:00
..