mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 23:08:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts an agent's queued runs when a slot is free. #21234, #21277, #21349 and #21391 order them: person work first, a recovery share, aging, and a share for the longest-waiting run > - On hosted staging, one agent with 20 slots had 295 queued runs. 139 of them were system recovery wakes that entered the queue within one hour. Runs that people asked for waited 7 hours > - Two rules let this happen. Recovery took every fourth start even when a person's run waited, and an aged recovery run moved into the agent-work tier ahead of todo tasks. The continuation producer also admitted any continuation held past 5 minutes, however deep the queue was > - This pull request keeps recovery behind people's work, gives a person's long-waiting run a bounded start, re-checks a recovery run before it starts, and admits recovery to an agent's queue a few runs at a time > - The benefit is that a task a person asked for starts within a bounded number of dispatches, and a recovery burst cannot flood an agent's queue ## Linked Issues or Issue Description Refs #21234, #21277, #21349, #21391, #21405, #21433, #21502 **What happened?** On hosted staging (2026-11-13), one agent had `maxConcurrentRuns` 20 and all 20 slots held hour-long runs. 295 runs were queued. 139 had wake reason `issue_recovery_action_restored` (source `execution.reconciled`), enqueued between about 10:46Z and 11:47Z. Two tasks that people created were stalled by a restart at 07:25Z. Their next runs were queued at 07:29Z and were still queued 7 hours later. Both were system wakes: one `issue_monitor_due`, one `issue_continuation_needed` from the assigned-work sweep. Recent starts had waited from 1 to 290 minutes. **Why did it happen?** 1. The recovery share gave every fourth start to system recovery, also while a person's run waited. 2. A recovery run that waited one hour aged into the agent-work tier. Recovery tasks are usually in progress, so they sorted ahead of todo tasks in that tier. 3. The oldest-run share served the oldest aged run of any class. Older agent work and older recovery came before a person's run. 4. Only a wake that a person requested counted as person work. A system wake on a task that a person created and waits on (a monitor check, a continuation after a restart) competed as ordinary work or recovery. 5. `deliverReconciledExecutions` admitted a continuation held past `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) whatever the queue depth. At one per agent per 15-second pass, one agent got about 139 continuations in an hour. **Expected behavior** A run that a person created or asked for never waits behind system recovery. Waiting never fails or cancels it. A long wait moves it forward, so it starts within a bounded number of dispatches. Recovery enters the queue a few runs at a time. A recovery run that is no longer needed is dropped quietly. **Steps to reproduce** 1. Give an agent 20 slots, all busy. 2. Queue 60 agent assignments, then two system wakes on tasks people created (a monitor check and a continuation), then 30 later person runs on in-progress tasks, then a burst of 139 recovery wakes. 3. Free one slot at a time while people and agents keep adding work. 4. See that the monitor check starts only at dispatch 244. ## What Changed - `server/src/services/heartbeat/queued-run-dispatch-order.ts`: - System recovery has its own tiers below agent work (fresh, then backlog). Aging moves old backlog only into the fresh recovery tier, never ahead of agent work. - The recovery share applies only when no ready person run waits. It comes out of agent work, never out of a person's turn. Within the share, the longest-waiting aged recovery run goes first. - The oldest-run share serves the longest-waiting aged run that a person asked for or whose task a person created (`issues.created_by_user_id`), first. This includes a system continuation of such a task. Then it serves the longest-waiting aged agent-work run. It never serves other recovery. - New dispatch fact `personTask`, read with the queued tasks' status and priority (same query). - `queuedRunDispatchLine` (the O(n log n) place-in-line order from #21405) follows the same rules. `queued-run-line.test.ts` still proves it equals the dispatcher's order over 400 random queues, now with `personTask` in them. - `server/src/services/heartbeat/queued-run-line.ts`: the start-history counters use `dispatchHistoryStart` and `dispatchCountersFromStarts`, so dispatch, the place-in-line view and the tests share one rule. - New pure helpers `oldestShareCandidate`, `dispatchHistoryStart` and `dispatchCountersFromStarts`. The queue and the tests use the same counter rule (which includes the #21391 rule that a person's recovery restart does not use up the recovery share). - `server/src/services/heartbeat/queue.ts`: - At claim, a system recovery run (`issue_recovery_action_restored`, `issue_continuation_needed`, not requested by a person) is re-checked. If the task is reassigned, or another run continues it (started after it was queued, running, a queued non-recovery run, or a newer queued recovery run), the run is cancelled before it starts with `recovery_no_longer_needed` and its wake is skipped. This is the same quiet path as the existing closed-task check: no failure, no provider work. A person's undelivered message on the run still keeps it. - `server/src/services/recovery-queue-admission.ts` (new): `QUEUED_RECOVERY_RUNS_PER_AGENT` = 3 and the queue-depth check. - `server/src/services/execution-recovery-resolution.ts`: a continuation is admitted only while its agent has fewer than 3 queued recovery runs, fresh or backlog. The 5-minute hold bound still orders held continuations, but no longer overrides the cap. The rest stays pending in its recovery action. The existing continuation pass (every 15 s while any continuation is pending) admits the next one after a queued recovery run starts or is dropped. No new polling. - `server/src/services/recovery/legacy-workspace-rehome.ts`: the pending-wake delivery for rehomed and released-writer tasks uses the same cap. A held wake waits 60 s for its next turn and does not count as a refusal. - The oldest-run share counts as served only by a start that people wait on while its candidate is one, so older agent work never uses up a person's turn (`servesOldestShare`, used by the dispatcher, the place-in-line order and the start history, which now records `personWaiting`). - `server/src/services/recovery/assigned-work-sweep.ts`: a rehome or released-writer wake held by the cap (`HELD_RECOVERY_WAKE_KEYS`) counts as the task's pending path while its assignee has queued recovery runs. - `server/src/services/recovery/assigned-work-sweep.ts` (with #21502's stalled-queued-run replacement): a start of work people wait on does not count as passing over a queued run that people do not wait on. Waiting behind people's work is intended, so the sweep does not cancel such a run and replace it with a recovery continuation. Test: `assigned-work-stalled-queued-run.test.ts`, "leaves an automatic run that waits behind work on tasks people created". It fails without the rule. - `server/src/services/recovery/assigned-work-sweep.ts`: a pending continuation still counts as the task's wake path past `PENDING_CONTINUATION_BOUND_MS` while its agent has queued recovery runs, so the sweep does not treat a held task as stranded. **Bounds** - A run a person asked for, or any run on a task a person created: once it has waited past `QUEUED_RUN_AGING_MS` (1 hour), it starts within `OLDEST_DISPATCH_SHARE` × (such aged runs queued before it + 1) dispatches. Before that, runs a person requested already go first. There is no time limit, and waiting never cancels it. - Recovery run, proposed bounded behavior: at most 3 queued per agent. When no person run waits, an aged recovery run starts within `RECOVERY_DISPATCH_SHARE` × (aged recovery runs before it + 1) dispatches. At start it is re-validated against the current task, and it is dropped quietly if stale. While people keep the agent busy, recovery waits. It is held durably in its recovery action, not lost and not failed. ## Verification Regression test that reproduces today's shape (`server/src/services/heartbeat/queued-run-dispatch-order.test.ts`, "deep per-agent queue: a recovery burst behind 20 busy slots"). It uses 20 busy slots, one dispatch per freed slot, 60 older agent assignments, the two 07:29 system wakes on tasks people created (monitor check and continuation), 30 later person runs, a 139-run recovery burst, and new person and assignment arrivals on every step. The same model, run against master's dispatcher and this one over 600 dispatches: | | Before (master) | After | | --- | --- | --- | | Dispatch at which the 07:29 monitor check starts | 244 (≈12 h at ~20 starts/h) | 4 | | Dispatch at which the 07:29 continuation starts | 5 | 8 | | Recovery starts while a person-requested run waited | 89 | 0 | With person-requested runs in the 07:29 slots instead (the first version of the test), master starts them at dispatch 244 and 248, and this change at 4 and 8. - Red on master: the new dispatch tests fail (`expected -1 to be greater than or equal to 0`, and recovery started ahead of agent assignments). Green with this change. - Admission cap, embedded Postgres (`recovery-wake-herd.test.ts`, "queues at most a few recovery runs per agent..."): with the cap removed, 12 of 12 continuations enter one agent's queue in 12 passes. With the cap, 3 enter, the rest stay pending, and all 12 are delivered as queued runs leave. The queue never holds more than 3 recovery runs. - Older agent work (`queued-run-dispatch-order.test.ts`): with 60 older, higher-priority assignments and no person arrivals, an aged `issue_monitor_due` run on a person-created task starts within `OLDEST_DISPATCH_SHARE` in dispatch and in the place-in-line order. With the old reset rule it does not start in 8 dispatches. - Held restored-task wake (`legacy-workspace-rehome.test.ts`): with the agent's recovery queue full, the wake stays pending with no failure counted, the sweep reads the task as `pending_intent`, and the wake is delivered after a queued recovery run leaves. - Start-time re-check, embedded Postgres, real heartbeat queue (`heartbeat-stale-queue-invalidation.test.ts`): a recovery run whose task was reassigned, or that a newer run already continued, is cancelled with `recovery_no_longer_needed`, its wake is `skipped`, and the adapter is never called. A recovery run that the task still needs starts. - Updated tests that encoded the old rules: recovery's share and aging now come out of agent work, not out of a person's turn. Sweep liveness while held is covered. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: passes. ## Risks - Behavior change: a system run on a task a person created now takes the oldest-run share ahead of older agent work. On an agent whose queue is mostly such tasks, agent-made subtasks get that share less often. They still start in normal order. - Behavior change: while people keep an agent fully busy, system recovery for that agent waits. This is intended. Recovery is held durably and starts once no person run waits. - Behavior change: a queued system recovery run can be cancelled at start as no longer needed. It is cancelled before any provider work, with the same quiet outcome as a continuation whose task closed. - A continuation held behind the cap stays pending longer than 7 minutes. The sweep now treats it as live while its agent has queued recovery runs. If such queued runs can never start (for example dependency-blocked), the next continuations wait behind them. - One extra indexed read per recovery claim, on the existing `(company_id, context_snapshot->>'issueId', created_at)` index. No schema change. Mixed versions are safe: an old server keeps its old order until it is replaced. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edit). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>