mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts an agent's queued runs. Each trigger (a run finishes, a wake arrives, a sweep runs) asks for one dispatch pass of that agent > - An agent could run more runs than its `maxConcurrentRuns`, dispatch passes could stack up without a limit, and every pass offered the same unclaimable runs again > - On hosted staging (e23549cd9c), one agent with limit 20 had 20–35 runs `running` and 170–270 queued. The logs showed 375 "offered a free slot but could not be claimed" lines and 117 "claim was rejected" lines in 10 minutes > - This pull request runs one pass per agent at a time, folds triggers into one follow-up pass, backs off refused claims, enforces the limit in the claim transaction, and reads only the columns the pass needs > - The benefit is that an agent never runs past its limit (also across two servers during a deploy), and a burst of triggers costs one or two passes, not one pass per trigger ## Linked Issues or Issue Description Refs #21391, #21242, #21234, #21277, #21324, #21349 **What happened?** On hosted staging (Coree23549cd9c), one agent ("Codie", `maxConcurrentRuns` 20) had 170–270 queued runs and 20–35 runs `running`, so it was over its limit. The logs showed a refused claim offered again on every pass. This pull request is about correctness and load. It is not the cause of the staging out-of-memory crashes: a separate leak (`sessionToolAuthorityEpochs` in `native-session-executor.ts`, about 3 MB per finished native run) explains those, and another change fixes it. Live again at 17:02Z on staging (Core04f12fd2be, which includes #21391): the same agent had 52 runs `running` against `maxConcurrentRuns` 40. All 52 started between 16:55Z and 17:02Z, right after a deploy released the backlog, with 112 runs still queued. So overlapping passes exceed the limit with claimable runs, not only with unclaimable ones. **Why did it happen?** 1. `withAgentStartLock` chained each trigger behind the previous one and measured its 30 s stale limit from when the predecessor was chained, not from when it started. When triggers arrived faster than one per pass, every trigger after about 30 s of backlog started its own pass next to the others. 2. Each pass counted free slots by itself, outside any lock that other passes or another server see. Overlapping passes and two servers during a deploy swap each claimed up to the limit. 3. Each pass read every queued run with `select *`, including the large `contextSnapshot`, and offered every refused run again with no backoff. A local reproduction of the staging shape by another session (250 unclaimable queued runs with 8 KB snapshots, one trigger per second, one23549cd9cand again on mastere7e28ba241): passes that run at the same time went 1 → 12 → 39 → 79 → 139 at t = 30/45/75/116/178 s, only 4 passes finished in 178 s, and post-GC heap grew by about 2.8 MB per stacked pass (519 MB → 1,061 MB at 190 passes, mostly the parsed `heartbeat_runs` rows and the dispatcher facts each pass holds). 242 s after the triggers stopped, 216 passes were still outstanding. **Expected behavior** At most one dispatch pass per agent runs at a time. Triggers that arrive during a pass cause one follow-up pass. The `running` count never goes above `maxConcurrentRuns`. A refused run is offered again after a bounded wait, and no queued run is stranded. **Steps to reproduce** 1. Give an agent a limit of 4, 12 aged queued runs that the chat-control gate refuses, and 12 claimable runs. 2. Start a pass. While its first claim waits, send 40 triggers, each after more than 30 s of the old lock's clock. 3. Count the passes, the refused offers, and the highest `running` count. ## What Changed - `server/src/services/agent-start-lock.ts`: replaces the stale-bypass lock with `dispatchForAgent`. One pass runs per agent. Triggers during a pass merge into one follow-up pass that all their callers share. A trigger from inside the agent's own pass does not wait on itself. A pass has a 60 s deadline on a monotonic clock, measured from when the pass started (not from when a waiter was chained). At the deadline the pass stops claiming (an abort signal) and the follow-up may start. A trigger that arrives after the deadline also ends the pass, so a stalled timer cannot keep an agent blocked. Post-pass effects run after the agent is free, as before. - `server/src/services/heartbeat/run-slot-admission.ts` (new): `assertAgentRunSlotFree` counts the agent's `running` runs and compares them with `maxConcurrentRunsSql()` inside the agent-scoped advisory lock that every claim already takes (`withExternalAdmissionGuard`). It throws `AgentRunSlotsFull` when the agent is full. The count leaves out the run that the claimed run replaces (`retryOfRunId` or `previousRunId`), so a successor that starts while its predecessor finishes is not refused. - `server/src/services/heartbeat/queue.ts`: - Every claim path (ordinary, queued comment, native review through `native-review-dispatch.ts`) checks the slot inside its claim transaction. A full agent leaves the run queued, and the pass stops. - The pass reads only the columns the dispatcher needs, with `issueId`, `wakeReason` and `previousRunId` taken from `contextSnapshot` in SQL. It reads the full row only for the run it is about to claim. - A refused claim (declined or 4xx) of a run that has waited more than an hour (the existing aging limit) is not offered again for 5 s, then 10, 20, 40, and at most 60 s. A person's run of that age waits 2 s. A run queued for less than an hour is never backed off, so a successor or retry that waits for its predecessor starts as soon as the predecessor ends. A backoff also ends when the run's task changes (status, `updatedAt` or execution run) or when a run of that task ends in this process. A backed-off run does not take the oldest-run share. The queued-run sweep (`resumeQueuedRuns`, `resumeQueuedRunsAfterDrain`, which the queued-run redrive calls) offers refused runs again regardless of the backoff. - A pass makes at most 24 claim attempts. When it reaches that limit it asks for one follow-up pass that skips every run the chain already tried, so the follow-up reaches runs further down the line. The chain ends when a pass does not reach the limit, so refused runs cannot cause a continuous chain of passes. Only this automatic follow-up carries the tried list. A wake, a completion or another fresh trigger that merges into it resets the list, so a run whose blocker cleared during the chain is offered again. - Admission, suppression and cutoff are read once per pass, not once per trigger. - Seam for a skip before the claim: in `runQueuedDispatchPass` (`claimQueuedRunsForAgent`), right after the skip line (`if (triedRunIds.has(candidate.id) || (!retryRefused && claimBackedOff(candidate.id, new Date(), taskStamp))) continue;`), before the claim-budget check, `claimAttempts += 1` and the full-row read. A `continue` there spends no claim attempt, reads no row, does not enter the claim backoff or the aged-decline note, and does not call `dispatcher.claimed`, so the run keeps its place and its oldest-run share. The dispatcher exposes `dispatcher.personOrigin(run)` (the person-origin fact it already derives) and `dispatcher.personRun(run)`. The candidate rows keep `id`, `companyId`, `agentId`, `status`, `invocationSource`, `retryOfRunId`, `wakeupRequestId`, `scheduledRetryAt`, `scheduledRetryAttempt`, `scheduledRetryReason`, `createdAt`, `startedAt`, and `contextSnapshot.issueId/wakeReason/previousRunId`. - Each pass is registered with the execution drain (`drainActiveRunExecutions`) until it ends or reaches its deadline. - Tests: new `server/src/services/agent-start-lock.test.ts` and `server/src/__tests__/queued-run-dispatch-stacking.test.ts`. `server/src/__tests__/heartbeat-start-lock.test.ts` is removed: it asserted the 30 s stale bypass that this pull request removes. `workspace-recovery-preparation-dispatch.test.ts` holds and counts passes through the new function. The chat-control fallback lookup from #21391 (`issueId = X OR nativeIssueId = X OR contextSnapshot->>'issueId' = X`) was measured on embedded Postgres with 200,000 runs in one company. Postgres uses a BitmapOr of three existing indexes (`heartbeat_runs_company_issue_created_idx`, `heartbeat_runs_company_native_issue_contract_id_uq`, `heartbeat_runs_company_ctx_issue_created_idx`), not a sequential scan: 0.33–0.45 ms per call for a recent previous run, a 30-day-old previous run, and no previous run. It is not changed. ## Verification Red/green, `server/src/__tests__/queued-run-dispatch-stacking.test.ts` (embedded Postgres, real heartbeat queue, limit 4, 12 refused aged runs, 12 claimable runs, 41 triggers): | Measure | master | this pull request | |---|---|---| | Passes | 42 | 2 | | Peak passes at the same time | 40 | 1 | | Peak claims in flight | 9 | 1 | | Refused-claim offers per trigger | 12 | 0.29 (12 in total) | | Peak `running` (limit 4) | 8 | 4 | | Passes outstanding after the burst | — | 0 | - "waits a bounded time before offering a declined/rejected run again": on master 10 triggers offer the refused run 10 times. With the fix they offer it once, and once the backoff passes it is offered again. The rejected case uses the 422 "Unable to resolve responsible user" path from the staging reproduction. - "starts a run that becomes ready during a chain of passes when a fresh trigger arrives": red at3d48c4bed2, green now. - "stops chaining passes once every refused run was tried": 60 refused runs and one trigger cause exactly 3 passes, then no more. - "starts runnable work queued behind more refused person runs than one pass tries": 30 refused person runs ahead of 3 runnable runs, with free slots. The runnable runs start after one trigger, in at most 3 passes. At68e64a2de9the same 24 person runs were offered on every pass and the runnable runs never started. - `server/src/services/agent-start-lock.test.ts`: one pass for a burst of 50 triggers plus one follow-up, no second pass after 45 s, a hung pass stopped at its deadline without overlap, no self-wait, settle after release, and errors passed to the caller. - `node server/scripts/typecheck-layers.mjs`: pass. - Vitest (embedded Postgres): `heartbeat/queue.test.ts` (102), `recovery-wake-herd.test.ts` (35), `heartbeat-queued-run-claim-isolation.test.ts`, `workspace-recovery-preparation-dispatch.test.ts`, `heartbeat-restart-claim-lapse.test.ts`, `issue-reassignment-queued-run.test.ts`, `queued-comment-consumption.test.ts`, `issue-queued-comment-queue.test.ts`, `chat-run-milestone-work.test.ts`. ## Risks - Refusals of runs queued for less than an hour are not backed off. A backoff there delayed successors that wait for in-process predecessor ownership, which can end without a task change (two CI tests failed with it). Per trigger, such a run costs at most one claim attempt. - A refused run that has waited more than an hour now waits up to 60 s (a person's run 2 s) before an event-driven pass offers it again, unless its task changes. The sweep and the redrive ignore the backoff. - A successor may start while the run it replaces is still `running`. The limit check leaves that one predecessor out of the count. - Claims of an agent that is at its limit now fail inside the transaction. A run that `executeRun` claims directly (for example a native resume) stays queued when the agent is full, and the next pass starts it. - A pass that passes its 60 s deadline can still finish one in-flight claim next to the follow-up pass. The limit check in the transaction keeps the `running` count within the limit. - No schema or migration change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>