mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The assigned-work sweep is the backstop that makes sure every open agent task has a live run, a pending path, or a named wait on a person > - It counts a task with a queued run as healthy. A queued run that the dispatcher declines on every pass is never started, so the task waits forever and the sweep never looks again > - On a hosted instance one agent had more than 30 queued continuations that the claim declined on every pass. The sweep saw each task as healthy > - This pull request makes the sweep recognize a queued run that the dispatcher keeps passing over, report it, and replace it once with a new continuation when it is automatic work > - The benefit is that an automatic task cannot stay stuck behind a run that never starts, while a person's work is never cancelled ## Linked Issues or Issue Description Refs #21391, #21349, #21433, #21394, #21443, #21450 **What happened?** An agent had more than 30 queued automatic continuations (`issue_continuation_needed`) that never started. The claim declined each one on every dispatch pass. #21391 fixed that specific cause in the chat-control gate. But the sweep still trusted any queued run as a pending path. A run that the claim declines for any persistent reason keeps the task "healthy" forever. **Expected behavior** A queued run that the dispatcher keeps passing over is not a healthy pending path. The sweep reports it. For automatic work it cancels the run through the run lifecycle and queues one new continuation in its place. For a person's work it reports the run and leaves it. **Steps to reproduce** 1. Queue an automatic run for a task and make the claim decline it persistently (for example, a chat-control admission marker that does not verify). 2. Let the run wait for more than one hour while its agent starts other runs. 3. Run the assigned-work sweep for the task. Before this change, it reports the task as `live_run` and does nothing. ## What Changed - New violation `queued_run_stalled` in `recovery/assigned-work-invariant.ts`. A queued run is stalled when all of these are true (read from the database, on the database clock): - it has been queued for longer than `QUEUED_RUN_AGING_MS` (60 minutes, from `queued-run-dispatch-order.ts`); - it is the oldest queued run of its agent; - its agent started more than `OLDEST_DISPATCH_SHARE` other runs after the run became its oldest queued run: after the later of the run's aging time and the moment the last older run of the agent started or finished. - A stalled run is not a live run, and it does not let the task's interruption projection ("working") count as a pending path. Its own wake is not counted as a pending wake. A stalled run on a blocked task still counts as live, so the block decides first. - `listSuspects` lets a task through when its only active run is a stalled queued run. - `act()` handles the violation after all existing early returns: - person work (the #21349 source-chain rule through `listPersonOriginSourceRunIds`, which this change exports, or an explicit user continuation): reported, never cancelled; - a replacement that stalls again (`replacesStalledRunId` in its context): reported, not cancelled again; - both log one warning per run, with the run, agent and how long it waited, and look again after `CONTINUATION_CAP_WINDOW_MS` (one hour); - automatic work: the existing claim under the task row lock re-checks the facts and the shared hourly cap and spacing. Inside that claim, before the person check, it reads the run's row version (`xmin`) and the number of wakes attached to it. Then the run is cancelled through the run lifecycle (`setRunStatusFromLive` from `queued`, with that version and wake count as its condition; wake cancelled, run event, `releaseIssueExecutionAndPromote` with `suppressImmediateRecovery`). A person's wake that joins the run after the check, or a claim that starts it, changes one of them, so the cancel does nothing. Then one continuation wake is queued with `previousRunId` and `replacesStalledRunId` set to the cancelled run. If the cancel does nothing, the unpublished claim entry is removed and nothing is queued. - New `heartbeat/stalled-queued-run.ts` builds that cancel from the run lifecycle functions. `heartbeat.ts` and `recovery/service.ts` pass it to the sweep. - Plain activity text: "Continued automatically: this task had waited over an hour to start while its agent started other work, so Paperclip queued it again." ### Why this signal - #21433 keeps refused claims and claim back-off in memory only (`declinedAgedClaimAt`, `claimBackoffs` in `heartbeat/queue.ts`). Another server cannot read them. Run rows can be read from any server, on the database clock. - After #21349, an aged run ranks no lower than ordinary work, and the oldest-run share starts the oldest ready queued run at least once in every `OLDEST_DISPATCH_SHARE` starts. So when a run has been its agent's oldest queued run since after the aging bound, and its agent started more runs in that time than the share allows, the dispatcher offered the run a slot and the claim declined it. Starts that happened while older runs were still waiting do not count: those older runs were getting their turns. - A run behind a full agent sees no such starts and is left alone. Under a slow backlog only the oldest queued run can match. So the younger queued runs keep their place in line. - Starts are counted only after the run reached 60 minutes of age, not after the current server's boot. 60 minutes is longer than a release swap overlap. So a run that only waits for a new server to reattach to its Runners does not match. #21450 will add a reattach-phase gate before this path. ### Why this does not set `retryOfRunId` An earlier plan added `retryOfRunId` to the sweep's continuation so the chat-control gate could follow it. An audit of its readers found three changes in behavior: the admission-time chat-control check would start to refuse sweep wakes; the claim would wait on the source run's cleanup (a possible wait forever); and the agent prompt would call the run "a Retry" that "re-delivers the wake". #21391 already makes these runs claimable. So this change leaves `retryOfRunId` unset, as today. The replacement links its source with `previousRunId`, as the sweep already does. ### Questions from review, answered from the code - **How often is the invariant pass triggered, and can two passes overlap?** The `assigned-work-invariant` delivery queue (`recovery/work.ts`, `scoped-deadline-work.ts`) runs on database-work notifications, on each task's own deadline (`retryAt`), and on a 30 s reconstruction of the suspect list. Passes are at least 1 s apart (`minimumPassIntervalMs` default). In one process, passes of the queue never overlap (`running` guard in `delivery-work-coordinator.ts`). Two servers can sweep at the same time. Each repair is serialized by the claim under the task row lock (`FOR UPDATE` in `claimContinuation`), which re-reads the facts and the spacing. This change does not add a new loop. - **Can two runs or no run exist for one task?** Two runs: no. The claim's spacing makes a second server back off, and the cancel happens before the new wake (a queued run on the task would otherwise absorb the wake). No run: there is a short gap between the cancel and the new wake, because wake admission locks the task row in its own transaction. If the server stops in that gap, the task has nothing pending, and the next pass repairs it as `no_pending_path` under the same cap. ## Verification - `server/src/__tests__/assigned-work-stalled-queued-run.test.ts` (new, embedded Postgres, real `createHeartbeatQueue` admission and `claimQueuedRun`): - an automatic run that the real claim declines persistently, whose agent started runs after it aged: reported, cancelled, exactly one replacement, and the real claim starts the replacement; - a run behind a full agent: left alone, not a suspect; - a person's declined Retry: reported, not cancelled, one warning; - a replacement that stalls again: not cancelled on two checks, one warning, next look one hour later; - two sweep instances on the same task at the same time: one replacement; - three aged runs queued in order, the first two get their turns slowly: the third is not stalled when it becomes oldest, and is stalled only after more than four further starts pass it over; - a person's message joins the run (real wake admission) between the claim and the cancel: the run is not cancelled, there is no replacement, and the person's wake stays attached to the run; - the run starts between the claim and the cancel: no cancel, no replacement. - First round, red before the fix: 4 failed, 1 passed (the full-agent case passes before and after). Green: 5 passed. - Review round: the three new tests on the previous head: 2 failed (the backlog case and the person-join case), 1 passed (the started-run case, which the `queued` check already covered). Green: 8 passed. - Existing files run one at a time: `assigned-work-invariant.test.ts` 31 passed, `assigned-work-sweep-round-trips.test.ts` 2 passed, `failed-continuation-continues-again.test.ts` 28 passed, `recovery-wake-herd.test.ts` 35 passed, `saved-work-recovery-overdue.test.ts` 4 passed. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: clean. - `pnpm verify`: not completed locally (the machine is heavily loaded and the run was stopped when the code changed). Required CI supplies the broad coverage. ## Risks - A task's oldest queued automatic run is cancelled and queued again at most once per run. Its replacement starts at the back of the line. Only a run that the dispatcher already passed over can match. - A replacement that stalls again is only reported, so a persistent claim refusal cannot cause repeated cancellations. - The new subqueries run only for queued runs older than one hour. They use the existing `(company_id, agent_id, started_at)` index. Healthy runs and the path before `turn.start` do not change. - #21443 (automatic-run brake) and #21450 (reattach first) touch `assigned-work-sweep.ts`. The new path comes after every early return that `act()` has today, so their gates can go before it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). A subagent on the same model audited the `retryOfRunId` readers. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>