mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue decides which queued run an agent starts when a slot frees up > - Recent changes start person-initiated work first and give recovery runs one start in four > - A system continuation of a person's interrupted Retry ranked as old-backlog recovery, and the recovery share always went to fresher recovery runs > - With a steady stream of new work, that continuation never started, so the person's task sat queued for hours > - This pull request adds aging and a longest-waiting share, keeps a person's interrupted request person-first, and passes the turn on when a long-queued run cannot be claimed > - The benefit is that every ready queued run starts within a bounded number of starts, and a person's Retry is not lost behind system work ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, one agent had 28 runs running and kept starting newer runs. The company had 67 queued runs, and 54 of them were older than one hour. A person pressed Retry on a task. The Retry run was interrupted by a server restart two minutes later. The system queued an automatic continuation (`issue_continuation_needed`) with `previousRunId` set to the interrupted Retry run. That continuation stayed queued for more than 2.5 hours. The task showed as "working". Other queued runs of the same person (some queued for more than 10 hours) also did not start. Root cause: - The dispatcher ranked the continuation as `recovery_backlog`, the last tier, because it was a system wake and its source run ended more than 30 minutes ago. The person-first check only looked at wake requests attached to the queued run itself. It did not look at the source run, which was the person's `task_retry`. - The 1-in-4 recovery share picked the first recovery run in sort order. Fresh recovery runs sort before backlog runs, and fresh recovery runs kept arriving, so the backlog run never got the share. - Nothing aged a queued run. A run in a lower tier or with a lower priority could wait forever. **Expected behavior** A system continuation of a person's Retry ranks as person-initiated. Every ready queued run starts within a bounded number of starts, even when fresh person and system work keeps arriving. A run that keeps failing to claim is reported and does not block the next-oldest run. **Steps to reproduce** 1. Give an agent one free slot and several queued runs. 2. Queue a person's `task_retry` run, start it, and interrupt it. 3. Queue a system continuation with `previousRunId` set to the interrupted run, and let its source age past 30 minutes. 4. Keep adding fresh system and person work. 5. See that the continuation never starts. **Paperclip version or commit** Staging Core7e7452e88f(contains #21234 and #21277). Fixed on top of origin/mastercb94108484. **Deployment mode** Hosted (authenticated, public exposure, managed cloud). **Agent adapter(s) involved** Any adapter. The bug is in queued-run dispatch order, before the adapter starts. Refs #21234, #21277 ## What Changed - `queued-run-dispatch-order.ts`: a ready run that has waited for one hour (`QUEUED_RUN_AGING_MS`) moves up to at least the work tier. - `queued-run-dispatch-order.ts`: every fourth start (`OLDEST_DISPATCH_SHARE`) goes to the longest-waiting ready run that has aged, whatever its class or priority. The dispatcher tracks this share inside one pass. - `queue.ts`: the dispatcher reads the agent's recent starts to know how many starts passed over the oldest run. One query now serves both the recovery share and the oldest-run share. - `queue.ts`: a queued run is person-initiated when any run in its source chain (`previousRunId`, then `retryOfRunId`, up to 8 runs, in one recursive query) was a person's request: a `task_retry` or chat wake, a run with a `taskRetryRequestId`, or a run whose wake request a user made. A person's Retry stays first after repeated interruptions. - `queue.ts`: when a run that has waited past the aging bound is offered a slot and cannot be claimed, the server logs a warning with the reason and the wait time. That run skips the oldest-run share for one aging period, so the next-oldest run takes the turn. It is still offered a slot in normal order on every pass. ## Verification - `npx vitest run src/services/heartbeat/queued-run-dispatch-order.test.ts src/__tests__/recovery-wake-herd.test.ts` (38 passed). - Red/green: with the old dispatcher and queue, these new tests fail and the fix makes them pass: - "starts an old queued continuation within a bounded number of starts while fresh person and system work keeps arriving" (unit, continuous stream of fresh person, work and recovery runs) - "serves the longest-waiting run once every share even when fresher work outranks it" (unit) - "lets an old-backlog recovery run age into the work tier" (unit) - "starts the system continuation of a person's interrupted Retry ahead of fresher system work" (embedded Postgres, the staging case) - "gives the longest-waiting queued run a free slot after three fresher starts passed it over" (embedded Postgres) - "keeps a person's Retry first after its system continuation is interrupted again" (embedded Postgres, two successive interruptions) - "starts the oldest continuations of a long backlog within a bound while later assignments and fresh recovery keep arriving" (unit, the staging shape: 3 continuations 4-5 hours old behind 80 later assignments) - Added coverage that passes before and after: "starts a system continuation that a person's Retry coalesced into ahead of fresher system work", "passes the oldest-run share to the next oldest run while the oldest one keeps failing to claim", and "does not use the oldest-run share for a run that has not waited past the aging bound". - Related suites pass: heartbeat-run-dispatch-work, heartbeat-run-dispatch-lifecycle, heartbeat-queued-run-claim-isolation, issue-reassignment-queued-run, heartbeat-startup-fresh-dispatch, heartbeat-stale-queue-invalidation, heartbeat-ephemeral-queued-migration, queued-comment-consumption (116 tests). - `tsc --noEmit -p server` passes. - Not related to this change: `heartbeat-stale-queue-invalidation` "resumes a rejected connection_intent outcome and promotes the claimed review task" also fails locally on the base commit without this change (one queued run, so the new code does not run). ## Risks - Behavior change: when runs have waited more than one hour, one start in four goes to the oldest of them, even when fresher person work is queued. A person's fresh work still gets the other three starts. - More runs can rank as person-initiated: system continuations and retries of a person's run. This is the intended outcome. - One more small recursive query per dispatch pass (the source chain, at most 8 hops per run, by primary key and the wake-request run index). It only runs when more than one run is queued and some have a source run. - The record of runs that failed to claim is in memory. After a restart, a stuck run gets the oldest-run share again once, then is skipped again. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edit, browser for read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>