Files
PaperClipAI/server/src
DottaandClaude Opus 5.5 ff971149c2 fix(dispatch): bound how long a queued run waits and keep a person's interrupted Retry person-first (#21349)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue decides which queued run an agent starts when a
slot frees up
> - Recent changes start person-initiated work first and give recovery
runs one start in four
> - A system continuation of a person's interrupted Retry ranked as
old-backlog recovery, and the recovery share always went to fresher
recovery runs
> - With a steady stream of new work, that continuation never started,
so the person's task sat queued for hours
> - This pull request adds aging and a longest-waiting share, keeps a
person's interrupted request person-first, and passes the turn on when a
long-queued run cannot be claimed
> - The benefit is that every ready queued run starts within a bounded
number of starts, and a person's Retry is not lost behind system work

## Linked Issues or Issue Description

**What happened?**

On a hosted staging instance, one agent had 28 runs running and kept
starting newer runs. The company had 67 queued runs, and 54 of them were
older than one hour. A person pressed Retry on a task. The Retry run was
interrupted by a server restart two minutes later. The system queued an
automatic continuation (`issue_continuation_needed`) with
`previousRunId` set to the interrupted Retry run. That continuation
stayed queued for more than 2.5 hours. The task showed as "working".
Other queued runs of the same person (some queued for more than 10
hours) also did not start.

Root cause:
- The dispatcher ranked the continuation as `recovery_backlog`, the last
tier, because it was a system wake and its source run ended more than 30
minutes ago. The person-first check only looked at wake requests
attached to the queued run itself. It did not look at the source run,
which was the person's `task_retry`.
- The 1-in-4 recovery share picked the first recovery run in sort order.
Fresh recovery runs sort before backlog runs, and fresh recovery runs
kept arriving, so the backlog run never got the share.
- Nothing aged a queued run. A run in a lower tier or with a lower
priority could wait forever.

**Expected behavior**

A system continuation of a person's Retry ranks as person-initiated.
Every ready queued run starts within a bounded number of starts, even
when fresh person and system work keeps arriving. A run that keeps
failing to claim is reported and does not block the next-oldest run.

**Steps to reproduce**

1. Give an agent one free slot and several queued runs.
2. Queue a person's `task_retry` run, start it, and interrupt it.
3. Queue a system continuation with `previousRunId` set to the
interrupted run, and let its source age past 30 minutes.
4. Keep adding fresh system and person work.
5. See that the continuation never starts.

**Paperclip version or commit**

Staging Core 7e7452e88f (contains #21234 and #21277). Fixed on top of
origin/master cb94108484.

**Deployment mode**

Hosted (authenticated, public exposure, managed cloud).

**Agent adapter(s) involved**

Any adapter. The bug is in queued-run dispatch order, before the adapter
starts.

Refs #21234, #21277

## What Changed

- `queued-run-dispatch-order.ts`: a ready run that has waited for one
hour (`QUEUED_RUN_AGING_MS`) moves up to at least the work tier.
- `queued-run-dispatch-order.ts`: every fourth start
(`OLDEST_DISPATCH_SHARE`) goes to the longest-waiting ready run that has
aged, whatever its class or priority. The dispatcher tracks this share
inside one pass.
- `queue.ts`: the dispatcher reads the agent's recent starts to know how
many starts passed over the oldest run. One query now serves both the
recovery share and the oldest-run share.
- `queue.ts`: a queued run is person-initiated when any run in its
source chain (`previousRunId`, then `retryOfRunId`, up to 8 runs, in one
recursive query) was a person's request: a `task_retry` or chat wake, a
run with a `taskRetryRequestId`, or a run whose wake request a user
made. A person's Retry stays first after repeated interruptions.
- `queue.ts`: when a run that has waited past the aging bound is offered
a slot and cannot be claimed, the server logs a warning with the reason
and the wait time. That run skips the oldest-run share for one aging
period, so the next-oldest run takes the turn. It is still offered a
slot in normal order on every pass.

## Verification

- `npx vitest run
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/recovery-wake-herd.test.ts` (38 passed).
- Red/green: with the old dispatcher and queue, these new tests fail and
the fix makes them pass:
- "starts an old queued continuation within a bounded number of starts
while fresh person and system work keeps arriving" (unit, continuous
stream of fresh person, work and recovery runs)
- "serves the longest-waiting run once every share even when fresher
work outranks it" (unit)
  - "lets an old-backlog recovery run age into the work tier" (unit)
- "starts the system continuation of a person's interrupted Retry ahead
of fresher system work" (embedded Postgres, the staging case)
- "gives the longest-waiting queued run a free slot after three fresher
starts passed it over" (embedded Postgres)
- "keeps a person's Retry first after its system continuation is
interrupted again" (embedded Postgres, two successive interruptions)
- "starts the oldest continuations of a long backlog within a bound
while later assignments and fresh recovery keep arriving" (unit, the
staging shape: 3 continuations 4-5 hours old behind 80 later
assignments)
- Added coverage that passes before and after: "starts a system
continuation that a person's Retry coalesced into ahead of fresher
system work", "passes the oldest-run share to the next oldest run while
the oldest one keeps failing to claim", and "does not use the oldest-run
share for a run that has not waited past the aging bound".
- Related suites pass: heartbeat-run-dispatch-work,
heartbeat-run-dispatch-lifecycle, heartbeat-queued-run-claim-isolation,
issue-reassignment-queued-run, heartbeat-startup-fresh-dispatch,
heartbeat-stale-queue-invalidation,
heartbeat-ephemeral-queued-migration, queued-comment-consumption (116
tests).
- `tsc --noEmit -p server` passes.
- Not related to this change: `heartbeat-stale-queue-invalidation`
"resumes a rejected connection_intent outcome and promotes the claimed
review task" also fails locally on the base commit without this change
(one queued run, so the new code does not run).

## Risks

- Behavior change: when runs have waited more than one hour, one start
in four goes to the oldest of them, even when fresher person work is
queued. A person's fresh work still gets the other three starts.
- More runs can rank as person-initiated: system continuations and
retries of a person's run. This is the intended outcome.
- One more small recursive query per dispatch pass (the source chain, at
most 8 hops per run, by primary key and the wake-request run index). It
only runs when more than one run is queued and some have a source run.
- The record of runs that failed to claim is in memory. After a restart,
a stuck run gets the oldest-run share again once, then is skipped again.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, browser for read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 04:59:56 -06:00
..
…
…
…
…
…
…
…