Files
PaperClipAI/server/src
DottaandClaude Opus 5.5 1355f93546 fix(dispatch): schedule a clean loop-limit continuation with its person's work (#22312)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue decides which queued run an agent starts next.
It puts system recovery in the lowest tier.
> - When a run hits the tool-call loop limit, the server queues a
continuation with the wake reason `issue_continuation_needed`. The
source run succeeded. It did not fail.
> - The queue classed that continuation as system recovery. While any
person-requested run was queued, it got no share. It started only when
the 60-minute aging bound expired.
> - In a restart-chaos wave, 4 of 7 loop continuations waited 60:00 to
60:28. One task broke the "no rest within budget" rule. One run then
resumed an hour-old sandbox, timed out and showed a Retry banner.
> - This pull request schedules a clean loop-limit continuation in the
tier of its task, so a person's task that continues its own work stays
with that person's work.
> - Real failure recovery keeps the recovery tier and the aggregate
share, so a failure loop cannot become high-priority churn.
> - The benefit is that a person's task continues within seconds, not
one hour, and the stale-sandbox Retry path is not reached.

## Linked Issues or Issue Description

Refs #21349

**What happened?**
A continuation queued after the tool-call loop limit waited for the
60-minute aging bound (`QUEUED_RUN_AGING_MS`) on an agent that had
person-requested runs queued. 4 of 7 loop continuations in one
restart-chaos full wave waited 60:00 to 60:28.

**Expected behavior**
A continuation of a person's task whose source run succeeded starts in
the same tier as that person's work. Failure retries stay in the
recovery tier.

**Steps to reproduce**
1. Queue a person's task on an agent with a concurrency limit of 1.
2. Let the run hit the tool-call loop limit, so the server queues an
`issue_continuation_needed` continuation with
`livenessContinuationReason: agent_tool_call_loop`.
3. Keep person-requested runs queued on the same agent.
4. The continuation waits until it is 60 minutes old.

**Paperclip version or commit**
`master` at 70a4351343.

## What Changed

- `queued-run-dispatch-order.ts`: add a `cleanContinuation` dispatch
fact. A clean continuation is not system recovery, so it takes the tier
of its task (person or work) and the normal readiness, priority and age
order. The module stays domain-agnostic: the wake reason is defined once
(`ISSUE_CONTINUATION_NEEDED_WAKE_REASON`, used in
`RECOVERY_WAKE_REASONS`).
- `queued-run-dispatch-order.ts`: one core predicate
(`countsAsRecoveryStart`) drives `isSystemRecoveryRun`, the in-pass
recovery counter, the place-in-line counter and the start-history
counter. Every start with a recovery wake reason that a person did not
ask for still uses up the recovery share. A clean continuation therefore
does not add recovery-reason starts.
- New `queued-run-provenance.ts`: the clean-continuation causes, the
source-run check and `QUEUED_RUN_ORDER_CONTEXT_KEYS`.
- `queue.ts` (`listQueuedRunCandidates`, the real dispatch query) and
`queued-run-line.ts` (the place-in-line query) both read
`QUEUED_RUN_ORDER_CONTEXT_KEYS`, so the dispatcher sees `retryReason`,
`livenessContinuationReason` and `livenessContinuationSourceRunId`.
- `queued-run-line.ts`: set `cleanContinuation` only when all of these
are true:
- the wake reason is `issue_continuation_needed` and
`livenessContinuationReason` is `agent_tool_call_loop`;
- the run has no `retryOfRunId`, `retryReason` or `previousRunId` (it is
not a failure retry);
- the source run (`livenessContinuationSourceRunId`, same company) has
status `succeeded`;
- the source run is not itself a loop continuation, so only the first
hop of a loop chain gets the faster tier.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/services/heartbeat/queued-run-line.test.ts
src/services/execution-projection-agent-queue.test.ts` passes (39
tests), plus `src/services/heartbeat/`,
`queued-run-dispatch-stacking.test.ts` and `recovery-wake-herd.test.ts`.
- Red before the fix, green after it:
- `starts a person's task continuing after the tool-call loop limit with
that person's work, not after the aging bound` (unit; failed with
`expected 'recovery' to be 'person'`)
- `counts a clean continuation's start against system recovery's share,
as the start history does` (unit)
- `starts a person's task continuing after the tool-call loop limit with
that person's work, and keeps failure recovery behind it` (embedded
Postgres; a failure retry, a chained loop continuation and a
continuation of a failed run all stay behind the person's work)
- Assembled heartbeat service
(`src/__tests__/heartbeat-clean-continuation-dispatch.test.ts`, embedded
Postgres, `resumeQueuedRunsForAgent` with one free slot), checks which
queued run actually starts:
- `starts a clean loop-limit continuation of a person's task before a
newer person assignment` (red without the `queue.ts` change, green with
it)
- `starts a newer person assignment before a failure retry of a person's
task` (guard)
- Guard that passes before and after: `keeps a failure retry of a
person's task in the recovery tier while people wait`.
- The `queuedRunDispatchLine` fuzz test now also generates clean
continuations and still matches the dispatcher order for 400 seeds.
- `pnpm verify` ran locally.

## Risks

- Low risk. The change is limited to dispatch order. No schema or API
change.
- A clean continuation can now start ahead of agent work. It is bounded:
one continuation per source run (existing idempotency key), only the
first hop of a loop chain, and only after a succeeded source run. A loop
chain's later hops and all failure retries stay in the recovery tier
with the existing share.
- Recovery-reason starts per hour do not increase. A clean continuation
start uses up recovery's share like before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code as an agent with tool use (shell, file edit, test runs). Extended
reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 05:24:53 -06:00
..
…
…
…
…
…