Files
PaperClipAI/server
DottaandClaude Opus 5.5 fde0328ff5 fix(recovery): deliver every pending continuation within a bounded time (#21324)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Execution recovery continues a task in a fresh run after a restart
or a lost run left it unfinished
> - #21234 paced these continuations: at most 1 per agent and 5 per
company per pass, and a continuation waits while its agent already has a
queued recovery run or a full queue
> - A continuation that waited 5 minutes must enter the queue anyway,
but that bound applied only to the full-queue check. The "agent already
has a queued recovery run" check was in the page query, and it had no
bound.
> - On staging, an agent had queued recovery runs that did not start for
hours. Its pending continuation was never read again, and its task
showed "working" with nothing running or queued
> - This pull request applies the 5-minute bound to both checks, stops
one undeliverable continuation from using its agent's or company's slot,
and stops the task view from showing "working" for a continuation that
is overdue
> - The benefit is that every pending continuation enters the queue
within a bounded time, also after a restart, and the pacing still
protects people from recovery herds

## Linked Issues or Issue Description

Refs #21234, #21135

**What happened?**

On staging, a task had a fresh-continuation hold with
`continuationDelivery: "pending"` since about 03:30Z. More than 3.5
hours later, no run had started for the task. The task view showed the
interruption state `working` with `canRetry: false`, so the user saw
nothing wrong. The task's agent had 20 concurrency slots and 11 running
runs. It also had queued `issue_continuation_needed` runs from 22:54Z
that had not started. The delivery pass reads pending continuations with
a query that leaves out every agent that has a queued recovery run
(`issue_recovery_action_restored` or `issue_continuation_needed`). That
condition had no time bound, so the pass never read this continuation
again. The 5-minute admission bound was never reached. Across the
company, 203 open tasks showed `working` while the API showed no queued
or running run for them. 162 queued runs were more than 1 hour old.

**Expected behavior**

Every pending continuation enters its agent's queue within a bounded
time, also when the agent's queue holds recovery runs that do not start,
and also after a server restart. Pacing still limits how many
continuations enter a queue at the same time. A task does not show
"working" when its continuation is overdue and nothing runs or is
queued.

**Steps to reproduce**

1. Give an agent a concurrency limit of 1.
2. Put one queued `issue_recovery_action_restored` run on that agent
that does not start.
3. Record a pending fresh continuation for another task of the same
agent.
4. Wait more than 5 minutes, and let the 15-second continuation pass
run.
5. See that the continuation stays `pending`, the task shows `working`,
and no run starts.

**Paperclip version or commit**

master at bbafa78fe2 (includes #21234, ade7e0643f)

**Deployment mode**

Hosted cloud (staging)

**Agent adapter(s) involved**

paperclip_runner (native runtime). The problem is in the shared
continuation delivery pass, so it applies to all adapters.

## What Changed

- `deliverReconciledExecutions` in
`server/src/services/execution-recovery-resolution.ts`: a continuation
held for 5 minutes or more is read even when its agent has a queued
recovery run. Before, only the full-queue check let it through.
- Each page of the pass now reads at most one candidate per agent
(`row_number() over (partition by agent)`), so one agent's backlog
cannot fill the page budget and hide other agents' continuations.
- A continuation counts against the company's limit only when its wake
puts a run in the queue. A continuation that was invalidated or replaced
by a newer run does not use its agent's turn.
- When an agent refuses a wake (no run, or an error), the pass skips
that agent for the rest of the pass and stamps the row with
`continuationDeliveryAttemptedAt`. Overdue rows always rank before rows
that are not overdue. Among overdue rows, the turn is the later of the
hold time and the last attempt, so a refused row goes behind rows held
before that attempt and ahead of rows held after it. One refused row
cannot pin its agent, and new arrivals cannot keep an older refused row
waiting.
- New module `server/src/services/recovery-continuation-pacing.ts` holds
the pacing constants and the "held since" rule. It adds
`RECOVERY_CONTINUATION_DELIVERY_BOUND_MS` (15 minutes). The old
constants are still exported from `execution-recovery-resolution.ts`.
- `taskInterruptionPresentations` in
`server/src/services/task-interruption-presentation.ts` shows a pending
continuation as "continuing" only within the delivery bound. After that,
the task shows its real state instead of `working`.
- No change to the schedule: the continuation duty runs only while
pending continuations exist, and it checks the database on start, so a
restart does not lose pending rows.

## Verification

- `pnpm exec vitest run src/__tests__/recovery-wake-herd.test.ts` (in
`server/`): 15 tests pass. New tests:
- A continuation held past the bound is delivered while the agent's
queue holds recovery runs that never start.
- After a restart with a continuation pending past the bound and a full
queue, a new schedule is due within one pass, the pass delivers the
continuation, and the schedule then goes idle.
- Continuations whose wake returns no run do not use the per-company
count. Another continuation of the same agent is delivered on the next
pass.
- An overdue continuation is refused once while the agent's queue is
full. On the next pass, it is delivered even though new continuations
keep arriving for the same agent.
- One agent refuses a backlog of 210 overdue continuations. Another
agent's continuation is delivered in the same pass, and the refusing
agent gets one wake attempt in that pass.
- A task whose continuation is pending past the delivery bound with
nothing running does not show `working`. A fresh pending continuation
still shows `working` with `canRetry: false`.
- Red/green: with the two source files reverted to master (and only the
new constant exported), all 4 first new tests fail. The backlog test
fails on the first PR head (200 refused wake attempts in one pass). The
temporary-refusal test fails on the second PR head (the overdue row
stays pending). With this change, all 15 pass.
- Related suites pass: fresh-continuation fallback, workspace recovery
fallback, native safe replacement, native workspace reconstruction, task
interruption presentation, queued-run dispatch order, execution recovery
resolution, assigned-work invariant, execution control work, task
interruption automatic path and query.
- `node scripts/typecheck-layers.mjs` (in `server/`) passes.

## Risks

- A continuation that waited 5 minutes now enters a queue that already
has recovery runs. The per-agent (1) and per-company (5) limits per pass
still apply, and the queue still starts person-initiated work first.
- An agent that refuses wakes gets one wake attempt per pass. Its
refused rows rotate through `continuationDeliveryAttemptedAt`, and they
do not block other agents.
- After 15 minutes without delivery, the task no longer shows `working`.
With this fix, delivery happens well within that bound, so this is a
safety net.
- No schema or migration changes. No new polling.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, and read-only staging
API requests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:12:13 -06:00
..