Files
PaperClipAI/server
DottaandClaude Opus 5.5 0ec155ac31 fix(recovery): enqueue a fresh interruption's continuation at once and record why a folded wake did not start (#21354)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops before it finishes, for example when a server swap
reaches a sandbox late, the fresh-continuation fallback records one
continuation of the task from its saved work
> - The execution-continuation worker turns that record into a queued
run. It took one record per agent per pass, and it took the oldest held
record first
> - After a large swap, the backlog of old records is hours deep. A
record for a run that ended a few minutes ago waited behind the whole
backlog. The record also stayed outside the queue while the agent's
queue was deeper than its concurrency limit
> - Meanwhile the task showed "todo" with nothing running, and the
assigned-work sweep's wakes for it were folded into the pending record
without any trace
> - This pull request queues a fresh interruption's continuation on the
next pass, keeps a share of every pass for the old backlog, and records
a skipped wake row when a system wake is folded into a pending
continuation
> - The benefit is that an interrupted task gets its next run promptly,
it waits in the agent's queue where the dispatcher ranks it, and nothing
is lost or invisible

## Linked Issues or Issue Description

Refs #21324, #21257, #21234 (earlier pacing and dispatch ranking that
this builds on).

**What happened?**
On a hosted staging instance, five tasks had runs that ended after a
server swap ("The runner restored the conversation after process loss,
but the previous turn is no longer active."). The fallback recorded a
continuation for each task 3 to 6 minutes after the run ended. Each
record stayed `continuationDelivery: pending` for 30 to 110 minutes with
no delivery attempt. During that time the worker delivered continuations
whose source runs ended 4 to 7 hours earlier, one per agent per pass.
The tasks read as failed and needing Retry. The sweep tried 3 to 4 times
per task, and each wake returned `null` without a wake row.

**Expected behavior**
A recorded continuation for a recent interruption becomes a queued run
within about one delivery pass. Old backlog continues to drain and is
not starved. A wake that is not started records why.

**Steps to reproduce**
1. Give an agent a queue that is deeper than its concurrency limit.
2. Record pending continuations for older interruptions that are held
past the hold bound.
3. Record a continuation for a run that ended a few minutes ago.
4. Run `deliverReconciledExecutions`. Before this change the fresh
record is not delivered. The oldest held record is delivered first, one
per agent per pass.

**Agent adapter(s) involved**
Not adapter-specific (core bug).

## What Changed

- `deliverReconciledExecutions` now sorts pending continuations into two
classes by when the source run ended. "Fresh" means that the source run
ended within `RECOVERY_BACKLOG_AGE_MS` (30 minutes), which is the same
value the dispatcher uses to tell fresh recovery from backlog.
- A fresh continuation is enqueued without the queue-depth room check.
It is limited only by the per-agent limit (one fresh continuation per
agent per pass) and the per-company cap per pass. Among fresh records,
the newest interruption goes first. A record whose wake was refused
moves behind records that were not attempted yet.
- Backlog continuations keep their earlier rules. These are the
queue-depth room check, the bypass after the 5-minute hold bound, oldest
held first, and one per agent per pass.
- Each pass runs three phases: fresh up to the company cap minus
`RECOVERY_CONTINUATION_BACKLOG_SHARE_PER_PASS` (1), then backlog up to
the cap, then fresh again up to the cap. When both classes wait, the
backlog gets at least 1 of the 5 slots per company per pass. The
per-company cap (5) does not change, so a swap that interrupts many runs
does not enqueue them all at once. All admissions still go through the
one `wake(...)` call.
- In `enqueueWakeup`, a system wake that is folded into the task's
pending fresh continuation now writes a `skipped` wake row with reason
`issue_continuation_pending`. Before this change it returned `null` with
no trace.
- Tests in `recovery-wake-herd.test.ts`:
- New: a fresh continuation is queued on the first pass while the
agent's queue is deeper than its limit.
- New: a fresh continuation is delivered ahead of an hours-old backlog
for the same agent, and the backlog keeps draining.
- New: many fresh continuations for one agent are enqueued one per pass,
newest first, and none is left behind.
- New: the backlog keeps a share of each pass while fresh continuations
fill the company cap.
- New: a refused fresh wake stays pending and is enqueued on the next
pass.
  - New: a folded system wake leaves a skipped row.
- Updated: existing tests that used a recent source run to mean "held
because the queue is full" now use backlog-age sources, and assertions
that depended on exactly one admission per agent per pass now check the
record under test.

## Verification

- Red before the fix: the 6 new tests failed. The 18 earlier tests still
passed.
- Green after the fix: `cd server && npx vitest run
src/__tests__/recovery-wake-herd.test.ts` gives 24 passed.
- `npx vitest run src/__tests__/execution-control-work.test.ts` gives 27
passed. `src/__tests__/issue-recovery-actions.test.ts` gives 112 passed.
`src/services/fresh-continuation-fallback.test.ts` gives 62 passed.
`src/services/workspace-recovery-fallback.test.ts` gives 19 passed.
`src/services/native-runtime/native-safe-replacement.test.ts` gives 62
passed.
- `cd server && npx tsc --noEmit -p tsconfig.json` passes.

## Risks

- Behavior shift: an agent with a full queue now gets fresh recovery
continuations queued at once. They wait in its queue, where the
dispatcher ranking (person-initiated first, recovery share) decides when
they start. They are no longer held outside the queue. The per-company
cap per pass still limits the burst.
- Old backlog drains at a similar pace as before. It always gets at
least one slot per company per pass when it waits.
- The skipped wake row adds one small row each time the sweep retries a
task that has a pending continuation. The sweep spaces its retries, so
the volume is low.
- No migration. No API or UI contract change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, read-only staging API calls).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 05:46:12 -06:00
..