Files
PaperClipAI/server
DottaandClaude Opus 5.5 19f8ed1ea2 fix(recovery): continue only recent interruptions from the held backlog, newest first, paced (#21222)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a server restarts or a deploy swaps servers, agent runs are
interrupted. The fresh-continuation fallback (#21135) continues each
interrupted task in a new run from its saved work.
> - The fallback also has a backlog pass. It continues tasks that older
releases left held by a recovery record.
> - The backlog pass had no age limit and no pacing. It took the oldest
records first, 25 per pass.
> - On the first hosted release with the fallback, it continued 176
tasks in 30 minutes. 74 of them had a source run that ended more than 7
days ago. The oldest ended four weeks ago. More than 100 runs were
queued, and one agent had about 130 runs.
> - The user-facing goal of the fallback is "an interruption is
invisible". It is not "run last week's failures again".
> - This pull request limits the backlog pass to recent interruptions,
newest first, at a paced rate.
> - The benefit is that a release no longer starts a herd of old,
unwanted runs, and recent interrupted work still continues by itself.

## Linked Issues or Issue Description

Refs #21135

**What happened?**
A hosted instance took its first release with the fresh-continuation
fallback. Within 30 minutes, the backlog pass continued 176 held tasks.
By source run age: 26 under 1 day, 76 from 1 to 7 days, 74 over 7 days.
The source end codes included `adapter_failed` (27),
`runner_remote_provider_artifact_incompatible` (14),
`native_provider_terminal_failed` (14) and `setup_failed` (8). These are
configuration or provider failures, not interruptions. The pass kept
firing at 12 to 36 continuations per 5 minutes, and the queued backlog
grew past 100 runs.

**Expected behavior**
The backlog pass continues a task only when the task was recently
interrupted by a restart, a deploy, a lost lease, a lost process or a
refused recovery. Old records and configuration or provider failures
stay as they are, for a person's Retry. The pass starts new runs at a
controlled rate, newest first.

**Steps to reproduce**
1. Have held recovery records that older releases saved, with source
runs that ended days or weeks ago, some with `adapter_failed`.
2. Deploy a release that contains #21135.
3. See that the disposition worker continues all of them, oldest first,
25 per pass, and queues a run for each.

## What Changed

- `continueHeldExecutions` and `heldExecutionsAwaitingFreshContinuation`
(server/src/services/fresh-continuation-fallback.ts):
- Continue a held record only if its source run ended within
`BACKLOG_CONTINUATION_MAX_AGE_MS` (24 hours) AND its error code is in
`BACKLOG_CONTINUATION_SOURCE_CODES` (restart, deploy, lease,
process-loss and recovery-refusal codes).
- Do not touch any other held record. Do not settle it and do not modify
it. A log line counts these records, at most once every 10 minutes.
  - Order by the source run's end time, newest first.
- Continue at most 5 records per pass and 10 per rolling minute per
server.
- The rule I chose, and why: 24 hours and interruption codes only. A
person who is waiting for a task notices within a day. A run that failed
for configuration or provider reasons fails again the same way, so a
person must change something and press Retry. The 7-day rule of the
assigned-work sweep is too wide for this pass: 76 of the 176 continued
tasks were from 1 to 7 days old.
- The live path does not change. Fresh interruptions come through
restart recovery and the disposition pass (`recovery_refused`), and
through continuation wakes. They are not paced and have no age rule.
- Upgrade safety: records that a release already continued carry
`freshContinuation` and are excluded as before. Queued continuation runs
are not changed.
- The 10-per-minute limit is per server. During a deploy overlap, two
servers can each continue 10 per minute. This is acceptable for a short
overlap, and it keeps this change free of shared state.

## Verification

- `cd server && npx vitest run
src/services/fresh-continuation-fallback.test.ts`: 53 passed.
- New test "continues only recent interruptions from the held backlog,
newest first, and leaves the rest untouched (staging herd)". It uses
mixed-age and mixed-cause records: 1 h and 6 h restart codes, 3-day and
28-day restart codes, and 2 h
`runner_remote_provider_artifact_incompatible` and `adapter_failed`.
Only the two recent restart-caused records continue, the 1 h record
first. The four other records keep the same status, evidence and
`updated_at` across repeated passes.
- New test "paces the held backlog to ten continuations a minute per
server": 12 eligible records give 10 continuations in the first minute
and the remaining 2 in the next minute.
- RED on origin/master: both new tests fail (`expected [] to have a
length of 1`; `expected 12 to be 10`).
- The existing tests for the live path (for example "continues tasks
that older releases left blocked by these codes", with a source run that
ended one minute ago) still pass.
- `pnpm --filter @paperclipai/server run typecheck`: clean.

## Risks

- A held record from 1 to 7 days ago, caused by an interruption, no
longer continues by itself. It stays as it was before #21135: the task
shows its recovery state and a person can press Retry.
- A record whose cause is not in the list (a future interruption code)
is not continued by the backlog pass. The live path still continues
fresh interruptions for any code that is not a person's.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 18:36:22 -06:00
..