Files
PaperClipAI/server
DottaandClaude Opus 5.5 42858fe4e5 fix(recovery): continue a setup failure after work in seconds, with no Retry (#22128)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops before it finishes, the recovery system saves its
files and then continues the task in a fresh run from that save
> - A native run that already did work can still fail during setup, for
example while the incoming server reattaches it after a graceful release
swap
> - That failure (`setup_failed`) is not a restart-class code, so the
backlog pass did not continue it. The task waited about 10 minutes and
showed Retry, although the system owned the next step
> - Recovery must be automatic and fast: a needed-Retry rate of 0%, and
recovery latency in seconds
> - This pull request continues such a task as soon as its saved work is
recovered, and shows it as "continuing" with no Retry while it waits
> - The benefit is that a setup failure after work is invisible to the
user: the task continues in seconds, not 10 minutes

## Linked Issues or Issue Description

Refs #22118 (that PR prevents the gateway refusal that caused this
instance; this PR fixes the class).

**What happened?**
In a release-swap test wave, a native run had done about 13 minutes of
agent work. It was handed to the incoming server at a graceful swap.
Reattachment failed and the run ended with `setup_failed` at 23:13:38Z.
Saved-work recovery saved the run's files 7 seconds later. The next run
started at 23:23:40Z, 10 minutes and 2 seconds after the failure. For
that whole time the task view showed `failed` with `canRetry: true`.

**Expected behavior**
A failure during setup or reattachment of a run that already did work
continues the task automatically within seconds. While the system owns
the next step, the task shows a system-owned "continuing" state and no
Retry. Retry shows only when nothing automatic is pending, and then it
works.

**Steps to reproduce**
1. Start a native run and let it do provider work.
2. Make the run fail during setup (for example a definite refusal of the
reattach resume), so it ends `setup_failed` with a restore hold on its
files.
3. Let saved-work recovery save the files.
4. See that no continuation starts until an unrelated system wake
reaches the 10-minute fresh-continuation bound, and that the task shows
Retry in the meantime.

**Paperclip version or commit**
`master` at 4afcbfc0c7.

Root causes:
- Latency: the backlog pass (`continueHeldExecutions`) selects held
records only for "recent interruptions" (`recentInterruptionSourceSql`):
restart-class codes, recovery runs, or setup failures before provider
work. A first run that failed setup after provider work matched none of
them. So nothing called the fallback after the restore hold settled. The
task continued only when a later system wake reached the fallback after
`FRESH_CONTINUATION_PENDING_BOUND_MS` (10 minutes).
- Latency: even for an eligible record, a check made while saved-work
recovery was running was deferred to its backed-off `nextCheckAt` (1 to
4 minutes). Recovery completion did not wake the disposition worker.
- Presentation: no fresh-continuation check was recorded and no
saved-work recovery was active, so the projection fell through to
`failed` with `canRetry: true`.

## What Changed

- `continuation-cause.ts`: export `SETUP_FAILURE_CODES` (`setup_failed`
and the transient setup codes).
- New `fresh-continuation-eligibility.ts`: the held-record and
eligibility predicates move here from `fresh-continuation-fallback.ts`
with no behavior change (`heldExecutionWhere`,
`interruptionEligibility`, `recentInterruptionSourceSql`, the backlog
age constants, and the Blocked-ownership check the fallback uses). The
fallback re-exports the public names. The fallback file goes from 992 to
about 900 lines.
- Eligibility: the backlog pass now continues a system-owned setup
failure of a run that did provider work
(`executionRecovery.providerWorkStarted` is not `false`) whose files are
held for saved-work recovery or were recovered by it. A failure before
provider work, a person-owned code, and an exhausted saved-work recovery
keep their Retry.
- `expediteFreshContinuationChecks` (fallback module): makes the run's
pending `exact_recovery_pending` checks due now and wakes the
execution-disposition worker. `firstPendingAt` is kept, so the wait
bounds do not reset. `workspace-recovery.ts` calls it in the transaction
that completes saved-work recovery.
- `continueHeldExecutions` records its check in the same transaction as
the decision. A saved-work completion on the same hold then waits for
that commit, so its expedite cannot be overwritten by a stale future
check.
- `runsAwaitingSystemContinuation` (eligibility module): the held
records the backlog pass will continue by itself: same predicates,
system eligibility only, no person's wait recorded, no pending question
or approval, and a Blocked status that the failure owns (the same check
the fallback makes).
- `task-interruption-presentation.ts`: such a run shows
`freshContinuation: continuing` (system-owned, `working`, no Retry). A
pending Retry of that run keeps its own wait message. A task at the
hourly cap keeps `retry_scheduled`.
- Tests:
- New `setup-failure-after-work-continuation.test.ts` (embedded
Postgres, real saved-work recovery): the full sequence, plus no-work and
person-owned cases.
- `fresh-continuation-fallback.test.ts`: eligibility, expedite, the
completion-during-decision ordering, the cap presentation, and the
exhausted case.
- `fresh-continuation-schedule-queries.test.ts`: the oracle predicate
gets the same clause, and the fixture includes `setup_failed`.

## Verification

- Red/green:
`server/src/services/setup-failure-after-work-continuation.test.ts`,
"continues in a fresh run as soon as its saved work is recovered, and
never offers Retry". The sequence: a run with work fails setup, a
restore hold is recorded, the disposition pass runs, real saved-work
recovery completes, the disposition pass runs again, continuations are
delivered. The task view is checked at each step. On master the first
check fails (`state: "failed"`). With the fix every step shows
`responsibility: system, state: working, canRetry: false`. The successor
is created less than 5 s after the decision, with `previousRunId` and
`forceFreshSession`.
- `setup-failure-after-work-continuation.test.ts` "leaves a setup
failure before any provider work on its Retry path" guards the
provider-work gate.
- Red/green: `fresh-continuation-fallback.test.ts` "continues a setup
failure of a run that already did work once its saved work is recovered"
(master: not selected, 0 continued).
- `fresh-continuation-fallback.test.ts` "keeps a setup failure whose
saved-work recovery is still under way ...": the check waits, then the
expedite makes it due at once and the next pass continues.
- `pnpm verify` (selected files) and focused runs:
`fresh-continuation-*`, `task-interruption-*`, `workspace-recovery-*`,
`runner-lost-mid-turn-continuation`,
`heartbeat-native-remote-failure-retention`, `assigned-work-invariant`.

Recovery latency for this class, from saved-work completion to the new
run:
- Before: 10 min 2 s (the 10-minute pending bound).
- After: the next disposition pass, which the completion wakes. The
worker is throttled to one pass per 15 s, so the bound is 15 s. The test
asserts under 5 s. End to end from the failure, add the saved-work
recovery time (7 s in the observed case).

## Risks

- More tasks continue automatically: every system-owned setup failure of
a run that did provider work and whose files were saved, also when the
provider turn was in flight. The fresh run never replays the old turn.
Its opening context names the cut-off run and asks the agent to check
unverified actions before it repeats them. This is the same contract as
restart-class interruptions.
- The per-task cap (6 continuations per rolling hour), the per-server
backlog pacing (10 per minute) and the aggregate automatic-run brake are
unchanged and apply to these continuations. Judge the change by
company-wide automatic run starts per hour and their success rate.
- The task view predicts the fallback's decision from the same
predicates and Blocked-ownership check. A rarer refusal (for example a
dependency wait) can show "continuing" until the next disposition pass
records it, then Retry shows.
- The task view runs one more bounded query when listed tasks have an
unfinished latest run.
- No schema change. Records saved by the previous release are read the
same way. No work is added before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 23:49:21 -06:00
..