Files
PaperClipAI/doc
DottaandClaude Opus 5.5 4dc5492c2d fix(recovery): retry a failed Retry run's sandbox resume failure when the sweep reaches it first (#21124)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable cloud sandbox resumes the same
sandbox on its next run
> - When the resume fails before provider work
(`reusable_sandbox_resume_failed`), the run gets a bounded automatic
retry
> - On a hosted staging server, 8 of 14 such failures in 10 hours did
not get that retry. Each task ended `blocked` and needed a person to
press Retry
> - In all 8, the stranded-task sweep blocked the task between the
failed write and the run's own retry scheduling. The block cleared the
claim that the retry needed
> - This pull request makes the sweep give such a run to the same
bounded retry scheduler, not escalate it
> - The benefit is that a transient sandbox resume failure no longer
costs the task, also when settlement is slow or the server stops during
an upgrade

## Linked Issues or Issue Description

Refs #21108, #21113, #21078

**What happened?**
A person pressed Retry on a task. The run tried to resume the task's
saved Daytona sandbox before provider work. The resume failed
(`cause=transport_timeout`, or `provider_unavailable status=503`), and
the run was written `failed` with `reusable_sandbox_resume_failed`. Then
the stranded-task sweep moved the task to `blocked` and cleared its
claim. This happened 1 to 43 seconds after the failed write. The run's
executor scheduled its bounded retry 2 to 81 seconds after the failed
write, and the gate refused it ("The automatic retry could not
revalidate the original user continuation."). Read-only audit of 14
failures (all legacy runtime, `executionRecovery: bootstrap,
providerWorkStarted: false`): 6 got their retry, and the 5 most recent
ones recovered. 8 were refused this way and wait for a person. In one
example the run was written failed at 14:23:31, the task was blocked at
14:24:14, and the retry was refused at 14:24:52. The server was being
replaced at that time.

**Expected behavior**
A transient preparation failure before provider work continues with its
bounded retry. The sweep must not take the claim away from that retry.
When the retry budget is spent, or the continuation is refused, the task
escalates once, as before.

**Steps to reproduce**
1. Press Retry on a task whose reusable sandbox resume fails before
provider work.
2. Let the run's executor be slow to settle the failure (more than the
remaining controller lease, at most 60 seconds), or stop its server
after the failed write.
3. Let the stranded-task sweep run on another server. The task becomes
`blocked`, and no retry is scheduled.

**Root cause**
`classifyContinuationFailure` lists `reusable_sandbox_resume_failed` and
the other transient preparation codes in
`NON_RETRYABLE_CONTINUATION_ERROR_CODES`. The sweep assumes that the
executor already scheduled the dedicated bounded retry, so it escalates
at once and never asks the scheduler. #21108 added a skip while the
executor still settles the run. That skip uses a live local execution or
an unexpired controller lease. The lease stops renewing when the run is
written terminal (renewal requires `status = 'running'`). So the skip
covers at most about 60 seconds after the last renewal, and it covers
nothing when the executor's server stops during settlement.

**Paperclip version or commit**
Staging at `a65644d1e1`. Current `master` has the same code path.

**Deployment mode**
Hosted (cloud), two servers alive during each upgrade.

**Agent adapter(s) involved**
`codex_local` on a Daytona sandbox environment. The change is
adapter-independent.

## What Changed

- `server/src/services/heartbeat/retries.ts`: add
`failedPreparationOwnsBoundedRetry(run)`. It is the persisted form of
the run-completion decision: a failed legacy run with no native session,
bootstrap evidence that provider work did not start, and a transient
preparation code. A restore failure counts only when its owner marked it
retryable. Codes that an accepted interaction retries itself are
excluded.
- `server/src/services/recovery/service.ts`: when the latest run is such
a run, the sweep does not escalate it as non-retryable. It goes to
`scheduleRecoveryRetry`, which is the run's own bounded scheduler. The
scheduler reuses a successor that already exists, so the executor and
the sweep create one successor. The task escalates when the scheduler
reports exhaustion or a refused continuation. When the scheduler
declines the retry for another reason (for example a permanent refusal),
the task escalates once as a non-retryable failure, as before. The
#21108 settling skip still runs first.
- `server/src/services/heartbeat.ts`: pass the predicate to the recovery
service.
- `doc/execution-semantics.md`: document the rule.

## Verification

- New tests in `server/src/services/task-retry.test.ts`, in the staging
shape: a Retry-admitted run failed with
`reusable_sandbox_resume_failed`, bootstrap evidence, and an expired
controller lease from another server.
- "retries a failed Retry run's transient preparation failure when the
sweep reaches it first" (`slow_executor`, `executor_gone`). Red before
the fix: the task became `blocked`, as on staging. Green after the fix:
the task stays `in_progress` with one `scheduled_retry` successor
(`transient_failure`, attempt 1), and dispatch accepts it. For
`slow_executor`, the executor's own scheduling then reuses the same
successor, and there is still one successor.
- "still escalates a failed Retry run's transient preparation failure
once its bounded retries are spent": the task becomes `blocked` with no
successor.
- "escalates a failed Retry run's transient preparation failure once
when the scheduler declines it permanently" (review finding): red before
the follow-up commit (the task stayed `in_progress` with no run), green
after (the task is `blocked` once, and a second sweep adds no notice).
- `pnpm exec vitest run src/services/task-retry.test.ts`: 114/114 pass.
- `pnpm exec vitest run src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/heartbeat-process-recovery.test.ts` (with task-retry): 631
of 633 passed. The 2 failures were 15-second timeouts under parallel
load in native Board-wait tests that this change does not touch. They
passed when run alone.
- `tsc -p server/tsconfig.json --noEmit` passes.
- Not run: a live staging reproduction. The staging evidence is
read-only (run events, task activity, run records).

## Risks

- The sweep can now schedule the bounded retry before the executor does.
The scheduler locks the task and the run and reuses an existing
successor, so there is never a second successor. All continuation gate
checks still apply.
- The sweep acts only after the #21108 settling skip lapses, so a
healthy run is not affected. No wait is added before a normal turn
start.
- No schema change. Rows from older releases follow the same rule
because the predicate reads only persisted fields.
- Out of scope: the provider-side causes (Daytona 503 at 14:21Z, resume
calls that time out). #21113 handles the in-run wait for idle transport
timeouts.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, read-only API reads, Vitest on embedded PostgreSQL) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:20:55 -06:00
..
…
…
…
…