Files
PaperClipAI/server
Devin FoleyandPaperclip a3f86e5cc8 fix(recovery): continue a recovered preparation that failed with a transient code (#22096)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote run starts its provider, Paperclip prepares the
sandbox: it restores the saved workspace and stages the agent files
> - When a legacy Runner preparation fails after the sandbox exists,
Paperclip keeps the sandbox, saves its files, and holds the task with
"preserve without replay"
> - A preparation that a restart or a lost process stopped continues by
itself after the files are saved. A preparation that failed with a
transient error does not: it waits for a person to click Retry, even
though its own bounded retry would continue it
> - The agent-file Git exclusion timeout also did not count as
transient. It failed with the generic `adapter_failed` code, so nothing
retried it, and the wake that started the run was lost
> - This pull request continues a recovered preparation that failed with
a transient preparation code, in the same bounded lane as a lost
preparation, and names the exclusion timeout as a transient preparation
timeout
> - The benefit is that a slow sandbox no longer stops a task until a
person notices it

## Linked Issues or Issue Description

Refs #22039, which retries the timed-out Git exclusion in place. This
change covers the case where all three attempts time out.

**What happened?**
A run failed in the preparing stage with "Could not exclude agent files
from task Git staging". No provider work started. The sandbox was kept,
and workspace recovery later saved its files with no changes. The
recovery hold recorded `replay: blocked`, so no new run started and the
wake was lost.

**Steps to reproduce**
1. Start a legacy Runner run on a sandbox that keeps its lease on
failure.
2. Make the agent-file Git exclusion time out on every attempt.
3. Let workspace recovery save the retained sandbox.

**Expected behavior**
After the files are saved, the run continues once in its bounded retry
lane, as a preparation stopped by a restart does.

**Actual behavior**
The run fails as `adapter_failed`. After recovery, the task waits for a
manual Retry.

## What Changed

- `failed-preparation-retry.ts` (new):
`isTransientWorkspacePreparationCode` and
`failedPreparationOwnsBoundedRetry` move here unchanged from
`heartbeat/retries.ts`, which re-exports them. The code list is now an
array, so SQL can use it. The move lets `cancelled-native-startup.ts`
use the predicate without an import cycle, because `retries.ts` already
imports that file.
- `cancelled-native-startup.ts`:
`isInterruptedNativePreparationCandidate` also accepts a failed
preparation that has a required workspace recovery and whose own bounded
retry owns it. All other conditions stay the same: legacy Runner, still
preparing, no provider evidence, released leases with stop receipts, and
a verified workspace repair before replay.
- `workspace-recovery.ts`: the recovery sweep lists the same runs, so
their continuation is delivered after the files are saved.
- `run-completion.ts`: a `RemoteWorkspacePreparationTimeoutError` that a
legacy run raises before it enters the adapter now records
`remote_workspace_prepare_timeout`, the same code the setup path already
records.
- `agent-instruction-files.ts`: after the last timed-out attempt, the
exclusion throws `RemoteWorkspacePreparationTimeoutError`. The message
stays "... (reason=timed_out)". `RemoteWorkspacePreparationTimeoutError`
takes an optional message for this.
- Tests: a recovered preparation that failed with
`remote_workspace_prepare_timeout` is listed by the sweep and continues
once; one that failed with `adapter_failed` still waits for Retry.
`failRun` names the timeout only when the adapter did not start. The
agent-file test checks the error type.

## Verification

- `npx vitest run` on the new and changed cases: the
retained-preparation continuation, the `failRun` classification, and the
agent-file suites. All pass.
- Each new case fails without its source change. With only the sweep
query reverted, the listing assertion fails.
- `npx vitest run` on 62 related server test files (workspace recovery
and its preparation suites, legacy execution recovery, run completion,
retry scheduling, recovery service, remote workspaces, task retry, fresh
continuation): 2361 pass, 16 skipped.
- `npx tsc --noEmit` passes for the server.

## Risks

- Low to medium. A failed preparation that kept its sandbox now
continues once its files are saved, if its failure code is transient.
The continuation uses the interrupted-preparation lane, which is bounded
to two attempts, and the existing failure budget still applies.
- A failure that is not transient (`adapter_failed`, `setup_failed`) is
unchanged: it still waits for Retry.
- A run whose failure is not a pre-provider remote preparation timeout
keeps `adapter_failed`.
- A task that is `done` or `cancelled` is unchanged. A scheduled retry
never runs on a closed task, so a wake on a closed task is still not
replayed after a failed preparation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 16:04:19 -08:00
..