Files
PaperClipAI/server
DottaandClaude Opus 5.5 4afcbfc0c7 fix(native): retry transient provider refusals while reattaching a handed-off run (#22118)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A release swap hands live native runs from the outgoing server to
the incoming server. The incoming server reattaches each run to its
original sandbox.
> - Reattachment first resumes the exact sandbox lease through the
sandbox provider plugin.
> - In a release-swap wave, one reattach resume got an HTTP 502 from the
provider gateway. The host did not retry it. The run failed with the
generic `setup_failed` code after 13 minutes of agent work.
> - The task then stayed blocked with a Retry banner for about 10
minutes, until the fresh-continuation fallback reached its bound and
started a new run.
> - This pull request retries a transient provider refusal during
reattachment, with a bounded budget. It also reads the provider status
from the plugin RPC error envelope, so the host sees the 502.
> - The benefit is that a brief gateway failure during a swap is
invisible to the user. The same run continues and no Retry banner shows.

## Linked Issues or Issue Description

**What happened?**
During a graceful release swap, the incoming server reattached a live
native run to its retained sandbox. The `environmentResumeLease` plugin
call failed with an HTML "502 Bad Gateway" body 280 ms after the
reattach started. The other 18 handed-off runs in the same swap
reattached in about 8 seconds. The host treated the 502 as an unknown
error, so the run failed with `setup_failed`. The task stayed `blocked`
with `canRetry: true` until the fresh-continuation fallback created a
new run 10 minutes later.

**Expected behavior**
A transient provider refusal (HTTP 408/425/429/5xx, a connection errno,
or a plugin RPC deadline) during reattachment is retried within a
bounded budget. The same run continues hands-off. A definite refusal
(for example HTTP 409) or a confirmed missing sandbox still fails at
once with its own named reason.

**Steps to reproduce**
1. Start a remote native run on a reusable sandbox.
2. Send SIGTERM to the server so it hands the run to the next server.
3. Make the first `environmentResumeLease` call on the incoming server
fail with HTTP 502.
4. See that the run fails with `setup_failed` and the task shows Retry.

**Paperclip version or commit**
`master` at ca34aca777 (release-swap wave) and ea154ce55e (this branch
base).

Root causes:
- Why setup failed: `environmentRunOrchestrator.acquireForRun` called
`resumeRunLease` once for a reattachment. Acquisition already retries
transient resume refusals, but reattachment did not. Also,
`transientSandboxResumeFailure` read `status` and `statusCode` only from
the thrown error. A plugin refusal arrives as `JsonRpcCallError`, which
carries the provider status only in the validated
`paperclip/environment-resume-error/v1` data envelope. So a plugin 502
was not transient even on the acquisition path.
- Why recovery took 10 minutes: `setup_failed` on a native run with
provider work is not a restart-class code, so the same-run reattach lane
does not take it. Failed-workspace preservation recorded a restore hold.
The fresh-continuation fallback waits for that hold up to
`FRESH_CONTINUATION_PENDING_BOUND_MS` (10 minutes) when it sees no
progress. The continuation started at 23:23:40, which is 10 minutes and
2 seconds after the run finished at 23:13:38.
- Why a Retry banner showed: no fresh-continuation check row existed yet
and no system recovery was active, so the presentation fell through to
`failed` with `canRetry: true`. The banner did not stay after the task
was done. The issue rests at `done` with `canRetry: false`. The grader
reported 10482 s because it closed the banner at its final read hours
later. Its last sample that saw the banner was 272 s after the first.

## What Changed

- `environment-run-orchestrator.ts`: the exact-lease resume during
reattachment goes through `retryTransientResumeCall`. The first call
keeps the provider's configured start deadline. A first call that used
up that deadline (plugin RPC TIMEOUT) is not repeated. After a transient
refusal, a 90 s budget (`REUSABLE_SANDBOX_RESUME_BUDGET_MS`) bounds each
later wait and passes the remaining budget as `timeoutMs`. A confirmed
missing sandbox and a definite refusal are not retried.
- `packages/plugins/sdk/src/environment-resume-error.ts`: owns one retry
policy for provider failures: `isRetryableEnvironmentHttpStatus` (408,
425, 429, 500, 502-504 and gateway/edge 520-527, 529, 598, 599; never
501 or 505) and `isRetryableEnvironmentErrorCode(code, scope)`.
ABORT_ERR is never retried. ENOTFOUND is retried only in the
`known_host` scope (an exact reattach to a host this run already
reached, the DNS-outage case). Lease acquisition keeps its canonical
semantics in the `first_acquisition` scope.
- `environment-provider-unavailable-error.ts`: lease acquisition uses
the SDK policy instead of its own sets.
- `reusable-sandbox-resume-replacement.ts`: owns the resume retry helper
and `transientSandboxResumeFailure(error, scope)`, which reads only
structured evidence (status, envelope errno, plugin RPC deadline,
sandbox state transition) and a few transport phrases. The acquisition
resume loop uses the same classifier and backoff helpers, and
`resumeFailureTimedOut` (label or envelope ETIMEDOUT) for its idle-start
extension.
- `environment-runtime.ts`: `EnvironmentDriverResumeInput.timeoutMs`
caps the sandbox plugin driver's resume RPC. Other drivers keep their
own deadline.
- Cloudflare bridge client: a bridge request deadline throws
`CloudflareBridgeTimeoutError` with code `ETIMEDOUT`. Daytona:
`SandboxLivenessTimeoutError` carries code `ETIMEDOUT`. Both reach the
host as a structured errno in the resume envelope.
- `tests/restart-chaos/staging-mix.ts`: `bannerShownMs` counts a visible
Retry banner at most 2 minutes past the last sample that saw it. Before,
it counted the whole unsampled gap until the final read.

Previous-release story: this changes no persisted state, launch
configuration, or pack identity. A run saved by the previous release
reattaches the same way. Only the in-process retry around the resume
call is new.

## Verification

- `pnpm verify` (selected Vitest files), and focused runs of the four
changed server test files and the two restart-chaos test files.
- Red/green: `server/src/__tests__/environment-run-orchestrator.test.ts`
- "retries a provider gateway refusal during reattachment and keeps the
original allocation" fails on master (1 call, rejects with the 502) and
passes with the fix.
- "treats a provider HTTP %i refusal during reattachment as transient"
(500, 502, 520, 599).
- "gives the first reattach resume the provider deadline and bounds
every retry by the budget".
- "does not repeat a reattach resume that used up the provider's start
deadline", "does not retry a definite provider refusal during
reattachment", and "preserves a retained sandbox startup failure without
acquiring a replacement" (default budget, one call).
- `server/src/services/reusable-sandbox-resume-retry.test.ts`: a table
over every classifier branch, and the helper's deadline rules.
- `server/src/__tests__/environment-runtime.test.ts`: "caps the exact
resume call at the caller's remaining budget".
- `server/src/services/environment-provider-unavailable-error.test.ts`:
scope table and the extended status lists. Orchestrator reattach cases:
ENOTFOUND retried, ABORT_ERR and 501 not.
-
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`:
"applies the idle-start extension to a provider liveness deadline named
only by its errno" (red/green).
- Cloudflare `bridge-client.test.ts` and Daytona `plugin.test.ts`: the
deadline errors carry ETIMEDOUT in the resume envelope.
- Red/green integration:
`server/src/__tests__/heartbeat-native-remote-failure-retention.test.ts`
"keeps reattaching through a gateway refusal during a graceful hand-off
and continues the same run". The run's controller is lost, restart
recovery claims it to reattach, and the first resume fails with the 502
envelope. On master the run fails before the runner step (1 resume
call). With the fix the resume is retried, the same run reaches the
runner step at controller generation 2, no replacement allocation is
acquired, and no second run is created.
- `tests/restart-chaos/staging-mix.test.ts`: banner tests for a late
hide sample and for a banner still open at report time.

## Risks

- The acquisition resume loop no longer retries a refusal whose only
evidence is "timeout" or "timed out" text. Cloudflare and Daytona now
name their own deadlines ETIMEDOUT, and the plugin RPC deadline is
structured, so those stay retried. A provider that reports a timeout
only as text fails at once.
- Lease acquisition now also retries gateway statuses 520-527, 529, 598
and 599.
- A reattach resume can now take the provider deadline plus up to 90 s
when it gets transient refusals. Each retry is the same exact-lease
lifecycle resume. It cannot acquire or replace a sandbox, so a retry
cannot create a second executor.
- The aggregate recovery brake is unchanged. This change adds no new run
starts and no work before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:11:04 -06:00
..