mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 14:35:55 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A release swap hands live native runs from the outgoing server to the incoming server. The incoming server reattaches each run to its original sandbox. > - Reattachment first resumes the exact sandbox lease through the sandbox provider plugin. > - In a release-swap wave, one reattach resume got an HTTP 502 from the provider gateway. The host did not retry it. The run failed with the generic `setup_failed` code after 13 minutes of agent work. > - The task then stayed blocked with a Retry banner for about 10 minutes, until the fresh-continuation fallback reached its bound and started a new run. > - This pull request retries a transient provider refusal during reattachment, with a bounded budget. It also reads the provider status from the plugin RPC error envelope, so the host sees the 502. > - The benefit is that a brief gateway failure during a swap is invisible to the user. The same run continues and no Retry banner shows. ## Linked Issues or Issue Description **What happened?** During a graceful release swap, the incoming server reattached a live native run to its retained sandbox. The `environmentResumeLease` plugin call failed with an HTML "502 Bad Gateway" body 280 ms after the reattach started. The other 18 handed-off runs in the same swap reattached in about 8 seconds. The host treated the 502 as an unknown error, so the run failed with `setup_failed`. The task stayed `blocked` with `canRetry: true` until the fresh-continuation fallback created a new run 10 minutes later. **Expected behavior** A transient provider refusal (HTTP 408/425/429/5xx, a connection errno, or a plugin RPC deadline) during reattachment is retried within a bounded budget. The same run continues hands-off. A definite refusal (for example HTTP 409) or a confirmed missing sandbox still fails at once with its own named reason. **Steps to reproduce** 1. Start a remote native run on a reusable sandbox. 2. Send SIGTERM to the server so it hands the run to the next server. 3. Make the first `environmentResumeLease` call on the incoming server fail with HTTP 502. 4. See that the run fails with `setup_failed` and the task shows Retry. **Paperclip version or commit** `master` atca34aca777(release-swap wave) andea154ce55e(this branch base). Root causes: - Why setup failed: `environmentRunOrchestrator.acquireForRun` called `resumeRunLease` once for a reattachment. Acquisition already retries transient resume refusals, but reattachment did not. Also, `transientSandboxResumeFailure` read `status` and `statusCode` only from the thrown error. A plugin refusal arrives as `JsonRpcCallError`, which carries the provider status only in the validated `paperclip/environment-resume-error/v1` data envelope. So a plugin 502 was not transient even on the acquisition path. - Why recovery took 10 minutes: `setup_failed` on a native run with provider work is not a restart-class code, so the same-run reattach lane does not take it. Failed-workspace preservation recorded a restore hold. The fresh-continuation fallback waits for that hold up to `FRESH_CONTINUATION_PENDING_BOUND_MS` (10 minutes) when it sees no progress. The continuation started at 23:23:40, which is 10 minutes and 2 seconds after the run finished at 23:13:38. - Why a Retry banner showed: no fresh-continuation check row existed yet and no system recovery was active, so the presentation fell through to `failed` with `canRetry: true`. The banner did not stay after the task was done. The issue rests at `done` with `canRetry: false`. The grader reported 10482 s because it closed the banner at its final read hours later. Its last sample that saw the banner was 272 s after the first. ## What Changed - `environment-run-orchestrator.ts`: the exact-lease resume during reattachment goes through `retryTransientResumeCall`. The first call keeps the provider's configured start deadline. A first call that used up that deadline (plugin RPC TIMEOUT) is not repeated. After a transient refusal, a 90 s budget (`REUSABLE_SANDBOX_RESUME_BUDGET_MS`) bounds each later wait and passes the remaining budget as `timeoutMs`. A confirmed missing sandbox and a definite refusal are not retried. - `packages/plugins/sdk/src/environment-resume-error.ts`: owns one retry policy for provider failures: `isRetryableEnvironmentHttpStatus` (408, 425, 429, 500, 502-504 and gateway/edge 520-527, 529, 598, 599; never 501 or 505) and `isRetryableEnvironmentErrorCode(code, scope)`. ABORT_ERR is never retried. ENOTFOUND is retried only in the `known_host` scope (an exact reattach to a host this run already reached, the DNS-outage case). Lease acquisition keeps its canonical semantics in the `first_acquisition` scope. - `environment-provider-unavailable-error.ts`: lease acquisition uses the SDK policy instead of its own sets. - `reusable-sandbox-resume-replacement.ts`: owns the resume retry helper and `transientSandboxResumeFailure(error, scope)`, which reads only structured evidence (status, envelope errno, plugin RPC deadline, sandbox state transition) and a few transport phrases. The acquisition resume loop uses the same classifier and backoff helpers, and `resumeFailureTimedOut` (label or envelope ETIMEDOUT) for its idle-start extension. - `environment-runtime.ts`: `EnvironmentDriverResumeInput.timeoutMs` caps the sandbox plugin driver's resume RPC. Other drivers keep their own deadline. - Cloudflare bridge client: a bridge request deadline throws `CloudflareBridgeTimeoutError` with code `ETIMEDOUT`. Daytona: `SandboxLivenessTimeoutError` carries code `ETIMEDOUT`. Both reach the host as a structured errno in the resume envelope. - `tests/restart-chaos/staging-mix.ts`: `bannerShownMs` counts a visible Retry banner at most 2 minutes past the last sample that saw it. Before, it counted the whole unsampled gap until the final read. Previous-release story: this changes no persisted state, launch configuration, or pack identity. A run saved by the previous release reattaches the same way. Only the in-process retry around the resume call is new. ## Verification - `pnpm verify` (selected Vitest files), and focused runs of the four changed server test files and the two restart-chaos test files. - Red/green: `server/src/__tests__/environment-run-orchestrator.test.ts` - "retries a provider gateway refusal during reattachment and keeps the original allocation" fails on master (1 call, rejects with the 502) and passes with the fix. - "treats a provider HTTP %i refusal during reattachment as transient" (500, 502, 520, 599). - "gives the first reattach resume the provider deadline and bounds every retry by the budget". - "does not repeat a reattach resume that used up the provider's start deadline", "does not retry a definite provider refusal during reattachment", and "preserves a retained sandbox startup failure without acquiring a replacement" (default budget, one call). - `server/src/services/reusable-sandbox-resume-retry.test.ts`: a table over every classifier branch, and the helper's deadline rules. - `server/src/__tests__/environment-runtime.test.ts`: "caps the exact resume call at the caller's remaining budget". - `server/src/services/environment-provider-unavailable-error.test.ts`: scope table and the extended status lists. Orchestrator reattach cases: ENOTFOUND retried, ABORT_ERR and 501 not. - `server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`: "applies the idle-start extension to a provider liveness deadline named only by its errno" (red/green). - Cloudflare `bridge-client.test.ts` and Daytona `plugin.test.ts`: the deadline errors carry ETIMEDOUT in the resume envelope. - Red/green integration: `server/src/__tests__/heartbeat-native-remote-failure-retention.test.ts` "keeps reattaching through a gateway refusal during a graceful hand-off and continues the same run". The run's controller is lost, restart recovery claims it to reattach, and the first resume fails with the 502 envelope. On master the run fails before the runner step (1 resume call). With the fix the resume is retried, the same run reaches the runner step at controller generation 2, no replacement allocation is acquired, and no second run is created. - `tests/restart-chaos/staging-mix.test.ts`: banner tests for a late hide sample and for a banner still open at report time. ## Risks - The acquisition resume loop no longer retries a refusal whose only evidence is "timeout" or "timed out" text. Cloudflare and Daytona now name their own deadlines ETIMEDOUT, and the plugin RPC deadline is structured, so those stay retried. A provider that reports a timeout only as text fails at once. - Lease acquisition now also retries gateway statuses 520-527, 529, 598 and 599. - A reattach resume can now take the provider deadline plus up to 90 s when it gets transient refusals. Each retry is the same exact-lease lifecycle resume. It cannot acquire or replace a sandbox, so a retry cannot create a second executor. - The aggregate recovery brake is unchanged. This change adds no new run starts and no work before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with tool use (shell, file edits, test runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>