mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run that uses a saved (reusable) sandbox resumes that sandbox during setup, before the provider work starts. > - When the sandbox provider has an outage, the resume gets an HTTP 5xx answer. Setup fails with `reusable_sandbox_resume_failed`. Paperclip keeps the lease and schedules the bounded bootstrap retry, which resumes the same lease when the provider recovers (#20626). > - The setup path writes the failed status before it decides the retry. That write sends the Sentry run-failure report. Thus each attempt during an outage creates an "unexpected failure" event, also when the retry recovers the run. > - The reviewed chat setup wait had the same defect. #20205 fixed it with a `pending` report handshake. > - This pull request uses the same handshake for a sandbox resume that the provider answered with an HTTP 5xx. > - The benefit: Sentry shows a sandbox resume failure only when the retry cannot recover it. ## Linked Issues or Issue Description **What happened?** Staging Sentry recorded `EnvironmentRunError: Failed to acquire lease for environment "… Sandbox" (sandbox): Reusable sandbox lease "…" could not be resumed (cause=provider_unavailable status=502); the lease was preserved and no replacement was created.` The events came in bursts: 5 different tasks in one minute (HTTP 503/504), and 2 tasks in three minutes on a different day (HTTP 502). All events had `failure_phase=setup` and no provider work. In the source run, the scheduled bootstrap retry got past the resume step. **Expected behavior** A provider outage that the bounded retry owns does not create a Sentry event. A retry that is exhausted, not scheduled, or that fails to schedule still creates one event. A refusal of the sandbox (an HTTP 4xx, or no status) still reports as before. **Steps to reproduce** 1. Use an agent with a reusable sandbox environment that has a saved lease. 2. Make the sandbox provider answer the resume with HTTP 502. 3. Start a run. Setup fails with `reusable_sandbox_resume_failed` and schedules the bootstrap retry. 4. See a Sentry run-failure event for each attempt. **Paperclip version or commit** `2756be2f32` (staging releases `d0057ee384`, `1d153f62b6`, `e23549cd9c` also show the events). I searched for duplicate PRs. #20626 added the retry and safe replacement for a sandbox that will not resume. It does not change when the failure is reported. #20205 is the precedent for the reviewed chat wait. Fixes PAPERCLIP-BACKEND-77 ## What Changed - `server/src/services/reusable-sandbox-resume-error.ts`: `ReusableSandboxResumeError.providerOutage` is true only when the closed HTTP status of the provider answer is 5xx. The text of a gateway page does not decide. - `server/src/services/heartbeat/run-completion.ts`: For a provider outage, the failed-status write sets `sandboxResumeRetry: "pending"`. After the retry decision, the setup path reports the run once with `scheduled` or `not_scheduled`. The deferred-report code is shared with the reviewed chat wait. - `server/src/services/run-failure-report.ts` and `run-failure-diagnostics.ts`: A `pending` sandbox resume write does not report. In `unexpected` mode, a `scheduled` retry of a failed `reusable_sandbox_resume_failed` run does not report. `all` mode still reports it. - Tests: `run-failure-report-sandbox-resume.test.ts` (new) covers the report filter. `run-completion-reviewed-chat-report.test.ts` now also covers the setup path for 502 and 504 with a scheduled retry, an exhausted retry, no retry, a scheduler failure, a 4xx refusal and a refusal without a status. ## Verification - `vitest run src/services/__tests__/run-failure-report-sandbox-resume.test.ts src/services/heartbeat/run-completion-reviewed-chat-report.test.ts src/services/__tests__/run-failure-report.test.ts`: 308 tests pass (embedded Postgres). - Red check: before the fix, the 6 new outage cases fail, because the failed-status write reports immediately. - Earlier, on the base commit, the related run-completion, retry scheduling, recovery and run-failure-report suites passed (407 tests). - `tsc --noEmit -p server/tsconfig.json`: no errors in server sources. - The full workspace install did not fit on the agent sandbox disk, so CI runs the complete affected-test selection. ## Risks - Low risk. Run status, error code, the retry and lease handling do not change. Only the time of the Sentry report changes. - If the process stops between the failed-status write and the retry decision, no report is sent for that attempt. The reviewed chat wait has the same accepted behavior. A later terminal failure of the retry still reports. - 4xx and unclassified resume failures keep the report on the failed-status write. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code agent harness with tool use (shell, file edits, Sentry and GitHub connectors). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>