Files
PaperClipAI/server/src/services/__tests__
1a50bd550e fix(sentry): report a saved-sandbox resume provider outage only after its retry decision (#21510)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A run that uses a saved (reusable) sandbox resumes that sandbox
during setup, before the provider work starts.
> - When the sandbox provider has an outage, the resume gets an HTTP 5xx
answer. Setup fails with `reusable_sandbox_resume_failed`. Paperclip
keeps the lease and schedules the bounded bootstrap retry, which resumes
the same lease when the provider recovers (#20626).
> - The setup path writes the failed status before it decides the retry.
That write sends the Sentry run-failure report. Thus each attempt during
an outage creates an "unexpected failure" event, also when the retry
recovers the run.
> - The reviewed chat setup wait had the same defect. #20205 fixed it
with a `pending` report handshake.
> - This pull request uses the same handshake for a sandbox resume that
the provider answered with an HTTP 5xx.
> - The benefit: Sentry shows a sandbox resume failure only when the
retry cannot recover it.

## Linked Issues or Issue Description

**What happened?**

Staging Sentry recorded `EnvironmentRunError: Failed to acquire lease
for environment "… Sandbox" (sandbox): Reusable sandbox lease "…" could
not be resumed (cause=provider_unavailable status=502); the lease was
preserved and no replacement was created.` The events came in bursts: 5
different tasks in one minute (HTTP 503/504), and 2 tasks in three
minutes on a different day (HTTP 502). All events had
`failure_phase=setup` and no provider work. In the source run, the
scheduled bootstrap retry got past the resume step.

**Expected behavior**

A provider outage that the bounded retry owns does not create a Sentry
event. A retry that is exhausted, not scheduled, or that fails to
schedule still creates one event. A refusal of the sandbox (an HTTP 4xx,
or no status) still reports as before.

**Steps to reproduce**

1. Use an agent with a reusable sandbox environment that has a saved
lease.
2. Make the sandbox provider answer the resume with HTTP 502.
3. Start a run. Setup fails with `reusable_sandbox_resume_failed` and
schedules the bootstrap retry.
4. See a Sentry run-failure event for each attempt.

**Paperclip version or commit**

`2756be2f32` (staging releases `d0057ee384`, `1d153f62b6`, `e23549cd9c`
also show the events).

I searched for duplicate PRs. #20626 added the retry and safe
replacement for a sandbox that will not resume. It does not change when
the failure is reported. #20205 is the precedent for the reviewed chat
wait.

Fixes PAPERCLIP-BACKEND-77

## What Changed

- `server/src/services/reusable-sandbox-resume-error.ts`:
`ReusableSandboxResumeError.providerOutage` is true only when the closed
HTTP status of the provider answer is 5xx. The text of a gateway page
does not decide.
- `server/src/services/heartbeat/run-completion.ts`: For a provider
outage, the failed-status write sets `sandboxResumeRetry: "pending"`.
After the retry decision, the setup path reports the run once with
`scheduled` or `not_scheduled`. The deferred-report code is shared with
the reviewed chat wait.
- `server/src/services/run-failure-report.ts` and
`run-failure-diagnostics.ts`: A `pending` sandbox resume write does not
report. In `unexpected` mode, a `scheduled` retry of a failed
`reusable_sandbox_resume_failed` run does not report. `all` mode still
reports it.
- Tests: `run-failure-report-sandbox-resume.test.ts` (new) covers the
report filter. `run-completion-reviewed-chat-report.test.ts` now also
covers the setup path for 502 and 504 with a scheduled retry, an
exhausted retry, no retry, a scheduler failure, a 4xx refusal and a
refusal without a status.

## Verification

- `vitest run
src/services/__tests__/run-failure-report-sandbox-resume.test.ts
src/services/heartbeat/run-completion-reviewed-chat-report.test.ts
src/services/__tests__/run-failure-report.test.ts`: 308 tests pass
(embedded Postgres).
- Red check: before the fix, the 6 new outage cases fail, because the
failed-status write reports immediately.
- Earlier, on the base commit, the related run-completion, retry
scheduling, recovery and run-failure-report suites passed (407 tests).
- `tsc --noEmit -p server/tsconfig.json`: no errors in server sources.
- The full workspace install did not fit on the agent sandbox disk, so
CI runs the complete affected-test selection.

## Risks

- Low risk. Run status, error code, the retry and lease handling do not
change. Only the time of the Sentry report changes.
- If the process stops between the failed-status write and the retry
decision, no report is sent for that attempt. The reviewed chat wait has
the same accepted behavior. A later terminal failure of the retry still
reports.
- 4xx and unclassified resume failures keep the report on the
failed-status write.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code
agent harness with tool use (shell, file edits, Sentry and GitHub
connectors).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 09:49:17 -08:00
..