fix(sentry): report a saved-sandbox resume provider outage only after its retry decision (#21510)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A run that uses a saved (reusable) sandbox resumes that sandbox
during setup, before the provider work starts.
> - When the sandbox provider has an outage, the resume gets an HTTP 5xx
answer. Setup fails with `reusable_sandbox_resume_failed`. Paperclip
keeps the lease and schedules the bounded bootstrap retry, which resumes
the same lease when the provider recovers (#20626).
> - The setup path writes the failed status before it decides the retry.
That write sends the Sentry run-failure report. Thus each attempt during
an outage creates an "unexpected failure" event, also when the retry
recovers the run.
> - The reviewed chat setup wait had the same defect. #20205 fixed it
with a `pending` report handshake.
> - This pull request uses the same handshake for a sandbox resume that
the provider answered with an HTTP 5xx.
> - The benefit: Sentry shows a sandbox resume failure only when the
retry cannot recover it.

## Linked Issues or Issue Description

**What happened?**

Staging Sentry recorded `EnvironmentRunError: Failed to acquire lease
for environment "… Sandbox" (sandbox): Reusable sandbox lease "…" could
not be resumed (cause=provider_unavailable status=502); the lease was
preserved and no replacement was created.` The events came in bursts: 5
different tasks in one minute (HTTP 503/504), and 2 tasks in three
minutes on a different day (HTTP 502). All events had
`failure_phase=setup` and no provider work. In the source run, the
scheduled bootstrap retry got past the resume step.

**Expected behavior**

A provider outage that the bounded retry owns does not create a Sentry
event. A retry that is exhausted, not scheduled, or that fails to
schedule still creates one event. A refusal of the sandbox (an HTTP 4xx,
or no status) still reports as before.

**Steps to reproduce**

1. Use an agent with a reusable sandbox environment that has a saved
lease.
2. Make the sandbox provider answer the resume with HTTP 502.
3. Start a run. Setup fails with `reusable_sandbox_resume_failed` and
schedules the bootstrap retry.
4. See a Sentry run-failure event for each attempt.

**Paperclip version or commit**

`2756be2f32` (staging releases `d0057ee384`, `1d153f62b6`, `e23549cd9c`
also show the events).

I searched for duplicate PRs. #20626 added the retry and safe
replacement for a sandbox that will not resume. It does not change when
the failure is reported. #20205 is the precedent for the reviewed chat
wait.

Fixes PAPERCLIP-BACKEND-77

## What Changed

- `server/src/services/reusable-sandbox-resume-error.ts`:
`ReusableSandboxResumeError.providerOutage` is true only when the closed
HTTP status of the provider answer is 5xx. The text of a gateway page
does not decide.
- `server/src/services/heartbeat/run-completion.ts`: For a provider
outage, the failed-status write sets `sandboxResumeRetry: "pending"`.
After the retry decision, the setup path reports the run once with
`scheduled` or `not_scheduled`. The deferred-report code is shared with
the reviewed chat wait.
- `server/src/services/run-failure-report.ts` and
`run-failure-diagnostics.ts`: A `pending` sandbox resume write does not
report. In `unexpected` mode, a `scheduled` retry of a failed
`reusable_sandbox_resume_failed` run does not report. `all` mode still
reports it.
- Tests: `run-failure-report-sandbox-resume.test.ts` (new) covers the
report filter. `run-completion-reviewed-chat-report.test.ts` now also
covers the setup path for 502 and 504 with a scheduled retry, an
exhausted retry, no retry, a scheduler failure, a 4xx refusal and a
refusal without a status.

## Verification

- `vitest run
src/services/__tests__/run-failure-report-sandbox-resume.test.ts
src/services/heartbeat/run-completion-reviewed-chat-report.test.ts
src/services/__tests__/run-failure-report.test.ts`: 308 tests pass
(embedded Postgres).
- Red check: before the fix, the 6 new outage cases fail, because the
failed-status write reports immediately.
- Earlier, on the base commit, the related run-completion, retry
scheduling, recovery and run-failure-report suites passed (407 tests).
- `tsc --noEmit -p server/tsconfig.json`: no errors in server sources.
- The full workspace install did not fit on the agent sandbox disk, so
CI runs the complete affected-test selection.

## Risks

- Low risk. Run status, error code, the retry and lease handling do not
change. Only the time of the Sentry report changes.
- If the process stops between the failed-status write and the retry
decision, no report is sent for that attempt. The reviewed chat wait has
the same accepted behavior. A later terminal failure of the retry still
reports.
- 4xx and unclassified resume failures keep the report on the
failed-status write.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code
agent harness with tool use (shell, file edits, Sentry and GitHub
connectors).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-20 09:49:17 -08:00
1 parent 88469bd2bc
commit 1a50bd550e
6 files changed
+211 -8

No files matched your search

@@ -0,0 +1,131 @@
import { randomUUID } from "node:crypto";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { agents, applyPendingMigrations, closeRegisteredClients, companies, createDb, heartbeatRuns } from "@paperclipai/db";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "../../__tests__/helpers/embedded-postgres.js";
const mockCaptureRunFailure = vi.hoisted(() => vi.fn());
vi.mock("../../sentry.js", () => ({
captureRunFailure: mockCaptureRunFailure,
}));
import { environmentResumeErrorData, JsonRpcCallError } from "@paperclipai/plugin-sdk";
import { reportRunFailure } from "../run-failure-report.js";
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE, ReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js";
const testDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL;
const embeddedPostgresSupport = testDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
describeEmbeddedPostgres("reportRunFailure sandbox resume provider outages", () => {
let db!: ReturnType<typeof createDb>;
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
let companyId!: string;
let agentId!: string;
let inheritedEnv: NodeJS.ProcessEnv;
beforeAll(async () => {
if (testDatabaseUrl) {
await applyPendingMigrations(testDatabaseUrl);
db = createDb(testDatabaseUrl);
} else {
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-run-failure-sandbox-resume-");
db = createDb(tempDb.connectionString);
}
}, 20_000);
beforeEach(async () => {
inheritedEnv = process.env;
process.env = Object.fromEntries(
["PATH", "HOME", "USER", "USERNAME", "LOGNAME", "USERPROFILE", "TMPDIR", "TEMP", "TMP"]
.flatMap((key) => inheritedEnv[key] === undefined ? [] : [[key, inheritedEnv[key]]]),
);
process.env.PAPERCLIP_SENTRY_RUN_FAILURES = "unexpected";
companyId = randomUUID();
agentId = randomUUID();
await db.insert(companies).values({
id: companyId,
name: "Paperclip",
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
requireBoardApprovalForNewAgents: false,
defaultResponsibleUserId: "responsible-user",
});
await db.insert(agents).values({
id: agentId,
companyId,
name: "SandboxBot",
role: "engineer",
status: "active",
adapterType: "paperclip_runner",
adapterConfig: {},
runtimeConfig: {},
permissions: {},
});
});
afterEach(async () => {
vi.clearAllMocks();
vi.unstubAllEnvs();
process.env = inheritedEnv;
await db.delete(heartbeatRuns);
await db.delete(agents);
await db.delete(companies);
});
afterAll(async () => {
if (testDatabaseUrl) await closeRegisteredClients(testDatabaseUrl);
await tempDb?.cleanup();
});
// The sandbox provider answered the saved-sandbox resume with a gateway
// page. The plugin worker sends only the closed status across JSON-RPC.
const providerOutage = () => new ReusableSandboxResumeError({ provider: "daytona", providerLeaseId: "lease-1",
cause: new JsonRpcCallError({ code: -32002, message: "<html><head><title>502 Bad Gateway</title></head></html>",
data: environmentResumeErrorData({ status: 502 }) }) });
function resumeFailureRun(
overrides: Partial<typeof heartbeatRuns.$inferSelect> = {},
): typeof heartbeatRuns.$inferSelect {
return {
id: randomUUID(),
companyId,
agentId,
status: "failed",
error: "Reusable sandbox lease could not be resumed",
errorCode: REUSABLE_SANDBOX_RESUME_FAILED_CODE,
runtimeMode: "legacy",
executionStage: "preparing",
nativeIssueId: null,
contextSnapshot: null,
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
...overrides,
} as unknown as typeof heartbeatRuns.$inferSelect;
}
it("keeps a provider outage local until its retry decision, and after a scheduled retry", async () => {
const run = resumeFailureRun();
const error = providerOutage();
await reportRunFailure(db, run, { phase: "setup", error, sandboxResumeRetry: "pending" });
await reportRunFailure(db, run, { phase: "setup", error, sandboxResumeRetry: "scheduled" });
// The setup path reports once more with its decision, so a pending write stays local in every mode.
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
await reportRunFailure(db, run, { phase: "setup", error, sandboxResumeRetry: "pending" });
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
});
it.each([
["no retry was scheduled", {}, "not_scheduled", "unexpected"],
["the decision is unknown", {}, undefined, "unexpected"],
["every run failure is reported", {}, "scheduled", "all"],
["the run failed for another reason", { errorCode: "setup_failed" }, "scheduled", "unexpected"],
] as const)("reports a sandbox resume provider outage when %s", async (_label, overrides, sandboxResumeRetry, mode) => {
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", mode);
const run = resumeFailureRun(overrides);
await reportRunFailure(db, run, { phase: "setup", error: providerOutage(), ...(sandboxResumeRetry ? { sandboxResumeRetry } : {}) });
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
expect(mockCaptureRunFailure.mock.calls[0]![0]).toMatchObject({ errorCode: run.errorCode, runStatus: "failed" });
});
});
@@ -4,6 +4,9 @@ import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vites
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { createAgentIdentityRedactor } from "../agent-identity-redaction.js";
import { ReviewedChatBindingNotReadyError } from "../reviewed-chat-binding-diagnostics.js";
import { environmentResumeErrorData, JsonRpcCallError } from "@paperclipai/plugin-sdk";
import { EnvironmentRunError } from "../environment-run-orchestrator.js";
import { ReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js";
import { reportRunFailure } from "../run-failure-report.js";
import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "./run-lifecycle.js";
import { createHeartbeatRunPreparation } from "./run-preparation.js";
@@ -67,7 +70,7 @@ function callbacks(db: Db) {
}
const support = await getEmbeddedPostgresTestSupport();
describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)("reviewed chat setup wait reporting", () => {
describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)("setup failure reporting after a bounded retry decision", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: Db;
beforeAll(async () => {
@@ -126,4 +129,42 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
expect.objectContaining({ phase: "setup", error,
reviewedChatRetry: scenario === "scheduled" || scenario === "interaction_scheduled" ? "scheduled" : "not_scheduled" }));
});
// The sandbox provider answered a saved-sandbox resume with HTTP 502, and the
// orchestrator wrapped the refusal exactly as staging recorded it.
const resumeRefusal = (cause: unknown) => new EnvironmentRunError("lease_acquire_failed", "Failed to acquire lease", {
cause: new ReusableSandboxResumeError({ provider: "daytona", providerLeaseId: "lease-1", cause }) });
const rpcRefusal = (httpStatus: number, title = `${httpStatus} Bad Gateway`) => new JsonRpcCallError({ code: -32002,
message: `<html><head><title>${title}</title></head></html>`, data: environmentResumeErrorData({ status: httpStatus }) });
it.each(["scheduled", "gateway_timeout", "not_scheduled", "exhausted", "scheduler_failed", "provider_refusal", "unclassified"] as const)(
"reports a sandbox resume provider outage only after its retry decision: %s", async scenario => {
const f = await fixture();
if (scenario === "scheduler_failed") f.deps.scheduleBoundedRetryForRun.mockRejectedValueOnce(new Error("fixture scheduler failure"));
else f.deps.scheduleBoundedRetryForRun.mockResolvedValueOnce(["scheduled", "gateway_timeout", "provider_refusal", "unclassified"].includes(scenario)
? { outcome: "scheduled" } as never
: scenario === "exhausted" ? { outcome: "retry_exhausted", attempt: 4, maxAttempts: 4 } as never
: { outcome: "not_scheduled", reason: "Fixture scheduler boundary", issueId: null });
// A 4xx names a refusal of this sandbox, and a failure without a status
// is unproven; neither is a provider outage.
const error = resumeRefusal(scenario === "provider_refusal" ? rpcRefusal(400)
: scenario === "unclassified" ? new Error("worker refused the resume")
: scenario === "gateway_timeout" ? rpcRefusal(504, "504 Gateway Time-out") : rpcRefusal(502));
await f.completion.failRunSetup(f.setup(error));
const write = f.deps.setRunStatusIfRunning.mock.calls.find(call => call[1] === "failed");
expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "reusable_sandbox_resume_failed" });
expect(f.deps.scheduleBoundedRetryForRun).toHaveBeenCalledOnce();
if (scenario === "provider_refusal" || scenario === "unclassified") {
// Other resume refusals keep their report on the failed-status write.
expect(write?.[3]).toEqual(expect.not.objectContaining({ sandboxResumeRetry: expect.anything() }));
expect(reportRunFailure).not.toHaveBeenCalled();
return;
}
expect(write?.[3]).toMatchObject({ phase: "setup", sandboxResumeRetry: "pending" });
expect(reportRunFailure).toHaveBeenCalledOnce();
expect(reportRunFailure).toHaveBeenCalledWith(db,
expect.objectContaining({ id: f.run.id, status: "failed", errorCode: "reusable_sandbox_resume_failed" }),
expect.objectContaining({ phase: "setup", error,
sandboxResumeRetry: scenario === "scheduled" || scenario === "gateway_timeout" ? "scheduled" : "not_scheduled" }));
});
});
@@ -1704,6 +1704,10 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
(sandboxDiskFull ? NATIVE_SANDBOX_DISK_FULL : null) ??
"setup_failed";
const failureCause = GENERIC_RUN_FAILURE_CODES.has(setupFailureErrorCode) ? thrownFailureCause : null;
// The bounded retry resumes the same lease once the provider recovers, so
// a provider outage is reported only after that retry decision.
const sandboxResumeOutage = setupFailureErrorCode === reusableSandboxResumeFailure?.code
&& reusableSandboxResumeFailure.providerOutage;
// Bounded kind and reason only: a workspace provider's own text never crosses into Core.
const workspaceProviderFailure = findWorkspaceProviderFailure(outerErr)?.summary() ?? null;
// A wrapped query error hides its driver cause (for example
@@ -1795,8 +1799,9 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
}, { error: outerErr, phase: "setup", secretValues: readFailureReportSecrets(),
// The Sentry report waits for the bounded retry decision below.
...(reviewedChatBindingNotReady ? { reviewedChatRetry: "pending" as const } : {}),
...(sandboxResumeOutage ? { sandboxResumeRetry: "pending" as const } : {}),
}, { setupFailure: true }).catch(() => ({ run: null, updated: false as const }));
let reviewedChatRetryScheduled = false;
let setupRetryScheduled = false;
if (!setupFailureWrite.updated) {
logger.info(
{
@@ -1900,7 +1905,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
);
return null;
});
reviewedChatRetryScheduled = retry?.outcome === "scheduled";
setupRetryScheduled = retry?.outcome === "scheduled";
storageWait = retry;
storageWaitScheduled = livenessRun.errorCode === STORAGE_UNAVAILABLE_CODE
&& (retry?.outcome === "scheduled" || retry?.outcome === "retry_exhausted");
@@ -1932,11 +1937,12 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
);
}
// This path owned the failed-status write, which deferred the report.
const reviewedChatReportRun = reviewedChatBindingNotReady && setupFailureWrite.updated
const deferredReportRun = (reviewedChatBindingNotReady || sandboxResumeOutage) && setupFailureWrite.updated
? failedRun ?? setupFailureWrite.run : null;
if (reviewedChatReportRun) {
void reportRunFailure(db, reviewedChatReportRun, { error: outerErr, phase: "setup", secretValues: readFailureReportSecrets(),
reviewedChatRetry: reviewedChatRetryScheduled ? "scheduled" : "not_scheduled" });
if (deferredReportRun) {
const retryDecision = setupRetryScheduled ? "scheduled" as const : "not_scheduled" as const;
void reportRunFailure(db, deferredReportRun, { error: outerErr, phase: "setup", secretValues: readFailureReportSecrets(),
...(reviewedChatBindingNotReady ? { reviewedChatRetry: retryDecision } : { sandboxResumeRetry: retryDecision }) });
}
// Ensure the agent is not left stuck in "running" if the setup-failure
// path owned the terminal transition. If another path already finalized
@@ -14,6 +14,11 @@ export class ReusableSandboxResumeError extends Error {
readonly code = REUSABLE_SANDBOX_RESUME_FAILED_CODE;
readonly provider: string;
readonly providerLeaseId: string;
/**
* The provider answered the resume with an HTTP 5xx. That is an outage of
* the provider, not a refusal of this sandbox, so the bounded retry owns it.
*/
readonly providerOutage: boolean;
constructor(input: {
provider: string;
@@ -41,6 +46,8 @@ export class ReusableSandboxResumeError extends Error {
this.name = "ReusableSandboxResumeError";
this.provider = input.provider;
this.providerLeaseId = input.providerLeaseId;
// Only the closed status decides: a gateway page's text can also read as a timeout.
this.providerOutage = failure?.status != null && failure.status >= 500 && failure.status <= 599;
}
}
@@ -46,6 +46,11 @@ export interface RunFailureReportOptions {
* decision; a scheduled retry owns recovery and reports its own outcome.
*/
reviewedChatRetry?: "pending" | "scheduled" | "not_scheduled";
/**
* The bounded retry decision for a saved-sandbox resume that the provider
* answered with an HTTP 5xx, with the same `pending` handshake.
*/
sandboxResumeRetry?: "pending" | "scheduled" | "not_scheduled";
}
export interface RunFailureException {
+14 -1
View File
@@ -23,6 +23,7 @@ import { NATIVE_PROVIDER_OVERLOADED_CODE } from "./native-runtime/native-provide
import { NATIVE_TOOL_RECEIPT_LIMIT_CODE } from "./native-runtime/native-tool-receipt-limit.js";
import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js";
import { REVIEWED_CHAT_BINDING_NOT_READY } from "./reviewed-chat-binding-diagnostics.js";
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "./reusable-sandbox-resume-error.js";
import { readProviderFailure } from "@paperclipai/shared";
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
@@ -347,6 +348,17 @@ function isScheduledReviewedChatRetry(run: HeartbeatRun, options: RunFailureRepo
options.reviewedChatRetry === "scheduled";
}
/**
* The sandbox provider answered a saved-sandbox resume with an HTTP 5xx before
* provider dispatch, and the setup path scheduled the bounded retry of the same
* lease. The retry owns recovery; an exhausted or unscheduled retry, and any
* other resume refusal, stays reportable.
*/
function isScheduledSandboxResumeRetry(run: HeartbeatRun, options: RunFailureReportOptions): boolean {
return run.status === "failed" && run.errorCode === REUSABLE_SANDBOX_RESUME_FAILED_CODE &&
options.sandboxResumeRetry === "scheduled";
}
const providerAuthErrorCodes = new Set(["claude_auth_required", "acpx_auth_required"]);
/**
@@ -401,7 +413,7 @@ export function reportRunFailure(db: Db, run: HeartbeatRun, options: RunFailureR
// The terminal write that ends the retry reports instead (`isNewTerminalRunFailure`).
if (isPendingNativeRetry(run)) return Promise.resolve();
// The setup path reports this run again once it has decided the retry.
if (options.reviewedChatRetry === "pending") return Promise.resolve();
if (options.reviewedChatRetry === "pending" || options.sandboxResumeRetry === "pending") return Promise.resolve();
if (mode === "unexpected" && (
isMissingSecretConfigurationBlocker(run, options) ||
isAiConnectionSelectionBlocker(run, options) ||
@@ -416,6 +428,7 @@ export function reportRunFailure(db: Db, run: HeartbeatRun, options: RunFailureR
isCodexAcpModelCapacityFailure(run, options) ||
isAcpProviderAccessFailure(run, options) ||
isScheduledReviewedChatRetry(run, options) ||
isScheduledSandboxResumeRetry(run, options) ||
isRestartInterruptedSetup(run)
)) return Promise.resolve();
const runStatus = run.status;