mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 19:02:37 +01:00
fix(sentry): report a saved-sandbox resume provider outage only after its retry decision (#21510)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run that uses a saved (reusable) sandbox resumes that sandbox during setup, before the provider work starts. > - When the sandbox provider has an outage, the resume gets an HTTP 5xx answer. Setup fails with `reusable_sandbox_resume_failed`. Paperclip keeps the lease and schedules the bounded bootstrap retry, which resumes the same lease when the provider recovers (#20626). > - The setup path writes the failed status before it decides the retry. That write sends the Sentry run-failure report. Thus each attempt during an outage creates an "unexpected failure" event, also when the retry recovers the run. > - The reviewed chat setup wait had the same defect. #20205 fixed it with a `pending` report handshake. > - This pull request uses the same handshake for a sandbox resume that the provider answered with an HTTP 5xx. > - The benefit: Sentry shows a sandbox resume failure only when the retry cannot recover it. ## Linked Issues or Issue Description **What happened?** Staging Sentry recorded `EnvironmentRunError: Failed to acquire lease for environment "… Sandbox" (sandbox): Reusable sandbox lease "…" could not be resumed (cause=provider_unavailable status=502); the lease was preserved and no replacement was created.` The events came in bursts: 5 different tasks in one minute (HTTP 503/504), and 2 tasks in three minutes on a different day (HTTP 502). All events had `failure_phase=setup` and no provider work. In the source run, the scheduled bootstrap retry got past the resume step. **Expected behavior** A provider outage that the bounded retry owns does not create a Sentry event. A retry that is exhausted, not scheduled, or that fails to schedule still creates one event. A refusal of the sandbox (an HTTP 4xx, or no status) still reports as before. **Steps to reproduce** 1. Use an agent with a reusable sandbox environment that has a saved lease. 2. Make the sandbox provider answer the resume with HTTP 502. 3. Start a run. Setup fails with `reusable_sandbox_resume_failed` and schedules the bootstrap retry. 4. See a Sentry run-failure event for each attempt. **Paperclip version or commit** `2756be2f32` (staging releases `d0057ee384`, `1d153f62b6`, `e23549cd9c` also show the events). I searched for duplicate PRs. #20626 added the retry and safe replacement for a sandbox that will not resume. It does not change when the failure is reported. #20205 is the precedent for the reviewed chat wait. Fixes PAPERCLIP-BACKEND-77 ## What Changed - `server/src/services/reusable-sandbox-resume-error.ts`: `ReusableSandboxResumeError.providerOutage` is true only when the closed HTTP status of the provider answer is 5xx. The text of a gateway page does not decide. - `server/src/services/heartbeat/run-completion.ts`: For a provider outage, the failed-status write sets `sandboxResumeRetry: "pending"`. After the retry decision, the setup path reports the run once with `scheduled` or `not_scheduled`. The deferred-report code is shared with the reviewed chat wait. - `server/src/services/run-failure-report.ts` and `run-failure-diagnostics.ts`: A `pending` sandbox resume write does not report. In `unexpected` mode, a `scheduled` retry of a failed `reusable_sandbox_resume_failed` run does not report. `all` mode still reports it. - Tests: `run-failure-report-sandbox-resume.test.ts` (new) covers the report filter. `run-completion-reviewed-chat-report.test.ts` now also covers the setup path for 502 and 504 with a scheduled retry, an exhausted retry, no retry, a scheduler failure, a 4xx refusal and a refusal without a status. ## Verification - `vitest run src/services/__tests__/run-failure-report-sandbox-resume.test.ts src/services/heartbeat/run-completion-reviewed-chat-report.test.ts src/services/__tests__/run-failure-report.test.ts`: 308 tests pass (embedded Postgres). - Red check: before the fix, the 6 new outage cases fail, because the failed-status write reports immediately. - Earlier, on the base commit, the related run-completion, retry scheduling, recovery and run-failure-report suites passed (407 tests). - `tsc --noEmit -p server/tsconfig.json`: no errors in server sources. - The full workspace install did not fit on the agent sandbox disk, so CI runs the complete affected-test selection. ## Risks - Low risk. Run status, error code, the retry and lease handling do not change. Only the time of the Sentry report changes. - If the process stops between the failed-status write and the retry decision, no report is sent for that attempt. The reviewed chat wait has the same accepted behavior. A later terminal failure of the retry still reports. - 4xx and unclassified resume failures keep the report on the failed-status write. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code agent harness with tool use (shell, file edits, Sentry and GitHub connectors). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
6 files changed
+211
-8
No files matched your search
@@ -0,0 +1,131 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { agents, applyPendingMigrations, closeRegisteredClients, companies, createDb, heartbeatRuns } from "@paperclipai/db";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "../../__tests__/helpers/embedded-postgres.js";
|
||||
|
||||
const mockCaptureRunFailure = vi.hoisted(() => vi.fn());
|
||||
|
||||
vi.mock("../../sentry.js", () => ({
|
||||
captureRunFailure: mockCaptureRunFailure,
|
||||
}));
|
||||
|
||||
import { environmentResumeErrorData, JsonRpcCallError } from "@paperclipai/plugin-sdk";
|
||||
import { reportRunFailure } from "../run-failure-report.js";
|
||||
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE, ReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js";
|
||||
|
||||
const testDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL;
|
||||
const embeddedPostgresSupport = testDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
describeEmbeddedPostgres("reportRunFailure sandbox resume provider outages", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
let companyId!: string;
|
||||
let agentId!: string;
|
||||
let inheritedEnv: NodeJS.ProcessEnv;
|
||||
|
||||
beforeAll(async () => {
|
||||
if (testDatabaseUrl) {
|
||||
await applyPendingMigrations(testDatabaseUrl);
|
||||
db = createDb(testDatabaseUrl);
|
||||
} else {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-run-failure-sandbox-resume-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
}
|
||||
}, 20_000);
|
||||
|
||||
beforeEach(async () => {
|
||||
inheritedEnv = process.env;
|
||||
process.env = Object.fromEntries(
|
||||
["PATH", "HOME", "USER", "USERNAME", "LOGNAME", "USERPROFILE", "TMPDIR", "TEMP", "TMP"]
|
||||
.flatMap((key) => inheritedEnv[key] === undefined ? [] : [[key, inheritedEnv[key]]]),
|
||||
);
|
||||
process.env.PAPERCLIP_SENTRY_RUN_FAILURES = "unexpected";
|
||||
companyId = randomUUID();
|
||||
agentId = randomUUID();
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Paperclip",
|
||||
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
defaultResponsibleUserId: "responsible-user",
|
||||
});
|
||||
await db.insert(agents).values({
|
||||
id: agentId,
|
||||
companyId,
|
||||
name: "SandboxBot",
|
||||
role: "engineer",
|
||||
status: "active",
|
||||
adapterType: "paperclip_runner",
|
||||
adapterConfig: {},
|
||||
runtimeConfig: {},
|
||||
permissions: {},
|
||||
});
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
vi.clearAllMocks();
|
||||
vi.unstubAllEnvs();
|
||||
process.env = inheritedEnv;
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(agents);
|
||||
await db.delete(companies);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (testDatabaseUrl) await closeRegisteredClients(testDatabaseUrl);
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
// The sandbox provider answered the saved-sandbox resume with a gateway
|
||||
// page. The plugin worker sends only the closed status across JSON-RPC.
|
||||
const providerOutage = () => new ReusableSandboxResumeError({ provider: "daytona", providerLeaseId: "lease-1",
|
||||
cause: new JsonRpcCallError({ code: -32002, message: "<html><head><title>502 Bad Gateway</title></head></html>",
|
||||
data: environmentResumeErrorData({ status: 502 }) }) });
|
||||
|
||||
function resumeFailureRun(
|
||||
overrides: Partial<typeof heartbeatRuns.$inferSelect> = {},
|
||||
): typeof heartbeatRuns.$inferSelect {
|
||||
return {
|
||||
id: randomUUID(),
|
||||
companyId,
|
||||
agentId,
|
||||
status: "failed",
|
||||
error: "Reusable sandbox lease could not be resumed",
|
||||
errorCode: REUSABLE_SANDBOX_RESUME_FAILED_CODE,
|
||||
runtimeMode: "legacy",
|
||||
executionStage: "preparing",
|
||||
nativeIssueId: null,
|
||||
contextSnapshot: null,
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
||||
...overrides,
|
||||
} as unknown as typeof heartbeatRuns.$inferSelect;
|
||||
}
|
||||
|
||||
it("keeps a provider outage local until its retry decision, and after a scheduled retry", async () => {
|
||||
const run = resumeFailureRun();
|
||||
const error = providerOutage();
|
||||
await reportRunFailure(db, run, { phase: "setup", error, sandboxResumeRetry: "pending" });
|
||||
await reportRunFailure(db, run, { phase: "setup", error, sandboxResumeRetry: "scheduled" });
|
||||
// The setup path reports once more with its decision, so a pending write stays local in every mode.
|
||||
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
|
||||
await reportRunFailure(db, run, { phase: "setup", error, sandboxResumeRetry: "pending" });
|
||||
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
["no retry was scheduled", {}, "not_scheduled", "unexpected"],
|
||||
["the decision is unknown", {}, undefined, "unexpected"],
|
||||
["every run failure is reported", {}, "scheduled", "all"],
|
||||
["the run failed for another reason", { errorCode: "setup_failed" }, "scheduled", "unexpected"],
|
||||
] as const)("reports a sandbox resume provider outage when %s", async (_label, overrides, sandboxResumeRetry, mode) => {
|
||||
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", mode);
|
||||
const run = resumeFailureRun(overrides);
|
||||
await reportRunFailure(db, run, { phase: "setup", error: providerOutage(), ...(sandboxResumeRetry ? { sandboxResumeRetry } : {}) });
|
||||
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
||||
expect(mockCaptureRunFailure.mock.calls[0]![0]).toMatchObject({ errorCode: run.errorCode, runStatus: "failed" });
|
||||
});
|
||||
});
|
||||
@@ -4,6 +4,9 @@ import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vites
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { createAgentIdentityRedactor } from "../agent-identity-redaction.js";
|
||||
import { ReviewedChatBindingNotReadyError } from "../reviewed-chat-binding-diagnostics.js";
|
||||
import { environmentResumeErrorData, JsonRpcCallError } from "@paperclipai/plugin-sdk";
|
||||
import { EnvironmentRunError } from "../environment-run-orchestrator.js";
|
||||
import { ReusableSandboxResumeError } from "../reusable-sandbox-resume-error.js";
|
||||
import { reportRunFailure } from "../run-failure-report.js";
|
||||
import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "./run-lifecycle.js";
|
||||
import { createHeartbeatRunPreparation } from "./run-preparation.js";
|
||||
@@ -67,7 +70,7 @@ function callbacks(db: Db) {
|
||||
}
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)("reviewed chat setup wait reporting", () => {
|
||||
describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)("setup failure reporting after a bounded retry decision", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: Db;
|
||||
beforeAll(async () => {
|
||||
@@ -126,4 +129,42 @@ describe.skipIf(!support.supported && !process.env.PAPERCLIP_TEST_DATABASE_URL)(
|
||||
expect.objectContaining({ phase: "setup", error,
|
||||
reviewedChatRetry: scenario === "scheduled" || scenario === "interaction_scheduled" ? "scheduled" : "not_scheduled" }));
|
||||
});
|
||||
|
||||
// The sandbox provider answered a saved-sandbox resume with HTTP 502, and the
|
||||
// orchestrator wrapped the refusal exactly as staging recorded it.
|
||||
const resumeRefusal = (cause: unknown) => new EnvironmentRunError("lease_acquire_failed", "Failed to acquire lease", {
|
||||
cause: new ReusableSandboxResumeError({ provider: "daytona", providerLeaseId: "lease-1", cause }) });
|
||||
const rpcRefusal = (httpStatus: number, title = `${httpStatus} Bad Gateway`) => new JsonRpcCallError({ code: -32002,
|
||||
message: `<html><head><title>${title}</title></head></html>`, data: environmentResumeErrorData({ status: httpStatus }) });
|
||||
|
||||
it.each(["scheduled", "gateway_timeout", "not_scheduled", "exhausted", "scheduler_failed", "provider_refusal", "unclassified"] as const)(
|
||||
"reports a sandbox resume provider outage only after its retry decision: %s", async scenario => {
|
||||
const f = await fixture();
|
||||
if (scenario === "scheduler_failed") f.deps.scheduleBoundedRetryForRun.mockRejectedValueOnce(new Error("fixture scheduler failure"));
|
||||
else f.deps.scheduleBoundedRetryForRun.mockResolvedValueOnce(["scheduled", "gateway_timeout", "provider_refusal", "unclassified"].includes(scenario)
|
||||
? { outcome: "scheduled" } as never
|
||||
: scenario === "exhausted" ? { outcome: "retry_exhausted", attempt: 4, maxAttempts: 4 } as never
|
||||
: { outcome: "not_scheduled", reason: "Fixture scheduler boundary", issueId: null });
|
||||
// A 4xx names a refusal of this sandbox, and a failure without a status
|
||||
// is unproven; neither is a provider outage.
|
||||
const error = resumeRefusal(scenario === "provider_refusal" ? rpcRefusal(400)
|
||||
: scenario === "unclassified" ? new Error("worker refused the resume")
|
||||
: scenario === "gateway_timeout" ? rpcRefusal(504, "504 Gateway Time-out") : rpcRefusal(502));
|
||||
await f.completion.failRunSetup(f.setup(error));
|
||||
const write = f.deps.setRunStatusIfRunning.mock.calls.find(call => call[1] === "failed");
|
||||
expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "reusable_sandbox_resume_failed" });
|
||||
expect(f.deps.scheduleBoundedRetryForRun).toHaveBeenCalledOnce();
|
||||
if (scenario === "provider_refusal" || scenario === "unclassified") {
|
||||
// Other resume refusals keep their report on the failed-status write.
|
||||
expect(write?.[3]).toEqual(expect.not.objectContaining({ sandboxResumeRetry: expect.anything() }));
|
||||
expect(reportRunFailure).not.toHaveBeenCalled();
|
||||
return;
|
||||
}
|
||||
expect(write?.[3]).toMatchObject({ phase: "setup", sandboxResumeRetry: "pending" });
|
||||
expect(reportRunFailure).toHaveBeenCalledOnce();
|
||||
expect(reportRunFailure).toHaveBeenCalledWith(db,
|
||||
expect.objectContaining({ id: f.run.id, status: "failed", errorCode: "reusable_sandbox_resume_failed" }),
|
||||
expect.objectContaining({ phase: "setup", error,
|
||||
sandboxResumeRetry: scenario === "scheduled" || scenario === "gateway_timeout" ? "scheduled" : "not_scheduled" }));
|
||||
});
|
||||
});
|
||||
@@ -1704,6 +1704,10 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
|
||||
(sandboxDiskFull ? NATIVE_SANDBOX_DISK_FULL : null) ??
|
||||
"setup_failed";
|
||||
const failureCause = GENERIC_RUN_FAILURE_CODES.has(setupFailureErrorCode) ? thrownFailureCause : null;
|
||||
// The bounded retry resumes the same lease once the provider recovers, so
|
||||
// a provider outage is reported only after that retry decision.
|
||||
const sandboxResumeOutage = setupFailureErrorCode === reusableSandboxResumeFailure?.code
|
||||
&& reusableSandboxResumeFailure.providerOutage;
|
||||
// Bounded kind and reason only: a workspace provider's own text never crosses into Core.
|
||||
const workspaceProviderFailure = findWorkspaceProviderFailure(outerErr)?.summary() ?? null;
|
||||
// A wrapped query error hides its driver cause (for example
|
||||
@@ -1795,8 +1799,9 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
|
||||
}, { error: outerErr, phase: "setup", secretValues: readFailureReportSecrets(),
|
||||
// The Sentry report waits for the bounded retry decision below.
|
||||
...(reviewedChatBindingNotReady ? { reviewedChatRetry: "pending" as const } : {}),
|
||||
...(sandboxResumeOutage ? { sandboxResumeRetry: "pending" as const } : {}),
|
||||
}, { setupFailure: true }).catch(() => ({ run: null, updated: false as const }));
|
||||
let reviewedChatRetryScheduled = false;
|
||||
let setupRetryScheduled = false;
|
||||
if (!setupFailureWrite.updated) {
|
||||
logger.info(
|
||||
{
|
||||
@@ -1900,7 +1905,7 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
|
||||
);
|
||||
return null;
|
||||
});
|
||||
reviewedChatRetryScheduled = retry?.outcome === "scheduled";
|
||||
setupRetryScheduled = retry?.outcome === "scheduled";
|
||||
storageWait = retry;
|
||||
storageWaitScheduled = livenessRun.errorCode === STORAGE_UNAVAILABLE_CODE
|
||||
&& (retry?.outcome === "scheduled" || retry?.outcome === "retry_exhausted");
|
||||
@@ -1932,11 +1937,12 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
|
||||
);
|
||||
}
|
||||
// This path owned the failed-status write, which deferred the report.
|
||||
const reviewedChatReportRun = reviewedChatBindingNotReady && setupFailureWrite.updated
|
||||
const deferredReportRun = (reviewedChatBindingNotReady || sandboxResumeOutage) && setupFailureWrite.updated
|
||||
? failedRun ?? setupFailureWrite.run : null;
|
||||
if (reviewedChatReportRun) {
|
||||
void reportRunFailure(db, reviewedChatReportRun, { error: outerErr, phase: "setup", secretValues: readFailureReportSecrets(),
|
||||
reviewedChatRetry: reviewedChatRetryScheduled ? "scheduled" : "not_scheduled" });
|
||||
if (deferredReportRun) {
|
||||
const retryDecision = setupRetryScheduled ? "scheduled" as const : "not_scheduled" as const;
|
||||
void reportRunFailure(db, deferredReportRun, { error: outerErr, phase: "setup", secretValues: readFailureReportSecrets(),
|
||||
...(reviewedChatBindingNotReady ? { reviewedChatRetry: retryDecision } : { sandboxResumeRetry: retryDecision }) });
|
||||
}
|
||||
// Ensure the agent is not left stuck in "running" if the setup-failure
|
||||
// path owned the terminal transition. If another path already finalized
|
||||
|
||||
@@ -14,6 +14,11 @@ export class ReusableSandboxResumeError extends Error {
|
||||
readonly code = REUSABLE_SANDBOX_RESUME_FAILED_CODE;
|
||||
readonly provider: string;
|
||||
readonly providerLeaseId: string;
|
||||
/**
|
||||
* The provider answered the resume with an HTTP 5xx. That is an outage of
|
||||
* the provider, not a refusal of this sandbox, so the bounded retry owns it.
|
||||
*/
|
||||
readonly providerOutage: boolean;
|
||||
|
||||
constructor(input: {
|
||||
provider: string;
|
||||
@@ -41,6 +46,8 @@ export class ReusableSandboxResumeError extends Error {
|
||||
this.name = "ReusableSandboxResumeError";
|
||||
this.provider = input.provider;
|
||||
this.providerLeaseId = input.providerLeaseId;
|
||||
// Only the closed status decides: a gateway page's text can also read as a timeout.
|
||||
this.providerOutage = failure?.status != null && failure.status >= 500 && failure.status <= 599;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -46,6 +46,11 @@ export interface RunFailureReportOptions {
|
||||
* decision; a scheduled retry owns recovery and reports its own outcome.
|
||||
*/
|
||||
reviewedChatRetry?: "pending" | "scheduled" | "not_scheduled";
|
||||
/**
|
||||
* The bounded retry decision for a saved-sandbox resume that the provider
|
||||
* answered with an HTTP 5xx, with the same `pending` handshake.
|
||||
*/
|
||||
sandboxResumeRetry?: "pending" | "scheduled" | "not_scheduled";
|
||||
}
|
||||
|
||||
export interface RunFailureException {
|
||||
|
||||
@@ -23,6 +23,7 @@ import { NATIVE_PROVIDER_OVERLOADED_CODE } from "./native-runtime/native-provide
|
||||
import { NATIVE_TOOL_RECEIPT_LIMIT_CODE } from "./native-runtime/native-tool-receipt-limit.js";
|
||||
import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js";
|
||||
import { REVIEWED_CHAT_BINDING_NOT_READY } from "./reviewed-chat-binding-diagnostics.js";
|
||||
import { REUSABLE_SANDBOX_RESUME_FAILED_CODE } from "./reusable-sandbox-resume-error.js";
|
||||
import { readProviderFailure } from "@paperclipai/shared";
|
||||
|
||||
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
|
||||
@@ -347,6 +348,17 @@ function isScheduledReviewedChatRetry(run: HeartbeatRun, options: RunFailureRepo
|
||||
options.reviewedChatRetry === "scheduled";
|
||||
}
|
||||
|
||||
/**
|
||||
* The sandbox provider answered a saved-sandbox resume with an HTTP 5xx before
|
||||
* provider dispatch, and the setup path scheduled the bounded retry of the same
|
||||
* lease. The retry owns recovery; an exhausted or unscheduled retry, and any
|
||||
* other resume refusal, stays reportable.
|
||||
*/
|
||||
function isScheduledSandboxResumeRetry(run: HeartbeatRun, options: RunFailureReportOptions): boolean {
|
||||
return run.status === "failed" && run.errorCode === REUSABLE_SANDBOX_RESUME_FAILED_CODE &&
|
||||
options.sandboxResumeRetry === "scheduled";
|
||||
}
|
||||
|
||||
const providerAuthErrorCodes = new Set(["claude_auth_required", "acpx_auth_required"]);
|
||||
|
||||
/**
|
||||
@@ -401,7 +413,7 @@ export function reportRunFailure(db: Db, run: HeartbeatRun, options: RunFailureR
|
||||
// The terminal write that ends the retry reports instead (`isNewTerminalRunFailure`).
|
||||
if (isPendingNativeRetry(run)) return Promise.resolve();
|
||||
// The setup path reports this run again once it has decided the retry.
|
||||
if (options.reviewedChatRetry === "pending") return Promise.resolve();
|
||||
if (options.reviewedChatRetry === "pending" || options.sandboxResumeRetry === "pending") return Promise.resolve();
|
||||
if (mode === "unexpected" && (
|
||||
isMissingSecretConfigurationBlocker(run, options) ||
|
||||
isAiConnectionSelectionBlocker(run, options) ||
|
||||
@@ -416,6 +428,7 @@ export function reportRunFailure(db: Db, run: HeartbeatRun, options: RunFailureR
|
||||
isCodexAcpModelCapacityFailure(run, options) ||
|
||||
isAcpProviderAccessFailure(run, options) ||
|
||||
isScheduledReviewedChatRetry(run, options) ||
|
||||
isScheduledSandboxResumeRetry(run, options) ||
|
||||
isRestartInterruptedSetup(run)
|
||||
)) return Promise.resolve();
|
||||
const runStatus = run.status;
|
||||
|
||||
Reference in new issue
Block a user