From 2a221e8e801a3fafa53d8476936c5818abff5578 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 30 Oct 2026 05:42:43 -0700 Subject: [PATCH] fix(sentry): keep scheduled native model capacity retries out of run-failure reports (#18295) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The run-failure reporter sends terminal run failures to Sentry when an operator sets `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` > - In `unexpected` mode, the reporter skips failures that a bounded automatic retry already owns. The Codex ACP model capacity failure (`codex_transient_upstream`) is one of these > - The native runner has the same capacity failure: Codex `serverOverloaded` ("Selected model is at capacity"). Native finalization schedules a bounded automatic retry for it (#15347) > - The reporter has no matching exclusion for the native path. Each scheduled capacity retry also sends a Sentry error, even when the retry recovers the task > - This pull request skips the report only when the native failure has typed capacity evidence, a committed retry decision, and the retry run that the decision created > - The benefit is that Sentry shows capacity failures only when automatic recovery stops (retries are exhausted, held, or not scheduled) ## Linked Issues or Issue Description Refs #15347 (native capacity retry). Refs #16407 (Codex ACP capacity exclusion). **What happened?** A native Codex run failed with "Selected model is at capacity. Please try a different model." (`native_provider_overloaded`). Native finalization scheduled the automatic capacity retry. The retry run succeeded and the task completed. The run-failure reporter still sent a Sentry error event for the first run. **Expected behavior** When the automatic capacity retry owns recovery, the run failure stays in the local run log and does not go to Sentry. This is the same as the Codex ACP capacity path. When retries are exhausted or held, or when no retry is scheduled, the failure goes to Sentry. **Steps to reproduce** 1. Set `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` with a backend Sentry DSN. 2. Run a native Codex agent turn that ends with a Codex `serverOverloaded` terminal error. 3. See that native finalization creates a `scheduled_retry` run with `scheduledRetryReason=native_provider_overloaded`. 4. See that Sentry also receives a run-failure event for the failed run. ## What Changed - `server/src/services/run-failure-report.ts`: add `isScheduledNativeCapacityRetry`. In `unexpected` mode it skips the report only when all of these are true: - the run is native, `failed`, has `errorCode=native_provider_overloaded`, and has a native task (`nativeIssueId`); - `resultJson.nativeProviderFailure` is the closed `codex`/`server_overloaded` diagnostic; - `resultJson.finalizationPhase` is `committed` and `finalizationReasonCode` is `native_provider_overloaded`; - the report is not from setup, and the run has no signal; - a successor run for the same task exists with `retryOfRunId=run.id` and `scheduledRetryReason=native_provider_overloaded`. The same status transaction creates this run. - The successor lookup filters on `company_id`, `context_snapshot ->> 'issueId'` and `created_at >= run.createdAt`, so the existing `heartbeat_runs_company_ctx_issue_created_idx` bounds it to that task's runs since the failure. No migration is needed. - A failed read of the retry evidence logs a warning and reports the failure. - `doc/observability.md`: document the new exclusion in the run failure context section. - Unit tests: the exclusion, and twelve cases that must still report (exhausted, held, no retry run, another retry lane, successor for another task, run with no native task, finalization not committed, wrong or missing diagnostic, legacy run, signal, timeout). Also setup phase, `all` mode, and an unreadable retry record. - Integration test (real native finalizer): no Sentry event for retries 1 and 2, one event when retries are exhausted, and one event when reassignment prevents a retry. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/__tests__/run-failure-report.test.ts src/services/native-runtime/native-provider-overload.integration.test.ts src/services/native-runtime/native-provider-failure.test.ts src/services/__tests__/status-arbiter.test.ts`: 359 tests pass. - Red/green: before the fix, the new unit test captured 2 events, and the integration test captured an event on the first scheduled retry. After the fix, both pass. - `EXPLAIN` on the migrated schema: `Index Scan using heartbeat_runs_company_ctx_issue_created_idx` with company, context issue and created time in `Index Cond`. - CI provides the full workspace typecheck. ## Risks - Low risk. The change applies only in `unexpected` mode and only to native runs with typed capacity evidence and an existing retry run. `all` mode still reports every failure. - The check adds one index-bounded read of `heartbeat_runs` for runs that match the in-memory capacity checks. Other runs do not do this read. - If a scheduled retry later fails, that retry run reports its own outcome. When retries are exhausted, the final failure reports. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with tool use (shell, file edits, test execution) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/observability.md | 9 +++ .../__tests__/run-failure-report.test.ts | 73 +++++++++++++++++++ ...tive-provider-overload.integration.test.ts | 19 ++++- server/src/services/run-failure-report.ts | 34 ++++++++- 4 files changed, 133 insertions(+), 2 deletions(-) diff --git a/doc/observability.md b/doc/observability.md index 8da760ebf8d..94103a3a5a2 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -1070,6 +1070,15 @@ report must come after setup and carry no signal. Other `service` failures, untyped capacity text, and Codex CLI transient upstream failures remain reportable. The run keeps its failed state and its terminal diagnostic. +A native Codex model capacity failure stays local only while its automatic +retry owns recovery. The run must carry `native_provider_overloaded`, the closed +`codex`/`server_overloaded` diagnostic, and a committed finalization whose +reason is `native_provider_overloaded`. The report also requires the +`native_provider_overloaded` retry run that the same status transaction created. +Exhausted or held capacity failures, other retry lanes, setup failures, signals, +and unreadable retry evidence remain reportable. A failed retry reports its own +outcome. + Expected pre-provider waits for a shared project workspace, a credential rotation lock, or an exhausted AI connection pool carry a server-owned expected cancellation receipt. They retain existing retry and lock behavior without a diff --git a/server/src/services/__tests__/run-failure-report.test.ts b/server/src/services/__tests__/run-failure-report.test.ts index 2ce0d6891fe..e471046ae03 100644 --- a/server/src/services/__tests__/run-failure-report.test.ts +++ b/server/src/services/__tests__/run-failure-report.test.ts @@ -430,6 +430,79 @@ describeEmbeddedPostgres("reportRunFailure", () => { expect(mockCaptureRunFailure).toHaveBeenCalledTimes(2); }); + /** The persisted shape after the native finalizer commits a capacity decision. */ + async function nativeCapacityFailure(input: { + run?: Partial; + resultJson?: Record; + retry?: Partial | null; + } = {}) { + await seedCompanyAndAgent({ adapterType: "paperclip_runner" }); + const run = buildRun({ + error: "Selected model is at capacity. Please try a different model.", + errorCode: "native_provider_overloaded", runtimeMode: "native", nativePhase: "committed", + executionStage: "preparing", exitCode: null, signal: null, + resultJson: { + nativeProviderFailure: { provider: "codex", category: "server_overloaded" }, + finalizationPhase: "committed", finalizationReasonCode: "native_provider_overloaded", + ...input.resultJson, + }, + ...input.run, + }); + const [stored] = await db.insert(heartbeatRuns).values(run).returning(); + if (input.retry !== null) { + await db.insert(heartbeatRuns).values({ + companyId, agentId, invocationSource: "automation", triggerDetail: "system", status: "scheduled_retry", + retryOfRunId: run.id, scheduledRetryReason: "native_provider_overloaded", scheduledRetryAttempt: 1, + scheduledRetryAt: new Date(Date.now() + 60_000), errorCode: "native_provider_overloaded", + contextSnapshot: { issueId: run.nativeIssueId, retryOfRunId: run.id }, ...input.retry, + }); + } + return stored!; + } + + it("keeps a native model capacity failure local once its automatic retry is scheduled", async () => { + const run = await nativeCapacityFailure(); + await reportRunFailure(db, run); + await reportRunFailure(db, run, { phase: "execute" }); + expect(mockCaptureRunFailure).not.toHaveBeenCalled(); + }); + + it.each([ + ["the retries are exhausted", { resultJson: { finalizationReasonCode: "native_provider_overloaded_exhausted" }, retry: null }], + ["the task holds the retry", { resultJson: { finalizationReasonCode: "native_provider_overloaded_held" }, retry: null }], + ["no retry run exists", { retry: null }], + ["the successor belongs to another retry lane", { retry: { scheduledRetryReason: "native_safe_replacement" } }], + ["the successor belongs to another task", { retry: { contextSnapshot: { issueId: randomUUID() } } }], + ["the run has no native task", { run: { nativeIssueId: null } }], + ["finalization is not committed", { resultJson: { finalizationPhase: "retryable_failure" } }], + ["the diagnostic is not typed capacity evidence", { resultJson: { nativeProviderFailure: { provider: "codex", category: "turn_failed" } } }], + ["the diagnostic is missing", { resultJson: { nativeProviderFailure: undefined } }], + ["the run is legacy", { run: { runtimeMode: "legacy" } }], + ["the run has a signal", { run: { signal: "SIGTERM" } }], + ["the run timed out", { run: { status: "timed_out" } }], + ] as const)("reports a native capacity failure when %s", async (_label, input) => { + const run = await nativeCapacityFailure(input as Parameters[0]); + await reportRunFailure(db, run); + expect(mockCaptureRunFailure).toHaveBeenCalledOnce(); + }); + + it("reports a native capacity retry during setup, in all mode, or when its retry cannot be read", async () => { + const run = await nativeCapacityFailure(); + await reportRunFailure(db, run, { phase: "setup" }); + const select = db.select.bind(db); + // The agent lookup succeeds; the retry read fails. + const spy = vi.spyOn(db, "select").mockImplementationOnce(select as typeof db.select) + .mockImplementation(() => { throw new Error("fixture retry read failed"); }); + try { + await reportRunFailure(db, run); + } finally { + spy.mockRestore(); + } + vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all"); + await reportRunFailure(db, run); + expect(mockCaptureRunFailure).toHaveBeenCalledTimes(3); + }); + function providerAccessRun(overrides: Partial = {}) { return buildRun({ error: "ACP agent reported a terminal access failure.\nInvalid API key · Fix external API key", diff --git a/server/src/services/native-runtime/native-provider-overload.integration.test.ts b/server/src/services/native-runtime/native-provider-overload.integration.test.ts index 862a7389bdd..82516e895d2 100644 --- a/server/src/services/native-runtime/native-provider-overload.integration.test.ts +++ b/server/src/services/native-runtime/native-provider-overload.integration.test.ts @@ -7,6 +7,10 @@ import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embed import { CONTROL_PLANE_CONFORMANCE_RESULT } from "../../vendor/paperclip-runner/testing.js"; const provider = vi.hoisted(() => ({ execute: vi.fn() })); +const sentry = vi.hoisted(() => ({ captureRunFailure: vi.fn() })); +vi.mock("../../sentry.js", async (importOriginal) => ({ + ...(await importOriginal()), captureRunFailure: sentry.captureRunFailure, +})); vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({ ...(await importOriginal()), executeNativeSession: provider.execute, })); @@ -20,12 +24,14 @@ import { heartbeatService } from "../heartbeat.js"; import { createPostgresWakeQueueAdapter } from "../../modules/wake-queue/adapters/postgres.js"; import { createReleaseIssueExecution } from "../../modules/wake-queue/application/use-cases.js"; import { buildExecutionContinuation } from "../execution-continuation.js"; +import { waitForPendingRunFailureReports } from "../run-failure-report.js"; describe("durable native provider capacity retry", () => { let temporary: Awaited>; let db: ReturnType; const companyId = randomUUID(), agentId = randomUUID(), otherAgentId = randomUUID(); beforeAll(async () => { + vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "unexpected"); temporary = await startEmbeddedPostgresTestDatabase("native-capacity-retry-"); db = createDb(temporary.connectionString); await db.insert(companies).values({ id: companyId, name: "Capacity regression", issuePrefix: "CAP" }); @@ -34,7 +40,7 @@ describe("durable native provider capacity retry", () => { { id: otherAgentId, companyId, name: "New owner", status: "active", adapterType: "paperclip_runner" }, ]); }); - afterAll(async () => { await temporary?.cleanup(); }); + afterAll(async () => { vi.unstubAllEnvs(); await temporary?.cleanup(); }); async function acceptCapacityFailure(runId: string, issueId: string, mutateOwner = false) { const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); @@ -95,10 +101,15 @@ describe("durable native provider capacity retry", () => { }); for (let failure = 0; failure <= 2; failure++) { const before = Date.now(); + sentry.captureRunFailure.mockClear(); await acceptCapacityFailure(runId, issueId); + await waitForPendingRunFailureReports(); const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.retryOfRunId, runId))); if (failure === 2) { expect(successors).toHaveLength(0); + // Exhaustion ends automatic recovery, so Sentry receives the failure. + expect(sentry.captureRunFailure).toHaveBeenCalledOnce(); + expect(sentry.captureRunFailure.mock.calls[0]![0]).toMatchObject({ runId, runStatus: "failed" }); expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "blocked", unblockDescriptor: { action: expect.stringContaining("Automatic retries exhausted") } }); break; } @@ -107,6 +118,8 @@ describe("durable native provider capacity retry", () => { expect(retry).toMatchObject({ status: "scheduled_retry", scheduledRetryAttempt: failure + 1, scheduledRetryReason: "native_provider_overloaded", error: NATIVE_PROVIDER_OVERLOADED_MESSAGE, contextSnapshot: { issueId, forceFreshSession: true, executionRetryAccounting: { version: 1, failureRetries: failure + 1, maxTurnContinuations: 0 } } }); expect(retry.contextSnapshot).not.toHaveProperty("wakeCommentId"); + // The scheduled retry owns recovery; this failure stays out of Sentry. + expect(sentry.captureRunFailure).not.toHaveBeenCalled(); expect(retry.scheduledRetryAt!.getTime() - before).toBeGreaterThanOrEqual(60_000 * 2 ** failure); expect(retry.scheduledRetryAt!.getTime() - Date.now()).toBeLessThanOrEqual(60_000 * 2 ** failure); const wakes = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.runId, retry.id)); @@ -170,8 +183,12 @@ describe("durable native provider capacity retry", () => { const issueId = randomUUID(), runId = randomUUID(); await db.insert(issues).values({ id: issueId, companyId, title: "Reassigned capacity task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: "capacity-owner" }); await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: "capacity-owner", contextSnapshot: { issueId } }); + sentry.captureRunFailure.mockClear(); await acceptCapacityFailure(runId, issueId, true); + await waitForPendingRunFailureReports(); expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0); + // No retry owns this failure, so it remains reportable. + expect(sentry.captureRunFailure).toHaveBeenCalledOnce(); expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "in_progress", assigneeAgentId: otherAgentId }); expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).filter(wake => wake.payload?.issueId === issueId)).toHaveLength(0); }); diff --git a/server/src/services/run-failure-report.ts b/server/src/services/run-failure-report.ts index c912b19a0d3..71b5494ca3a 100644 --- a/server/src/services/run-failure-report.ts +++ b/server/src/services/run-failure-report.ts @@ -1,4 +1,4 @@ -import { and, eq } from "drizzle-orm"; +import { and, eq, gte, sql } from "drizzle-orm"; import { agents, dotRunnerAssignments, grokBotDispatches, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; import { captureRunFailure, type RunFailureStatus } from "../sentry.js"; import { @@ -15,6 +15,7 @@ import { readConnectionFailure, type ConnectionFailure } from "@paperclipai/adap import { isAiConnectionConfigurationReason } from "./ai-connection-configuration-failure.js"; import { readRunFailureReportingMode, type RunFailureReportingMode } from "../sentry-run-failure-policy.js"; import { RESTRICTED_WORKSPACE_SOURCE_REASON } from "./workspace-source-restriction.js"; +import { NATIVE_PROVIDER_OVERLOADED_CODE, NATIVE_PROVIDER_OVERLOADED_DIAGNOSTIC } from "./native-runtime/native-provider-failure.js"; type HeartbeatRun = typeof heartbeatRuns.$inferSelect; @@ -236,6 +237,32 @@ function isCodexAcpModelCapacityFailure(run: HeartbeatRun, options: RunFailureRe options.phase !== "setup" && run.signal == null; } +/** + * A committed Codex `serverOverloaded` terminal chose the bounded capacity + * retry, and the same status transaction created that retry run. The retry + * owns recovery; a later failure reports itself. Held and exhausted capacity + * failures have no automatic retry and stay reportable. The retry carries the + * task in its context, so the company/context-issue index bounds the read to + * that task's runs created since this one. + */ +async function isScheduledNativeCapacityRetry(db: Db, run: HeartbeatRun, options: RunFailureReportOptions): Promise { + const result = asRecord(run.resultJson); + const diagnostic = asRecord(result?.nativeProviderFailure); + if (run.status !== "failed" || run.runtimeMode !== "native" || run.errorCode !== NATIVE_PROVIDER_OVERLOADED_CODE || + run.nativeIssueId == null || + options.phase === "setup" || run.signal != null || result?.finalizationPhase !== "committed" || + result.finalizationReasonCode !== NATIVE_PROVIDER_OVERLOADED_CODE || + diagnostic?.provider !== NATIVE_PROVIDER_OVERLOADED_DIAGNOSTIC.provider || + diagnostic.category !== NATIVE_PROVIDER_OVERLOADED_DIAGNOSTIC.category) return false; + const [retry] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, run.companyId), + sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${run.nativeIssueId}`, gte(heartbeatRuns.createdAt, run.createdAt), + eq(heartbeatRuns.agentId, run.agentId), + eq(heartbeatRuns.retryOfRunId, run.id), eq(heartbeatRuns.scheduledRetryReason, NATIVE_PROVIDER_OVERLOADED_CODE), + )).limit(1); + return retry !== undefined; +} + const providerAuthErrorCodes = new Set(["claude_auth_required", "acpx_auth_required"]); /** @@ -336,6 +363,11 @@ async function captureTerminalRunFailure( return false; }); if (rejectedExternally) return; + const capacityRetry = await isScheduledNativeCapacityRetry(db, run, options).catch((err) => { + logger.warn({ err, runId: run.id }, "failed to read native capacity retry for run failure report"); + return false; + }); + if (capacityRetry) return; } const taskId = readTaskId(run);