fix(sentry): keep a live-heartbeat native capacity retry out of Sentry (#19320)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends unexpected run failures to Sentry, and it keeps
known, recovered failures local
> - A native Codex "model at capacity" failure gets an automatic retry.
That failure must stay out of Sentry while its retry owns recovery
(#18295, #19051)
> - The check reads the `nativeProviderFailure` diagnostic from the
failed run's `resultJson`. On the live heartbeat path, the failed-status
write replaces `resultJson` with the adapter's copy, and that copy has
no diagnostic
> - So every live-path capacity failure with a scheduled retry still
goes to Sentry. Only the reconciliation path, which the tests cover,
kept the failure local
> - This pull request removes the diagnostic check. The scheduled retry
run is the proof
> - The benefit is that recovered capacity failures no longer create
Sentry noise on the normal live path

## Linked Issues or Issue Description

Refs #18295
Refs #19051

**What happened?**
A native Codex run failed with `native_provider_overloaded`. The
committed status decision scheduled the automatic capacity retry, and
the retry run existed before the report. The run failure still went to
Sentry. In a staging sample, every live-path capacity failure since
#19051 was reported (1 of 1).

**Expected behavior**
When the `native_provider_overloaded` retry run exists for the same
task, the failed run does not go to Sentry. The retry reports its own
outcome.

**Steps to reproduce**
1. Run a native Codex agent through the live heartbeat path
(`finalizeNativeRun` without `projectRunStatus`).
2. Let the provider return `serverOverloaded`. The status decision
schedules the capacity retry.
3. The heartbeat writes the terminal `failed` status with the adapter's
`resultJson`, and `emitTerminalAgentTaskRun` calls `reportRunFailure`
with that row.
4. The row has no `nativeProviderFailure`, so
`isScheduledNativeCapacityRetry` returns false and the event goes to
Sentry.

**Relevant logs or output**
The failed run row later shows `nativeProviderFailure:
codex/server_overloaded`, because a later reconciliation write restores
it. The row that the heartbeat write reports does not have the field.
The same overwrite also explains the missing `finalizationReasonCode`
that #19051 fixed.

## What Changed

- `server/src/services/run-failure-report.ts`:
`isScheduledNativeCapacityRetry` no longer requires
`resultJson.nativeProviderFailure`. The run must still be native and
failed, carry `native_provider_overloaded`, have a native task, have no
signal, and not be in setup. The same-task successor with
`scheduledRetryReason: native_provider_overloaded` is still required.
- The successor is enough proof. Only the committed capacity status
decision creates that successor, and only `forTerminal` in
`native-provider-failure.ts` produces `native_provider_overloaded`
together with the typed `codex`/`server_overloaded` diagnostic.
- `run-failure-report.test.ts`: a new case keeps the failure local when
the row has only the adapter's `resultJson` (no diagnostic, no
finalization fields) and the successor exists. The "diagnostic is
missing" case now reports only when no retry run exists. The case for a
non-capacity diagnostic with a capacity successor is removed, because
that state cannot occur.
- `doc/observability.md`: the native capacity paragraph now says that
the successor is the proof and that the run's own `resultJson` is not
required.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/__tests__/run-failure-report.test.ts`: 284/284 pass with
this change.
- With the `master` version of `run-failure-report.ts`, the new case
fails: 1 failed, 283 passed. So the test catches the bug.
- `native-provider-failure.test.ts`: 57/57 pass.
- `native-provider-overload.integration.test.ts` did not load in my
sandbox, because `@paperclipai/plugin-sdk` was not built. CI must run
it.

## Risks

- Low risk. The change only removes one condition from a Sentry filter.
- These failures still report: no retry run, exhausted or held retries,
other retry lanes, a successor for a different task, legacy runs,
signals, timeouts, setup failures, and unreadable retry evidence.
- The change does not affect run status, retries, recovery, or the `all`
reporting mode.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use
(shell, repository reads, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Devin FoleyandClaude Opus 5.5 authored and GitHub committed 2026-11-05 12:04:33 -08:00
1 parent 62a5aa7705
commit 8aa28940bf
3 files changed
+16 -16

No files matched your search

+6 -5
View File
@@ -1134,11 +1134,12 @@ untyped capacity text, and Codex CLI transient upstream failures remain reportab
The run keeps its failed state and its terminal diagnostic.
A native Codex model capacity failure stays local only while its automatic
retry owns recovery. The run must carry `native_provider_overloaded` and the
closed `codex`/`server_overloaded` diagnostic. The proof of the retry is the
`native_provider_overloaded` successor run for the same task, which only the
committed capacity status decision creates. The run's own finalization fields
are not required: a late terminal write can replace them with an earlier result.
retry owns recovery. The run must carry `native_provider_overloaded`. The proof
of the retry is the `native_provider_overloaded` successor run for the same
task, which only the committed capacity status decision creates, and only for
the typed `codex`/`server_overloaded` diagnostic. The run's own `resultJson` is
not required: the live heartbeat's failed-status write replaces it with the
adapter's copy, which lacks the finalizer's diagnostic and finalization fields.
Exhausted or held capacity failures, other retry lanes, setup failures, signals,
and unreadable retry evidence remain reportable. A failed retry reports its own
outcome.
@@ -470,9 +470,11 @@ describeEmbeddedPostgres("reportRunFailure", () => {
it.each([
["no finalization fields", { finalizationPhase: undefined, finalizationReasonCode: undefined }],
["an earlier finalization phase", { finalizationPhase: "retryable_failure", finalizationReasonCode: undefined }],
["only the adapter's resultJson", { nativeProviderFailure: undefined, finalizationPhase: undefined, finalizationReasonCode: undefined }],
] as const)("keeps a scheduled native capacity retry local when the run row has %s", async (_label, resultJson) => {
// A late terminal write can replace the finalizer's fields with an earlier
// resultJson. The successor created by the committed decision still proves it.
// The live heartbeat's failed-status write replaces the finalizer's
// resultJson with the adapter's copy. The successor created by the
// committed decision still proves the scheduled retry.
const run = await nativeCapacityFailure({ resultJson });
await reportRunFailure(db, run);
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
@@ -486,8 +488,7 @@ describeEmbeddedPostgres("reportRunFailure", () => {
["the successor belongs to another task", { retry: { contextSnapshot: { issueId: randomUUID() } } }],
["the run has no native task", { run: { nativeIssueId: null } }],
["finalization has no fields and no retry run exists", { resultJson: { finalizationPhase: undefined, finalizationReasonCode: undefined }, retry: null }],
["the diagnostic is not typed capacity evidence", { resultJson: { nativeProviderFailure: { provider: "codex", category: "turn_failed" } } }],
["the diagnostic is missing", { resultJson: { nativeProviderFailure: undefined } }],
["the diagnostic is missing and no retry run exists", { resultJson: { nativeProviderFailure: undefined }, retry: null }],
["the run is legacy", { run: { runtimeMode: "legacy" } }],
["the run has a signal", { run: { signal: "SIGTERM" } }],
["the run timed out", { run: { status: "timed_out" } }],
+5 -7
View File
@@ -15,7 +15,7 @@ import { readConnectionFailure, type ConnectionFailure } from "@paperclipai/adap
import { isAiConnectionConfigurationReason } from "./ai-connection-configuration-failure.js";
import { readRunFailureReportingMode, type RunFailureReportingMode } from "../sentry-run-failure-policy.js";
import { RESTRICTED_WORKSPACE_SOURCE_REASON } from "./workspace-source-restriction.js";
import { NATIVE_PROVIDER_OVERLOADED_CODE, NATIVE_PROVIDER_OVERLOADED_DIAGNOSTIC } from "./native-runtime/native-provider-failure.js";
import { NATIVE_PROVIDER_OVERLOADED_CODE } from "./native-runtime/native-provider-failure.js";
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
@@ -240,19 +240,17 @@ function isCodexAcpModelCapacityFailure(run: HeartbeatRun, options: RunFailureRe
/**
* A Codex `serverOverloaded` terminal chose the bounded capacity retry. Only
* that committed status decision creates a successor with this retry reason,
* so the successor row is the proof. The run's own finalization fields are
* not: a late terminal write can replace them with an earlier `resultJson`.
* so the successor row is the proof. The run's own `resultJson` is not: the
* live heartbeat's failed-status write replaces it with the adapter's copy,
* which lacks the finalizer's diagnostic and finalization fields.
* The retry owns recovery; a later failure reports itself. Held and exhausted
* capacity failures have no automatic retry and stay reportable. The retry
* carries the task in its context, so the company/context-issue index bounds
* the read to that task's runs created since this one.
*/
async function isScheduledNativeCapacityRetry(db: Db, run: HeartbeatRun, options: RunFailureReportOptions): Promise<boolean> {
const diagnostic = asRecord(asRecord(run.resultJson)?.nativeProviderFailure);
if (run.status !== "failed" || run.runtimeMode !== "native" || run.errorCode !== NATIVE_PROVIDER_OVERLOADED_CODE ||
run.nativeIssueId == null || options.phase === "setup" || run.signal != null ||
diagnostic?.provider !== NATIVE_PROVIDER_OVERLOADED_DIAGNOSTIC.provider ||
diagnostic.category !== NATIVE_PROVIDER_OVERLOADED_DIAGNOSTIC.category) return false;
run.nativeIssueId == null || options.phase === "setup" || run.signal != null) return false;
const [retry] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, run.companyId),
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${run.nativeIssueId}`, gte(heartbeatRuns.createdAt, run.createdAt),