fix(sentry): keep a continued native tool-receipt-limit stop out of Sentry (#22399)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends unexpected run failures to Sentry as run-failure
events. It keeps known, recovered failures local
> - The runner stops a native turn when the turn holds too many tool
results (`semantic_tool_turn_receipt_limit`). The finalizer then
continues the task in a fresh session
> - The runner's stop has no recorded cancellation request. So the
cancelled run keeps `unknown`, unexpected cancellation provenance, and
`reportRunFailure` sends it to Sentry as "Native session cancelled"
> - The continuation owns recovery, so this event is noise. The native
capacity retry had the same problem, and #19051 and #19320 fixed it with
the successor run as the proof
> - This pull request uses the same successor-run proof for the
tool-receipt-limit continuation
> - The benefit is that a continued receipt-limit stop does not make
Sentry noise. Exhausted continuations still report

## Linked Issues or Issue Description

Refs #19051, #19320

**What happened?**
On staging, a native Codex run stopped at its tool-result receipt limit.
The finalizer committed the `semantic_tool_turn_receipt_limit` decision
and scheduled a fresh-session continuation run (`retryOfRunId` = the
stopped run, `scheduledRetryReason` =
`semantic_tool_turn_receipt_limit`). The continuation succeeded and the
task finished. Sentry still got a run-failure event: `Error: Native
session cancelled`, `run_status=cancelled`, cancellation `source:
unknown`, `expected: false`.

**Expected behavior**
A receipt-limit stop that has a scheduled continuation stays local. The
continuation reports itself if it fails. A stop with no continuation
(exhausted, or the user stopped it) stays reportable as before.

**Steps to reproduce**
1. Run a native turn that reaches the semantic-tool receipt limit.
2. Let the finalizer schedule the fresh-session continuation.
3. With `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected`, see that the
cancelled run is sent to Sentry.

## What Changed

- `server/src/services/run-failure-report.ts`:
`isScheduledNativeCapacityRetry` becomes `isScheduledNativeRetry`. It
reads the successor run for the lane that can own the run:
`native_provider_overloaded` for a failed capacity run (unchanged), and
`semantic_tool_turn_receipt_limit` for any other failed or cancelled
native run. The query, task binding, setup-phase, signal, and
read-failure behavior do not change.
- `server/src/services/__tests__/run-failure-report.test.ts`: regression
tests with the persisted shape of the staging run, plus negative cases
(no successor, other retry lane, other task, no native task, legacy,
signal, timed out, setup phase, `all` mode, failed read).
- The shared `buildRun` fixture in `run-failure-report.test.ts` now has
`createdAt`, like a persisted run row. Without it, the wider
scheduled-retry read failed on in-memory test runs and logged a spurious
warning. Real report inputs are full database rows, so production code
does not change for this.
-
`server/src/services/native-runtime/native-tool-receipt-limit.integration.test.ts`:
the real native executor and finalizer path now also asserts the Sentry
outcome. A continued stop sends no report. A cancelled turn without a
continuation still sends one. The fixture run now has `startedAt`, as a
real started run does. Without it, no cancelled-run report can occur,
and the new assertion would prove nothing.

## Verification

- Red before the fix: the two new unit cases failed (`captureRunFailure`
was called), and the integration assertion failed (`expected [...] to
have a length of +0 but got 1`).
- Green after the fix:
- `npx vitest run src/services/__tests__/run-failure-report.test.ts
src/__tests__/run-failure-sentry.test.ts
src/__tests__/run-failure-sentry-real-sdk.test.ts
src/services/native-runtime/native-provider-overload.integration.test.ts
src/services/native-runtime/native-tool-receipt-limit.integration.test.ts
src/services/native-runtime/status-arbiter.test.ts` → 383 passed, 13
skipped.
  - `node scripts/typecheck-layers.mjs` in `server` → no errors.
- `pnpm verify` did not finish in the local sandbox in the time limit.
The only failures were 3 cases in
`server/src/__tests__/archived-nested-repository.test.ts`. They also
fail on unchanged `master` in the same sandbox, and they do not use this
code. CI runs the full suite.

## Risks

- Low risk. The change only removes Sentry events. It does not change
run status, task status, retries, or the user's view.
- A failed or cancelled native run that is not a capacity failure now
does one more indexed read before its report. That read is bounded by
company, task, and agent, and it happens only when a report is about to
be sent.
- A successor with this retry reason exists only when the status arbiter
proved the receipt-limit stop from a persisted runner diagnostic. A user
Stop refuses the continuation, and an exhausted lane creates none, so
these runs still report.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, code search, and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-20 06:37:30 -08:00
1 parent a097949864
commit 5db2ccdae6
3 files changed
+102 -9

No files matched your search

@@ -122,6 +122,7 @@ describeEmbeddedPostgres("reportRunFailure", () => {
errorCode: "process_lost",
nativeIssueId: randomUUID(),
contextSnapshot: null,
createdAt: new Date(),
...overrides,
} as unknown as typeof heartbeatRuns.$inferSelect;
}
@@ -530,6 +531,79 @@ describeEmbeddedPostgres("reportRunFailure", () => {
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(3);
});
async function nativeToolReceiptLimitStop(input: {
run?: Partial<typeof heartbeatRuns.$inferSelect>;
retry?: Partial<typeof heartbeatRuns.$inferInsert> | null;
} = {}) {
await seedCompanyAndAgent({ adapterType: "paperclip_runner" });
const run = buildRun({
status: "cancelled", error: "Native session cancelled", errorCode: "cancelled",
runtimeMode: "native", nativePhase: "committed", exitCode: 1, signal: null,
startedAt: new Date(0), finishedAt: new Date(1000),
resultJson: {
stopReason: "cancelled", finalizationPhase: "committed", finalizationReasonCode: "semantic_tool_turn_receipt_limit",
cancellation: { source: "unknown", expected: false, initiator: { type: "system" },
reason: "Execution stopped without a recorded cancellation request: Native session cancelled",
recordedAt: new Date(1000).toISOString() },
},
...input.run,
});
const [stored] = await db.insert(heartbeatRuns).values(run).returning();
if (input.retry !== null) {
await db.insert(heartbeatRuns).values({
companyId, agentId, invocationSource: "automation", triggerDetail: "system", status: "scheduled_retry",
retryOfRunId: run.id, scheduledRetryReason: "semantic_tool_turn_receipt_limit", scheduledRetryAttempt: 1,
scheduledRetryAt: new Date(Date.now() + 5_000), errorCode: "semantic_tool_turn_receipt_limit",
contextSnapshot: { issueId: run.nativeIssueId, retryOfRunId: run.id }, ...input.retry,
});
}
return stored!;
}
it("keeps a native tool-receipt-limit stop local once its fresh-session continuation is scheduled", async () => {
const run = await nativeToolReceiptLimitStop();
await reportRunFailure(db, run);
await reportRunFailure(db, run, { phase: "execute" });
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
});
it("keeps a failed native run local when the finalizer scheduled its receipt-limit continuation", async () => {
const run = await nativeToolReceiptLimitStop({ run: { status: "failed", errorCode: "native_run_failed",
error: "The provider turn failed", resultJson: { finalizationPhase: "committed" } } });
await reportRunFailure(db, run);
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
});
it.each([
["the continuations are exhausted", { retry: null }],
["the successor belongs to another retry lane", { retry: { scheduledRetryReason: "native_safe_replacement" } }],
["the successor belongs to another task", { retry: { contextSnapshot: { issueId: randomUUID() } } }],
["the run has no native task", { run: { nativeIssueId: null } }],
["the run is legacy", { run: { runtimeMode: "legacy" } }],
["the run has a signal", { run: { signal: "SIGTERM" } }],
["the run timed out", { run: { status: "timed_out" } }],
] as const)("reports a native tool-receipt-limit stop when %s", async (_label, input) => {
const run = await nativeToolReceiptLimitStop(input as Parameters<typeof nativeToolReceiptLimitStop>[0]);
await reportRunFailure(db, run);
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
});
it("reports a native tool-receipt-limit stop during setup, in all mode, or when its continuation cannot be read", async () => {
const run = await nativeToolReceiptLimitStop();
await reportRunFailure(db, run, { phase: "setup" });
const select = db.select.bind(db);
const spy = vi.spyOn(db, "select").mockImplementationOnce(select as typeof db.select)
.mockImplementation(() => { throw new Error("fixture continuation read failed"); });
try {
await reportRunFailure(db, run);
} finally {
spy.mockRestore();
}
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
await reportRunFailure(db, run);
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(3);
});
function providerAccessRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
return buildRun({
error: "ACP agent reported a terminal access failure.\nInvalid API key · Fix external API key",
@@ -8,6 +8,10 @@ import { CONTROL_PLANE_CONFORMANCE_RESULT } from "../../vendor/paperclip-runner/
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
const receiptLimitRead = vi.hoisted(() => ({ after: null as null | (() => Promise<unknown>) }));
const sentry = vi.hoisted(() => ({ captureRunFailure: vi.fn((_event: { runId: string }) => true) }));
vi.mock("../../sentry.js", async (importOriginal) => ({
...(await importOriginal<typeof import("../../sentry.js")>()), captureRunFailure: sentry.captureRunFailure,
}));
vi.mock("./native-tool-receipt-limit.js", async (importOriginal) => {
const original = await importOriginal<typeof import("./native-tool-receipt-limit.js")>();
return { ...original, readPersistedNativeToolReceiptLimitStop: async (...args: Parameters<typeof original.readPersistedNativeToolReceiptLimitStop>) => {
@@ -33,24 +37,30 @@ import { createReleaseIssueExecution } from "../../modules/wake-queue/applicatio
import { taskInterruptionPresentation } from "../task-interruption-presentation.js";
import { materializeNativeToolReceiptLimitContinuation } from "./native-provider-capacity-retry.js";
import { NativeStatusRaceError } from "./status-decision-committer.js";
import { waitForPendingRunFailureReports } from "../run-failure-report.js";
describe("native turn stopped at its tool receipt limit", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
const companyId = randomUUID(), agentId = randomUUID();
beforeAll(async () => {
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "unexpected");
temporary = await startEmbeddedPostgresTestDatabase("native-tool-receipt-limit-");
db = createDb(temporary.connectionString);
await db.insert(companies).values({ id: companyId, name: "Receipt limit regression", issuePrefix: "RCL" });
await db.insert(agents).values({ id: agentId, companyId, name: "Receipt worker", status: "active", adapterType: "paperclip_runner", adapterConfig: { provider: "opencode", model: "test-model" } });
});
afterAll(async () => { await temporary?.cleanup(); });
afterAll(async () => { vi.unstubAllEnvs(); await temporary?.cleanup(); });
async function sentryReports(runId: string) {
await waitForPendingRunFailureReports();
return sentry.captureRunFailure.mock.calls.filter(([event]) => event.runId === runId);
}
let finalTurnId: string | null = null;
async function stopAtReceiptLimit(runId: string, issueId: string, diagnosticCode: string, beforeFinalize?: () => Promise<unknown>,
diagnosticTurnId?: string) {
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
const [run] = await db.update(heartbeatRuns).set({ status: "running", runtimeMode: "native", nativeIssueId: issueId, error: null, errorCode: null }).where(eq(heartbeatRuns.id, runId)).returning();
const [run] = await db.update(heartbeatRuns).set({ status: "running", startedAt: new Date(), runtimeMode: "native", nativeIssueId: issueId, error: null, errorCode: null }).where(eq(heartbeatRuns.id, runId)).returning();
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
@@ -115,6 +125,7 @@ describe("native turn stopped at its tool receipt limit", () => {
contextSnapshot: { issueId, forceFreshSession: true, livenessContinuationSourceRunId: runId,
livenessContinuationReason: NATIVE_TOOL_RECEIPT_LIMIT_CODE, livenessContinuationInstruction: NATIVE_TOOL_RECEIPT_LIMIT_HINT } });
expect(retry!.contextSnapshot).not.toHaveProperty("wakeCommentId");
expect(await sentryReports(runId)).toHaveLength(0);
expect(retry!.scheduledRetryAt!.getTime() - Date.now()).toBeLessThanOrEqual(NATIVE_TOOL_RECEIPT_LIMIT_CONTINUATION_DELAY_MS);
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
expect(decision).toMatchObject({ reasonCode: NATIVE_TOOL_RECEIPT_LIMIT_CODE });
@@ -170,6 +181,7 @@ describe("native turn stopped at its tool receipt limit", () => {
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
expect(decision).toMatchObject({ reasonCode: "cancellation_run_only" });
expect(await sentryReports(runId)).toEqual([[expect.objectContaining({ runStatus: "cancelled" })]]);
});
it("refuses to create the continuation under the run lock once a person's Stop is saved", async () => {
+14 -7
View File
@@ -20,6 +20,7 @@ import { RESTRICTED_WORKSPACE_SOURCE_REASON } from "./workspace-source-restricti
import { PLUGIN_RUNNER_AI_CONNECTION_REQUIRED } from "./plugin-runner-ai-connection.js";
import { PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE } from "./project-storage-environment-reason.js";
import { NATIVE_PROVIDER_OVERLOADED_CODE } from "./native-runtime/native-provider-failure.js";
import { NATIVE_TOOL_RECEIPT_LIMIT_CODE } from "./native-runtime/native-tool-receipt-limit.js";
import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js";
import { REVIEWED_CHAT_BINDING_NOT_READY } from "./reviewed-chat-binding-diagnostics.js";
import { readProviderFailure } from "@paperclipai/shared";
@@ -313,14 +314,20 @@ function isCodexAcpModelCapacityFailure(run: HeartbeatRun, options: RunFailureRe
* carries the task in its context, so the company/context-issue index bounds
* the read to that task's runs created since this one.
*/
async function isScheduledNativeCapacityRetry(db: Db, run: HeartbeatRun, options: RunFailureReportOptions): Promise<boolean> {
if (run.status !== "failed" || run.runtimeMode !== "native" || run.errorCode !== NATIVE_PROVIDER_OVERLOADED_CODE ||
run.nativeIssueId == null || options.phase === "setup" || run.signal != null) return false;
function scheduledNativeRetryReason(run: HeartbeatRun, options: RunFailureReportOptions): string | null {
if (run.runtimeMode !== "native" || run.nativeIssueId == null || options.phase === "setup" || run.signal != null) return null;
if (run.status === "failed" && run.errorCode === NATIVE_PROVIDER_OVERLOADED_CODE) return NATIVE_PROVIDER_OVERLOADED_CODE;
return run.status === "failed" || run.status === "cancelled" ? NATIVE_TOOL_RECEIPT_LIMIT_CODE : null;
}
async function isScheduledNativeRetry(db: Db, run: HeartbeatRun, options: RunFailureReportOptions): Promise<boolean> {
const reason = scheduledNativeRetryReason(run, options);
if (!reason) return false;
const [retry] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, run.companyId),
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${run.nativeIssueId}`, gte(heartbeatRuns.createdAt, run.createdAt),
eq(heartbeatRuns.agentId, run.agentId),
eq(heartbeatRuns.retryOfRunId, run.id), eq(heartbeatRuns.scheduledRetryReason, NATIVE_PROVIDER_OVERLOADED_CODE),
eq(heartbeatRuns.retryOfRunId, run.id), eq(heartbeatRuns.scheduledRetryReason, reason),
)).limit(1);
return retry !== undefined;
}
@@ -446,11 +453,11 @@ async function captureTerminalRunFailure(
return false;
});
if (rejectedExternally) return;
const capacityRetry = await isScheduledNativeCapacityRetry(db, run, options).catch((err) => {
logger.warn({ err, runId: run.id }, "failed to read native capacity retry for run failure report");
const scheduledRetry = await isScheduledNativeRetry(db, run, options).catch((err) => {
logger.warn({ err, runId: run.id }, "failed to read native scheduled retry for run failure report");
return false;
});
if (capacityRetry) return;
if (scheduledRetry) return;
const restartEndedUsageLimit = await isRestartEndedNativeUsageLimit(db, run, options).catch((err) => {
logger.warn({ err, runId: run.id }, "failed to read native usage-limit terminal for run failure report");
return false;