From 5db2ccdae63bacafa0099f5a7e9660e5340ad83e Mon Sep 17 00:00:00 2001 From: paperclipbugbot Date: Fri, 20 Nov 2026 06:37:30 -0800 Subject: [PATCH] fix(sentry): keep a continued native tool-receipt-limit stop out of Sentry (#22399) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server sends unexpected run failures to Sentry as run-failure events. It keeps known, recovered failures local > - The runner stops a native turn when the turn holds too many tool results (`semantic_tool_turn_receipt_limit`). The finalizer then continues the task in a fresh session > - The runner's stop has no recorded cancellation request. So the cancelled run keeps `unknown`, unexpected cancellation provenance, and `reportRunFailure` sends it to Sentry as "Native session cancelled" > - The continuation owns recovery, so this event is noise. The native capacity retry had the same problem, and #19051 and #19320 fixed it with the successor run as the proof > - This pull request uses the same successor-run proof for the tool-receipt-limit continuation > - The benefit is that a continued receipt-limit stop does not make Sentry noise. Exhausted continuations still report ## Linked Issues or Issue Description Refs #19051, #19320 **What happened?** On staging, a native Codex run stopped at its tool-result receipt limit. The finalizer committed the `semantic_tool_turn_receipt_limit` decision and scheduled a fresh-session continuation run (`retryOfRunId` = the stopped run, `scheduledRetryReason` = `semantic_tool_turn_receipt_limit`). The continuation succeeded and the task finished. Sentry still got a run-failure event: `Error: Native session cancelled`, `run_status=cancelled`, cancellation `source: unknown`, `expected: false`. **Expected behavior** A receipt-limit stop that has a scheduled continuation stays local. The continuation reports itself if it fails. A stop with no continuation (exhausted, or the user stopped it) stays reportable as before. **Steps to reproduce** 1. Run a native turn that reaches the semantic-tool receipt limit. 2. Let the finalizer schedule the fresh-session continuation. 3. With `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected`, see that the cancelled run is sent to Sentry. ## What Changed - `server/src/services/run-failure-report.ts`: `isScheduledNativeCapacityRetry` becomes `isScheduledNativeRetry`. It reads the successor run for the lane that can own the run: `native_provider_overloaded` for a failed capacity run (unchanged), and `semantic_tool_turn_receipt_limit` for any other failed or cancelled native run. The query, task binding, setup-phase, signal, and read-failure behavior do not change. - `server/src/services/__tests__/run-failure-report.test.ts`: regression tests with the persisted shape of the staging run, plus negative cases (no successor, other retry lane, other task, no native task, legacy, signal, timed out, setup phase, `all` mode, failed read). - The shared `buildRun` fixture in `run-failure-report.test.ts` now has `createdAt`, like a persisted run row. Without it, the wider scheduled-retry read failed on in-memory test runs and logged a spurious warning. Real report inputs are full database rows, so production code does not change for this. - `server/src/services/native-runtime/native-tool-receipt-limit.integration.test.ts`: the real native executor and finalizer path now also asserts the Sentry outcome. A continued stop sends no report. A cancelled turn without a continuation still sends one. The fixture run now has `startedAt`, as a real started run does. Without it, no cancelled-run report can occur, and the new assertion would prove nothing. ## Verification - Red before the fix: the two new unit cases failed (`captureRunFailure` was called), and the integration assertion failed (`expected [...] to have a length of +0 but got 1`). - Green after the fix: - `npx vitest run src/services/__tests__/run-failure-report.test.ts src/__tests__/run-failure-sentry.test.ts src/__tests__/run-failure-sentry-real-sdk.test.ts src/services/native-runtime/native-provider-overload.integration.test.ts src/services/native-runtime/native-tool-receipt-limit.integration.test.ts src/services/native-runtime/status-arbiter.test.ts` โ†’ 383 passed, 13 skipped. - `node scripts/typecheck-layers.mjs` in `server` โ†’ no errors. - `pnpm verify` did not finish in the local sandbox in the time limit. The only failures were 3 cases in `server/src/__tests__/archived-nested-repository.test.ts`. They also fail on unchanged `master` in the same sandbox, and they do not use this code. CI runs the full suite. ## Risks - Low risk. The change only removes Sentry events. It does not change run status, task status, retries, or the user's view. - A failed or cancelled native run that is not a capacity failure now does one more indexed read before its report. That read is bounded by company, task, and agent, and it happens only when a report is about to be sent. - A successor with this retry reason exists only when the status arbiter proved the receipt-limit stop from a persisted runner diagnostic. A user Stop refuses the continuation, and an exhausted lane creates none, so these runs still report. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, code search, and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge ๐Ÿค– Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- .../__tests__/run-failure-report.test.ts | 74 +++++++++++++++++++ ...ive-tool-receipt-limit.integration.test.ts | 16 +++- server/src/services/run-failure-report.ts | 21 ++++-- 3 files changed, 102 insertions(+), 9 deletions(-) diff --git a/server/src/services/__tests__/run-failure-report.test.ts b/server/src/services/__tests__/run-failure-report.test.ts index 01a7dae4263..6c203e2549d 100644 --- a/server/src/services/__tests__/run-failure-report.test.ts +++ b/server/src/services/__tests__/run-failure-report.test.ts @@ -122,6 +122,7 @@ describeEmbeddedPostgres("reportRunFailure", () => { errorCode: "process_lost", nativeIssueId: randomUUID(), contextSnapshot: null, + createdAt: new Date(), ...overrides, } as unknown as typeof heartbeatRuns.$inferSelect; } @@ -530,6 +531,79 @@ describeEmbeddedPostgres("reportRunFailure", () => { expect(mockCaptureRunFailure).toHaveBeenCalledTimes(3); }); + async function nativeToolReceiptLimitStop(input: { + run?: Partial; + retry?: Partial | null; + } = {}) { + await seedCompanyAndAgent({ adapterType: "paperclip_runner" }); + const run = buildRun({ + status: "cancelled", error: "Native session cancelled", errorCode: "cancelled", + runtimeMode: "native", nativePhase: "committed", exitCode: 1, signal: null, + startedAt: new Date(0), finishedAt: new Date(1000), + resultJson: { + stopReason: "cancelled", finalizationPhase: "committed", finalizationReasonCode: "semantic_tool_turn_receipt_limit", + cancellation: { source: "unknown", expected: false, initiator: { type: "system" }, + reason: "Execution stopped without a recorded cancellation request: Native session cancelled", + recordedAt: new Date(1000).toISOString() }, + }, + ...input.run, + }); + const [stored] = await db.insert(heartbeatRuns).values(run).returning(); + if (input.retry !== null) { + await db.insert(heartbeatRuns).values({ + companyId, agentId, invocationSource: "automation", triggerDetail: "system", status: "scheduled_retry", + retryOfRunId: run.id, scheduledRetryReason: "semantic_tool_turn_receipt_limit", scheduledRetryAttempt: 1, + scheduledRetryAt: new Date(Date.now() + 5_000), errorCode: "semantic_tool_turn_receipt_limit", + contextSnapshot: { issueId: run.nativeIssueId, retryOfRunId: run.id }, ...input.retry, + }); + } + return stored!; + } + + it("keeps a native tool-receipt-limit stop local once its fresh-session continuation is scheduled", async () => { + const run = await nativeToolReceiptLimitStop(); + await reportRunFailure(db, run); + await reportRunFailure(db, run, { phase: "execute" }); + expect(mockCaptureRunFailure).not.toHaveBeenCalled(); + }); + + it("keeps a failed native run local when the finalizer scheduled its receipt-limit continuation", async () => { + const run = await nativeToolReceiptLimitStop({ run: { status: "failed", errorCode: "native_run_failed", + error: "The provider turn failed", resultJson: { finalizationPhase: "committed" } } }); + await reportRunFailure(db, run); + expect(mockCaptureRunFailure).not.toHaveBeenCalled(); + }); + + it.each([ + ["the continuations are exhausted", { retry: null }], + ["the successor belongs to another retry lane", { retry: { scheduledRetryReason: "native_safe_replacement" } }], + ["the successor belongs to another task", { retry: { contextSnapshot: { issueId: randomUUID() } } }], + ["the run has no native task", { run: { nativeIssueId: null } }], + ["the run is legacy", { run: { runtimeMode: "legacy" } }], + ["the run has a signal", { run: { signal: "SIGTERM" } }], + ["the run timed out", { run: { status: "timed_out" } }], + ] as const)("reports a native tool-receipt-limit stop when %s", async (_label, input) => { + const run = await nativeToolReceiptLimitStop(input as Parameters[0]); + await reportRunFailure(db, run); + expect(mockCaptureRunFailure).toHaveBeenCalledOnce(); + }); + + it("reports a native tool-receipt-limit stop during setup, in all mode, or when its continuation cannot be read", async () => { + const run = await nativeToolReceiptLimitStop(); + await reportRunFailure(db, run, { phase: "setup" }); + const select = db.select.bind(db); + const spy = vi.spyOn(db, "select").mockImplementationOnce(select as typeof db.select) + .mockImplementation(() => { throw new Error("fixture continuation read failed"); }); + try { + await reportRunFailure(db, run); + } finally { + spy.mockRestore(); + } + vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all"); + await reportRunFailure(db, run); + expect(mockCaptureRunFailure).toHaveBeenCalledTimes(3); + }); + function providerAccessRun(overrides: Partial = {}) { return buildRun({ error: "ACP agent reported a terminal access failure.\nInvalid API key ยท Fix external API key", diff --git a/server/src/services/native-runtime/native-tool-receipt-limit.integration.test.ts b/server/src/services/native-runtime/native-tool-receipt-limit.integration.test.ts index 594a1ee0262..8a37785b7c3 100644 --- a/server/src/services/native-runtime/native-tool-receipt-limit.integration.test.ts +++ b/server/src/services/native-runtime/native-tool-receipt-limit.integration.test.ts @@ -8,6 +8,10 @@ import { CONTROL_PLANE_CONFORMANCE_RESULT } from "../../vendor/paperclip-runner/ const provider = vi.hoisted(() => ({ execute: vi.fn() })); const receiptLimitRead = vi.hoisted(() => ({ after: null as null | (() => Promise) })); +const sentry = vi.hoisted(() => ({ captureRunFailure: vi.fn((_event: { runId: string }) => true) })); +vi.mock("../../sentry.js", async (importOriginal) => ({ + ...(await importOriginal()), captureRunFailure: sentry.captureRunFailure, +})); vi.mock("./native-tool-receipt-limit.js", async (importOriginal) => { const original = await importOriginal(); return { ...original, readPersistedNativeToolReceiptLimitStop: async (...args: Parameters) => { @@ -33,24 +37,30 @@ import { createReleaseIssueExecution } from "../../modules/wake-queue/applicatio import { taskInterruptionPresentation } from "../task-interruption-presentation.js"; import { materializeNativeToolReceiptLimitContinuation } from "./native-provider-capacity-retry.js"; import { NativeStatusRaceError } from "./status-decision-committer.js"; +import { waitForPendingRunFailureReports } from "../run-failure-report.js"; describe("native turn stopped at its tool receipt limit", () => { let temporary: Awaited>; let db: ReturnType; const companyId = randomUUID(), agentId = randomUUID(); beforeAll(async () => { + vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "unexpected"); temporary = await startEmbeddedPostgresTestDatabase("native-tool-receipt-limit-"); db = createDb(temporary.connectionString); await db.insert(companies).values({ id: companyId, name: "Receipt limit regression", issuePrefix: "RCL" }); await db.insert(agents).values({ id: agentId, companyId, name: "Receipt worker", status: "active", adapterType: "paperclip_runner", adapterConfig: { provider: "opencode", model: "test-model" } }); }); - afterAll(async () => { await temporary?.cleanup(); }); + afterAll(async () => { vi.unstubAllEnvs(); await temporary?.cleanup(); }); + async function sentryReports(runId: string) { + await waitForPendingRunFailureReports(); + return sentry.captureRunFailure.mock.calls.filter(([event]) => event.runId === runId); + } let finalTurnId: string | null = null; async function stopAtReceiptLimit(runId: string, issueId: string, diagnosticCode: string, beforeFinalize?: () => Promise, diagnosticTurnId?: string) { const [issue] = await db.select().from(issues).where(eq(issues.id, issueId)); - const [run] = await db.update(heartbeatRuns).set({ status: "running", runtimeMode: "native", nativeIssueId: issueId, error: null, errorCode: null }).where(eq(heartbeatRuns.id, runId)).returning(); + const [run] = await db.update(heartbeatRuns).set({ status: "running", startedAt: new Date(), runtimeMode: "native", nativeIssueId: issueId, error: null, errorCode: null }).where(eq(heartbeatRuns.id, runId)).returning(); await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() }); const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId)); @@ -115,6 +125,7 @@ describe("native turn stopped at its tool receipt limit", () => { contextSnapshot: { issueId, forceFreshSession: true, livenessContinuationSourceRunId: runId, livenessContinuationReason: NATIVE_TOOL_RECEIPT_LIMIT_CODE, livenessContinuationInstruction: NATIVE_TOOL_RECEIPT_LIMIT_HINT } }); expect(retry!.contextSnapshot).not.toHaveProperty("wakeCommentId"); + expect(await sentryReports(runId)).toHaveLength(0); expect(retry!.scheduledRetryAt!.getTime() - Date.now()).toBeLessThanOrEqual(NATIVE_TOOL_RECEIPT_LIMIT_CONTINUATION_DELAY_MS); const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId)); expect(decision).toMatchObject({ reasonCode: NATIVE_TOOL_RECEIPT_LIMIT_CODE }); @@ -170,6 +181,7 @@ describe("native turn stopped at its tool receipt limit", () => { expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0); const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId)); expect(decision).toMatchObject({ reasonCode: "cancellation_run_only" }); + expect(await sentryReports(runId)).toEqual([[expect.objectContaining({ runStatus: "cancelled" })]]); }); it("refuses to create the continuation under the run lock once a person's Stop is saved", async () => { diff --git a/server/src/services/run-failure-report.ts b/server/src/services/run-failure-report.ts index 50d1a074ded..a514a340bd8 100644 --- a/server/src/services/run-failure-report.ts +++ b/server/src/services/run-failure-report.ts @@ -20,6 +20,7 @@ import { RESTRICTED_WORKSPACE_SOURCE_REASON } from "./workspace-source-restricti import { PLUGIN_RUNNER_AI_CONNECTION_REQUIRED } from "./plugin-runner-ai-connection.js"; import { PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE } from "./project-storage-environment-reason.js"; import { NATIVE_PROVIDER_OVERLOADED_CODE } from "./native-runtime/native-provider-failure.js"; +import { NATIVE_TOOL_RECEIPT_LIMIT_CODE } from "./native-runtime/native-tool-receipt-limit.js"; import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js"; import { REVIEWED_CHAT_BINDING_NOT_READY } from "./reviewed-chat-binding-diagnostics.js"; import { readProviderFailure } from "@paperclipai/shared"; @@ -313,14 +314,20 @@ function isCodexAcpModelCapacityFailure(run: HeartbeatRun, options: RunFailureRe * carries the task in its context, so the company/context-issue index bounds * the read to that task's runs created since this one. */ -async function isScheduledNativeCapacityRetry(db: Db, run: HeartbeatRun, options: RunFailureReportOptions): Promise { - if (run.status !== "failed" || run.runtimeMode !== "native" || run.errorCode !== NATIVE_PROVIDER_OVERLOADED_CODE || - run.nativeIssueId == null || options.phase === "setup" || run.signal != null) return false; +function scheduledNativeRetryReason(run: HeartbeatRun, options: RunFailureReportOptions): string | null { + if (run.runtimeMode !== "native" || run.nativeIssueId == null || options.phase === "setup" || run.signal != null) return null; + if (run.status === "failed" && run.errorCode === NATIVE_PROVIDER_OVERLOADED_CODE) return NATIVE_PROVIDER_OVERLOADED_CODE; + return run.status === "failed" || run.status === "cancelled" ? NATIVE_TOOL_RECEIPT_LIMIT_CODE : null; +} + +async function isScheduledNativeRetry(db: Db, run: HeartbeatRun, options: RunFailureReportOptions): Promise { + const reason = scheduledNativeRetryReason(run, options); + if (!reason) return false; const [retry] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( eq(heartbeatRuns.companyId, run.companyId), sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${run.nativeIssueId}`, gte(heartbeatRuns.createdAt, run.createdAt), eq(heartbeatRuns.agentId, run.agentId), - eq(heartbeatRuns.retryOfRunId, run.id), eq(heartbeatRuns.scheduledRetryReason, NATIVE_PROVIDER_OVERLOADED_CODE), + eq(heartbeatRuns.retryOfRunId, run.id), eq(heartbeatRuns.scheduledRetryReason, reason), )).limit(1); return retry !== undefined; } @@ -446,11 +453,11 @@ async function captureTerminalRunFailure( return false; }); if (rejectedExternally) return; - const capacityRetry = await isScheduledNativeCapacityRetry(db, run, options).catch((err) => { - logger.warn({ err, runId: run.id }, "failed to read native capacity retry for run failure report"); + const scheduledRetry = await isScheduledNativeRetry(db, run, options).catch((err) => { + logger.warn({ err, runId: run.id }, "failed to read native scheduled retry for run failure report"); return false; }); - if (capacityRetry) return; + if (scheduledRetry) return; const restartEndedUsageLimit = await isRestartEndedNativeUsageLimit(db, run, options).catch((err) => { logger.warn({ err, runId: run.id }, "failed to read native usage-limit terminal for run failure report"); return false;