mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
fix(sentry): keep a continued native tool-receipt-limit stop out of Sentry (#22399)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server sends unexpected run failures to Sentry as run-failure events. It keeps known, recovered failures local > - The runner stops a native turn when the turn holds too many tool results (`semantic_tool_turn_receipt_limit`). The finalizer then continues the task in a fresh session > - The runner's stop has no recorded cancellation request. So the cancelled run keeps `unknown`, unexpected cancellation provenance, and `reportRunFailure` sends it to Sentry as "Native session cancelled" > - The continuation owns recovery, so this event is noise. The native capacity retry had the same problem, and #19051 and #19320 fixed it with the successor run as the proof > - This pull request uses the same successor-run proof for the tool-receipt-limit continuation > - The benefit is that a continued receipt-limit stop does not make Sentry noise. Exhausted continuations still report ## Linked Issues or Issue Description Refs #19051, #19320 **What happened?** On staging, a native Codex run stopped at its tool-result receipt limit. The finalizer committed the `semantic_tool_turn_receipt_limit` decision and scheduled a fresh-session continuation run (`retryOfRunId` = the stopped run, `scheduledRetryReason` = `semantic_tool_turn_receipt_limit`). The continuation succeeded and the task finished. Sentry still got a run-failure event: `Error: Native session cancelled`, `run_status=cancelled`, cancellation `source: unknown`, `expected: false`. **Expected behavior** A receipt-limit stop that has a scheduled continuation stays local. The continuation reports itself if it fails. A stop with no continuation (exhausted, or the user stopped it) stays reportable as before. **Steps to reproduce** 1. Run a native turn that reaches the semantic-tool receipt limit. 2. Let the finalizer schedule the fresh-session continuation. 3. With `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected`, see that the cancelled run is sent to Sentry. ## What Changed - `server/src/services/run-failure-report.ts`: `isScheduledNativeCapacityRetry` becomes `isScheduledNativeRetry`. It reads the successor run for the lane that can own the run: `native_provider_overloaded` for a failed capacity run (unchanged), and `semantic_tool_turn_receipt_limit` for any other failed or cancelled native run. The query, task binding, setup-phase, signal, and read-failure behavior do not change. - `server/src/services/__tests__/run-failure-report.test.ts`: regression tests with the persisted shape of the staging run, plus negative cases (no successor, other retry lane, other task, no native task, legacy, signal, timed out, setup phase, `all` mode, failed read). - The shared `buildRun` fixture in `run-failure-report.test.ts` now has `createdAt`, like a persisted run row. Without it, the wider scheduled-retry read failed on in-memory test runs and logged a spurious warning. Real report inputs are full database rows, so production code does not change for this. - `server/src/services/native-runtime/native-tool-receipt-limit.integration.test.ts`: the real native executor and finalizer path now also asserts the Sentry outcome. A continued stop sends no report. A cancelled turn without a continuation still sends one. The fixture run now has `startedAt`, as a real started run does. Without it, no cancelled-run report can occur, and the new assertion would prove nothing. ## Verification - Red before the fix: the two new unit cases failed (`captureRunFailure` was called), and the integration assertion failed (`expected [...] to have a length of +0 but got 1`). - Green after the fix: - `npx vitest run src/services/__tests__/run-failure-report.test.ts src/__tests__/run-failure-sentry.test.ts src/__tests__/run-failure-sentry-real-sdk.test.ts src/services/native-runtime/native-provider-overload.integration.test.ts src/services/native-runtime/native-tool-receipt-limit.integration.test.ts src/services/native-runtime/status-arbiter.test.ts` → 383 passed, 13 skipped. - `node scripts/typecheck-layers.mjs` in `server` → no errors. - `pnpm verify` did not finish in the local sandbox in the time limit. The only failures were 3 cases in `server/src/__tests__/archived-nested-repository.test.ts`. They also fail on unchanged `master` in the same sandbox, and they do not use this code. CI runs the full suite. ## Risks - Low risk. The change only removes Sentry events. It does not change run status, task status, retries, or the user's view. - A failed or cancelled native run that is not a capacity failure now does one more indexed read before its report. That read is bounded by company, task, and agent, and it happens only when a report is about to be sent. - A successor with this retry reason exists only when the status arbiter proved the receipt-limit stop from a persisted runner diagnostic. A user Stop refuses the continuation, and an exhausted lane creates none, so these runs still report. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, code search, and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
3 files changed
+102
-9
No files matched your search
@@ -122,6 +122,7 @@ describeEmbeddedPostgres("reportRunFailure", () => {
|
||||
errorCode: "process_lost",
|
||||
nativeIssueId: randomUUID(),
|
||||
contextSnapshot: null,
|
||||
createdAt: new Date(),
|
||||
...overrides,
|
||||
} as unknown as typeof heartbeatRuns.$inferSelect;
|
||||
}
|
||||
@@ -530,6 +531,79 @@ describeEmbeddedPostgres("reportRunFailure", () => {
|
||||
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
async function nativeToolReceiptLimitStop(input: {
|
||||
run?: Partial<typeof heartbeatRuns.$inferSelect>;
|
||||
retry?: Partial<typeof heartbeatRuns.$inferInsert> | null;
|
||||
} = {}) {
|
||||
await seedCompanyAndAgent({ adapterType: "paperclip_runner" });
|
||||
const run = buildRun({
|
||||
status: "cancelled", error: "Native session cancelled", errorCode: "cancelled",
|
||||
runtimeMode: "native", nativePhase: "committed", exitCode: 1, signal: null,
|
||||
startedAt: new Date(0), finishedAt: new Date(1000),
|
||||
resultJson: {
|
||||
stopReason: "cancelled", finalizationPhase: "committed", finalizationReasonCode: "semantic_tool_turn_receipt_limit",
|
||||
cancellation: { source: "unknown", expected: false, initiator: { type: "system" },
|
||||
reason: "Execution stopped without a recorded cancellation request: Native session cancelled",
|
||||
recordedAt: new Date(1000).toISOString() },
|
||||
},
|
||||
...input.run,
|
||||
});
|
||||
const [stored] = await db.insert(heartbeatRuns).values(run).returning();
|
||||
if (input.retry !== null) {
|
||||
await db.insert(heartbeatRuns).values({
|
||||
companyId, agentId, invocationSource: "automation", triggerDetail: "system", status: "scheduled_retry",
|
||||
retryOfRunId: run.id, scheduledRetryReason: "semantic_tool_turn_receipt_limit", scheduledRetryAttempt: 1,
|
||||
scheduledRetryAt: new Date(Date.now() + 5_000), errorCode: "semantic_tool_turn_receipt_limit",
|
||||
contextSnapshot: { issueId: run.nativeIssueId, retryOfRunId: run.id }, ...input.retry,
|
||||
});
|
||||
}
|
||||
return stored!;
|
||||
}
|
||||
|
||||
it("keeps a native tool-receipt-limit stop local once its fresh-session continuation is scheduled", async () => {
|
||||
const run = await nativeToolReceiptLimitStop();
|
||||
await reportRunFailure(db, run);
|
||||
await reportRunFailure(db, run, { phase: "execute" });
|
||||
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps a failed native run local when the finalizer scheduled its receipt-limit continuation", async () => {
|
||||
const run = await nativeToolReceiptLimitStop({ run: { status: "failed", errorCode: "native_run_failed",
|
||||
error: "The provider turn failed", resultJson: { finalizationPhase: "committed" } } });
|
||||
await reportRunFailure(db, run);
|
||||
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
["the continuations are exhausted", { retry: null }],
|
||||
["the successor belongs to another retry lane", { retry: { scheduledRetryReason: "native_safe_replacement" } }],
|
||||
["the successor belongs to another task", { retry: { contextSnapshot: { issueId: randomUUID() } } }],
|
||||
["the run has no native task", { run: { nativeIssueId: null } }],
|
||||
["the run is legacy", { run: { runtimeMode: "legacy" } }],
|
||||
["the run has a signal", { run: { signal: "SIGTERM" } }],
|
||||
["the run timed out", { run: { status: "timed_out" } }],
|
||||
] as const)("reports a native tool-receipt-limit stop when %s", async (_label, input) => {
|
||||
const run = await nativeToolReceiptLimitStop(input as Parameters<typeof nativeToolReceiptLimitStop>[0]);
|
||||
await reportRunFailure(db, run);
|
||||
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("reports a native tool-receipt-limit stop during setup, in all mode, or when its continuation cannot be read", async () => {
|
||||
const run = await nativeToolReceiptLimitStop();
|
||||
await reportRunFailure(db, run, { phase: "setup" });
|
||||
const select = db.select.bind(db);
|
||||
const spy = vi.spyOn(db, "select").mockImplementationOnce(select as typeof db.select)
|
||||
.mockImplementation(() => { throw new Error("fixture continuation read failed"); });
|
||||
try {
|
||||
await reportRunFailure(db, run);
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "all");
|
||||
await reportRunFailure(db, run);
|
||||
expect(mockCaptureRunFailure).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
function providerAccessRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
|
||||
return buildRun({
|
||||
error: "ACP agent reported a terminal access failure.\nInvalid API key · Fix external API key",
|
||||
|
||||
@@ -8,6 +8,10 @@ import { CONTROL_PLANE_CONFORMANCE_RESULT } from "../../vendor/paperclip-runner/
|
||||
|
||||
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
|
||||
const receiptLimitRead = vi.hoisted(() => ({ after: null as null | (() => Promise<unknown>) }));
|
||||
const sentry = vi.hoisted(() => ({ captureRunFailure: vi.fn((_event: { runId: string }) => true) }));
|
||||
vi.mock("../../sentry.js", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("../../sentry.js")>()), captureRunFailure: sentry.captureRunFailure,
|
||||
}));
|
||||
vi.mock("./native-tool-receipt-limit.js", async (importOriginal) => {
|
||||
const original = await importOriginal<typeof import("./native-tool-receipt-limit.js")>();
|
||||
return { ...original, readPersistedNativeToolReceiptLimitStop: async (...args: Parameters<typeof original.readPersistedNativeToolReceiptLimitStop>) => {
|
||||
@@ -33,24 +37,30 @@ import { createReleaseIssueExecution } from "../../modules/wake-queue/applicatio
|
||||
import { taskInterruptionPresentation } from "../task-interruption-presentation.js";
|
||||
import { materializeNativeToolReceiptLimitContinuation } from "./native-provider-capacity-retry.js";
|
||||
import { NativeStatusRaceError } from "./status-decision-committer.js";
|
||||
import { waitForPendingRunFailureReports } from "../run-failure-report.js";
|
||||
|
||||
describe("native turn stopped at its tool receipt limit", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
const companyId = randomUUID(), agentId = randomUUID();
|
||||
beforeAll(async () => {
|
||||
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "unexpected");
|
||||
temporary = await startEmbeddedPostgresTestDatabase("native-tool-receipt-limit-");
|
||||
db = createDb(temporary.connectionString);
|
||||
await db.insert(companies).values({ id: companyId, name: "Receipt limit regression", issuePrefix: "RCL" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Receipt worker", status: "active", adapterType: "paperclip_runner", adapterConfig: { provider: "opencode", model: "test-model" } });
|
||||
});
|
||||
afterAll(async () => { await temporary?.cleanup(); });
|
||||
afterAll(async () => { vi.unstubAllEnvs(); await temporary?.cleanup(); });
|
||||
async function sentryReports(runId: string) {
|
||||
await waitForPendingRunFailureReports();
|
||||
return sentry.captureRunFailure.mock.calls.filter(([event]) => event.runId === runId);
|
||||
}
|
||||
|
||||
let finalTurnId: string | null = null;
|
||||
async function stopAtReceiptLimit(runId: string, issueId: string, diagnosticCode: string, beforeFinalize?: () => Promise<unknown>,
|
||||
diagnosticTurnId?: string) {
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
const [run] = await db.update(heartbeatRuns).set({ status: "running", runtimeMode: "native", nativeIssueId: issueId, error: null, errorCode: null }).where(eq(heartbeatRuns.id, runId)).returning();
|
||||
const [run] = await db.update(heartbeatRuns).set({ status: "running", startedAt: new Date(), runtimeMode: "native", nativeIssueId: issueId, error: null, errorCode: null }).where(eq(heartbeatRuns.id, runId)).returning();
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
|
||||
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
|
||||
@@ -115,6 +125,7 @@ describe("native turn stopped at its tool receipt limit", () => {
|
||||
contextSnapshot: { issueId, forceFreshSession: true, livenessContinuationSourceRunId: runId,
|
||||
livenessContinuationReason: NATIVE_TOOL_RECEIPT_LIMIT_CODE, livenessContinuationInstruction: NATIVE_TOOL_RECEIPT_LIMIT_HINT } });
|
||||
expect(retry!.contextSnapshot).not.toHaveProperty("wakeCommentId");
|
||||
expect(await sentryReports(runId)).toHaveLength(0);
|
||||
expect(retry!.scheduledRetryAt!.getTime() - Date.now()).toBeLessThanOrEqual(NATIVE_TOOL_RECEIPT_LIMIT_CONTINUATION_DELAY_MS);
|
||||
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
|
||||
expect(decision).toMatchObject({ reasonCode: NATIVE_TOOL_RECEIPT_LIMIT_CODE });
|
||||
@@ -170,6 +181,7 @@ describe("native turn stopped at its tool receipt limit", () => {
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
|
||||
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
|
||||
expect(decision).toMatchObject({ reasonCode: "cancellation_run_only" });
|
||||
expect(await sentryReports(runId)).toEqual([[expect.objectContaining({ runStatus: "cancelled" })]]);
|
||||
});
|
||||
|
||||
it("refuses to create the continuation under the run lock once a person's Stop is saved", async () => {
|
||||
|
||||
@@ -20,6 +20,7 @@ import { RESTRICTED_WORKSPACE_SOURCE_REASON } from "./workspace-source-restricti
|
||||
import { PLUGIN_RUNNER_AI_CONNECTION_REQUIRED } from "./plugin-runner-ai-connection.js";
|
||||
import { PROJECT_STORAGE_ENVIRONMENT_INCOMPATIBLE } from "./project-storage-environment-reason.js";
|
||||
import { NATIVE_PROVIDER_OVERLOADED_CODE } from "./native-runtime/native-provider-failure.js";
|
||||
import { NATIVE_TOOL_RECEIPT_LIMIT_CODE } from "./native-runtime/native-tool-receipt-limit.js";
|
||||
import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js";
|
||||
import { REVIEWED_CHAT_BINDING_NOT_READY } from "./reviewed-chat-binding-diagnostics.js";
|
||||
import { readProviderFailure } from "@paperclipai/shared";
|
||||
@@ -313,14 +314,20 @@ function isCodexAcpModelCapacityFailure(run: HeartbeatRun, options: RunFailureRe
|
||||
* carries the task in its context, so the company/context-issue index bounds
|
||||
* the read to that task's runs created since this one.
|
||||
*/
|
||||
async function isScheduledNativeCapacityRetry(db: Db, run: HeartbeatRun, options: RunFailureReportOptions): Promise<boolean> {
|
||||
if (run.status !== "failed" || run.runtimeMode !== "native" || run.errorCode !== NATIVE_PROVIDER_OVERLOADED_CODE ||
|
||||
run.nativeIssueId == null || options.phase === "setup" || run.signal != null) return false;
|
||||
function scheduledNativeRetryReason(run: HeartbeatRun, options: RunFailureReportOptions): string | null {
|
||||
if (run.runtimeMode !== "native" || run.nativeIssueId == null || options.phase === "setup" || run.signal != null) return null;
|
||||
if (run.status === "failed" && run.errorCode === NATIVE_PROVIDER_OVERLOADED_CODE) return NATIVE_PROVIDER_OVERLOADED_CODE;
|
||||
return run.status === "failed" || run.status === "cancelled" ? NATIVE_TOOL_RECEIPT_LIMIT_CODE : null;
|
||||
}
|
||||
|
||||
async function isScheduledNativeRetry(db: Db, run: HeartbeatRun, options: RunFailureReportOptions): Promise<boolean> {
|
||||
const reason = scheduledNativeRetryReason(run, options);
|
||||
if (!reason) return false;
|
||||
const [retry] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, run.companyId),
|
||||
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${run.nativeIssueId}`, gte(heartbeatRuns.createdAt, run.createdAt),
|
||||
eq(heartbeatRuns.agentId, run.agentId),
|
||||
eq(heartbeatRuns.retryOfRunId, run.id), eq(heartbeatRuns.scheduledRetryReason, NATIVE_PROVIDER_OVERLOADED_CODE),
|
||||
eq(heartbeatRuns.retryOfRunId, run.id), eq(heartbeatRuns.scheduledRetryReason, reason),
|
||||
)).limit(1);
|
||||
return retry !== undefined;
|
||||
}
|
||||
@@ -446,11 +453,11 @@ async function captureTerminalRunFailure(
|
||||
return false;
|
||||
});
|
||||
if (rejectedExternally) return;
|
||||
const capacityRetry = await isScheduledNativeCapacityRetry(db, run, options).catch((err) => {
|
||||
logger.warn({ err, runId: run.id }, "failed to read native capacity retry for run failure report");
|
||||
const scheduledRetry = await isScheduledNativeRetry(db, run, options).catch((err) => {
|
||||
logger.warn({ err, runId: run.id }, "failed to read native scheduled retry for run failure report");
|
||||
return false;
|
||||
});
|
||||
if (capacityRetry) return;
|
||||
if (scheduledRetry) return;
|
||||
const restartEndedUsageLimit = await isRestartEndedNativeUsageLimit(db, run, options).catch((err) => {
|
||||
logger.warn({ err, runId: run.id }, "failed to read native usage-limit terminal for run failure report");
|
||||
return false;
|
||||
|
||||
Reference in new issue
Block a user