Files
PaperClipAI/server/src/services/native-runtime/native-tool-receipt-limit.integration.test.ts
T
5db2ccdae6 fix(sentry): keep a continued native tool-receipt-limit stop out of Sentry (#22399)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends unexpected run failures to Sentry as run-failure
events. It keeps known, recovered failures local
> - The runner stops a native turn when the turn holds too many tool
results (`semantic_tool_turn_receipt_limit`). The finalizer then
continues the task in a fresh session
> - The runner's stop has no recorded cancellation request. So the
cancelled run keeps `unknown`, unexpected cancellation provenance, and
`reportRunFailure` sends it to Sentry as "Native session cancelled"
> - The continuation owns recovery, so this event is noise. The native
capacity retry had the same problem, and #19051 and #19320 fixed it with
the successor run as the proof
> - This pull request uses the same successor-run proof for the
tool-receipt-limit continuation
> - The benefit is that a continued receipt-limit stop does not make
Sentry noise. Exhausted continuations still report

## Linked Issues or Issue Description

Refs #19051, #19320

**What happened?**
On staging, a native Codex run stopped at its tool-result receipt limit.
The finalizer committed the `semantic_tool_turn_receipt_limit` decision
and scheduled a fresh-session continuation run (`retryOfRunId` = the
stopped run, `scheduledRetryReason` =
`semantic_tool_turn_receipt_limit`). The continuation succeeded and the
task finished. Sentry still got a run-failure event: `Error: Native
session cancelled`, `run_status=cancelled`, cancellation `source:
unknown`, `expected: false`.

**Expected behavior**
A receipt-limit stop that has a scheduled continuation stays local. The
continuation reports itself if it fails. A stop with no continuation
(exhausted, or the user stopped it) stays reportable as before.

**Steps to reproduce**
1. Run a native turn that reaches the semantic-tool receipt limit.
2. Let the finalizer schedule the fresh-session continuation.
3. With `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected`, see that the
cancelled run is sent to Sentry.

## What Changed

- `server/src/services/run-failure-report.ts`:
`isScheduledNativeCapacityRetry` becomes `isScheduledNativeRetry`. It
reads the successor run for the lane that can own the run:
`native_provider_overloaded` for a failed capacity run (unchanged), and
`semantic_tool_turn_receipt_limit` for any other failed or cancelled
native run. The query, task binding, setup-phase, signal, and
read-failure behavior do not change.
- `server/src/services/__tests__/run-failure-report.test.ts`: regression
tests with the persisted shape of the staging run, plus negative cases
(no successor, other retry lane, other task, no native task, legacy,
signal, timed out, setup phase, `all` mode, failed read).
- The shared `buildRun` fixture in `run-failure-report.test.ts` now has
`createdAt`, like a persisted run row. Without it, the wider
scheduled-retry read failed on in-memory test runs and logged a spurious
warning. Real report inputs are full database rows, so production code
does not change for this.
-
`server/src/services/native-runtime/native-tool-receipt-limit.integration.test.ts`:
the real native executor and finalizer path now also asserts the Sentry
outcome. A continued stop sends no report. A cancelled turn without a
continuation still sends one. The fixture run now has `startedAt`, as a
real started run does. Without it, no cancelled-run report can occur,
and the new assertion would prove nothing.

## Verification

- Red before the fix: the two new unit cases failed (`captureRunFailure`
was called), and the integration assertion failed (`expected [...] to
have a length of +0 but got 1`).
- Green after the fix:
- `npx vitest run src/services/__tests__/run-failure-report.test.ts
src/__tests__/run-failure-sentry.test.ts
src/__tests__/run-failure-sentry-real-sdk.test.ts
src/services/native-runtime/native-provider-overload.integration.test.ts
src/services/native-runtime/native-tool-receipt-limit.integration.test.ts
src/services/native-runtime/status-arbiter.test.ts` → 383 passed, 13
skipped.
  - `node scripts/typecheck-layers.mjs` in `server` → no errors.
- `pnpm verify` did not finish in the local sandbox in the time limit.
The only failures were 3 cases in
`server/src/__tests__/archived-nested-repository.test.ts`. They also
fail on unchanged `master` in the same sandbox, and they do not use this
code. CI runs the full suite.

## Risks

- Low risk. The change only removes Sentry events. It does not change
run status, task status, retries, or the user's view.
- A failed or cancelled native run that is not a capacity failure now
does one more indexed read before its report. That read is bounded by
company, task, and agent, and it happens only when a report is about to
be sent.
- A successor with this retry reason exists only when the status arbiter
proved the receipt-limit stop from a persisted runner diagnostic. A user
Stop refuses the continuation, and an exhausted lane creates none, so
these runs still report.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, code search, and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 06:37:30 -08:00

209 lines
17 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, agentWakeupRequests, companies, completionContracts, createDb, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, statusDecisionEffects, statusDecisions } from "@paperclipai/db";
import type { ControlPlanePort, NativeExecutionInputV1, PrpEvent, PrpTerminalState } from "@paperclipai/paperclip-runner";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { CONTROL_PLANE_CONFORMANCE_RESULT } from "../../vendor/paperclip-runner/testing.js";
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
const receiptLimitRead = vi.hoisted(() => ({ after: null as null | (() => Promise<unknown>) }));
const sentry = vi.hoisted(() => ({ captureRunFailure: vi.fn((_event: { runId: string }) => true) }));
vi.mock("../../sentry.js", async (importOriginal) => ({
...(await importOriginal<typeof import("../../sentry.js")>()), captureRunFailure: sentry.captureRunFailure,
}));
vi.mock("./native-tool-receipt-limit.js", async (importOriginal) => {
const original = await importOriginal<typeof import("./native-tool-receipt-limit.js")>();
return { ...original, readPersistedNativeToolReceiptLimitStop: async (...args: Parameters<typeof original.readPersistedNativeToolReceiptLimitStop>) => {
const stopped = await original.readPersistedNativeToolReceiptLimitStop(...args);
const after = receiptLimitRead.after;
receiptLimitRead.after = null;
await after?.();
return stopped;
} };
});
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()), executeNativeSession: provider.execute,
}));
import { executePaperclipNativeSession } from "./native-session-executor.js";
import { buildNativeCompletionContract } from "./completion-contracts.js";
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
import { finalizeNativeRun } from "./native-run-finalizer.js";
import { NATIVE_TOOL_RECEIPT_LIMIT_CODE, NATIVE_TOOL_RECEIPT_LIMIT_CONTINUATION_DELAY_MS, NATIVE_TOOL_RECEIPT_LIMIT_HINT,
readPersistedNativeToolReceiptLimitStop } from "./native-tool-receipt-limit.js";
import { heartbeatService } from "../heartbeat.js";
import { createPostgresWakeQueueAdapter } from "../../modules/wake-queue/adapters/postgres.js";
import { createReleaseIssueExecution } from "../../modules/wake-queue/application/use-cases.js";
import { taskInterruptionPresentation } from "../task-interruption-presentation.js";
import { materializeNativeToolReceiptLimitContinuation } from "./native-provider-capacity-retry.js";
import { NativeStatusRaceError } from "./status-decision-committer.js";
import { waitForPendingRunFailureReports } from "../run-failure-report.js";
describe("native turn stopped at its tool receipt limit", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
const companyId = randomUUID(), agentId = randomUUID();
beforeAll(async () => {
vi.stubEnv("PAPERCLIP_SENTRY_RUN_FAILURES", "unexpected");
temporary = await startEmbeddedPostgresTestDatabase("native-tool-receipt-limit-");
db = createDb(temporary.connectionString);
await db.insert(companies).values({ id: companyId, name: "Receipt limit regression", issuePrefix: "RCL" });
await db.insert(agents).values({ id: agentId, companyId, name: "Receipt worker", status: "active", adapterType: "paperclip_runner", adapterConfig: { provider: "opencode", model: "test-model" } });
});
afterAll(async () => { vi.unstubAllEnvs(); await temporary?.cleanup(); });
async function sentryReports(runId: string) {
await waitForPendingRunFailureReports();
return sentry.captureRunFailure.mock.calls.filter(([event]) => event.runId === runId);
}
let finalTurnId: string | null = null;
async function stopAtReceiptLimit(runId: string, issueId: string, diagnosticCode: string, beforeFinalize?: () => Promise<unknown>,
diagnosticTurnId?: string) {
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
const [run] = await db.update(heartbeatRuns).set({ status: "running", startedAt: new Date(), runtimeMode: "native", nativeIssueId: issueId, error: null, errorCode: null }).where(eq(heartbeatRuns.id, runId)).returning();
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
const contractInput = buildNativeCompletionContract(issue, { revision: contract.revision });
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
const execution: NativeExecutionInputV1 = {
schema: "paperclip.native-execution-input.v1", binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
provider: { kind: "codex", model: "test-model" },
task: { identifier: issueId, title: issue.title, description: null, prompt: "Continue the task using its history.", workMode: "standard" },
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", contract: contractInput },
interactionResponses: [], credentialBindings: [],
};
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
const event: PrpEvent = {
schema: "paperclip.prp.event.v1", runId, normalizedSessionId: native.normalizedSessionId, turnId: diagnosticTurnId ?? native.turnId,
sourceInstanceId: native.runnerInstanceId, sourceEventId: "receipt-limit", sourceSeq: 1, sourceKind: "runner", eventType: "harness.diagnostic", schemaVersion: 1, priority: 0, emittedAt: new Date().toISOString(),
payload: { provider: "opencode", code: diagnosticCode, operationId: "get_task", callId: "call-80",
message: "The active provider turn reached its durable semantic-tool receipt limit and was interrupted", paperclipExecuted: false },
};
const terminal: PrpTerminalState = { schema: "paperclip.prp.terminal.v1", turnTerminalState: "interrupted", runTerminalState: "cancelled", reportedWorkDisposition: "needs_review" };
const result = { ...CONTROL_PLANE_CONFORMANCE_RESULT, reportedWorkDisposition: "needs_review" as const,
summary: "The OpenCode run stopped before it completed.",
completionClaim: { ...CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim, contractRevision: contractInput.revision },
attentionRequests: [{ kind: "review" as const, summary: "Review the stopped OpenCode run before continuing.", ownerClass: "human" as const }] };
provider.execute.mockReset().mockImplementation(async (options: { controlPlane: ControlPlanePort }) => {
await options.controlPlane.openRun({ identity: { companyId, issueId, runId, agentId, sessionId: native.normalizedSessionId }, backendKind: "mock", sourceInstanceId: native.runnerInstanceId });
await options.controlPlane.appendEvent(event);
await options.controlPlane.completeRun({ result, terminal, turnId: native.turnId });
return { result, terminal, turnId: native.turnId, normalizedSessionId: native.normalizedSessionId, providerSessionId: "fixture-session", driverKind: "codex_app_server", driverVersion: "fixture", nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null };
});
const unexpected = async () => { throw new Error("Unexpected provider call"); };
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: { descriptor: unexpected, openSession: unexpected } });
await beforeFinalize?.();
for (let replay = 0; replay < 2; replay++) await finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
expect(provider.execute).toHaveBeenCalledTimes(1);
finalTurnId = native.turnId;
return (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]!;
}
function release() {
const unexpected = vi.fn(async () => { throw new Error("Unexpected recovery"); });
return createReleaseIssueExecution({
issueLock: createPostgresWakeQueueAdapter(db, { resolveResponsibleUserId: unexpected, getRoutineEnv: unexpected, resolveSessionBeforeForWakeup: unexpected }),
recovery: { escalateStrandedAssignedIssue: unexpected, escalateStrandedRecoveryIssueInPlace: unexpected },
});
}
it("continues the task at once in a fresh session and shows no Retry", async () => {
const issueId = randomUUID(), runId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Parallel reads task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: "receipt-owner" });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: "receipt-owner", contextSnapshot: { issueId, wakeCommentId: "original-message" } });
const stopped = await stopAtReceiptLimit(runId, issueId, NATIVE_TOOL_RECEIPT_LIMIT_CODE);
expect(stopped.status).toBe("cancelled");
expect(await readPersistedNativeToolReceiptLimitStop(db, stopped, finalTurnId)).toBe(true);
expect(await readPersistedNativeToolReceiptLimitStop(db, stopped, "another-turn")).toBe(false);
const [retry, ...others] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.retryOfRunId, runId)));
expect(others).toHaveLength(0);
expect(retry).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: NATIVE_TOOL_RECEIPT_LIMIT_CODE, scheduledRetryAttempt: 1,
contextSnapshot: { issueId, forceFreshSession: true, livenessContinuationSourceRunId: runId,
livenessContinuationReason: NATIVE_TOOL_RECEIPT_LIMIT_CODE, livenessContinuationInstruction: NATIVE_TOOL_RECEIPT_LIMIT_HINT } });
expect(retry!.contextSnapshot).not.toHaveProperty("wakeCommentId");
expect(await sentryReports(runId)).toHaveLength(0);
expect(retry!.scheduledRetryAt!.getTime() - Date.now()).toBeLessThanOrEqual(NATIVE_TOOL_RECEIPT_LIMIT_CONTINUATION_DELAY_MS);
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
expect(decision).toMatchObject({ reasonCode: NATIVE_TOOL_RECEIPT_LIMIT_CODE });
expect(await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, decision!.id)))
.toEqual(expect.arrayContaining([expect.objectContaining({ effectKind: "schedule_retry", targetType: "heartbeat_run", targetId: retry!.id })]));
expect((await release()({ companyId, runId, now: new Date() })).outcome.kind).toBe("released");
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
expect(issue).toMatchObject({ status: "in_progress", assigneeAgentId: agentId });
expect(await taskInterruptionPresentation(db, issue!)).toMatchObject({ state: "retry_scheduled", canRetry: false, userAction: null });
const scheduler = heartbeatService(db, { runtimeEnv: {} });
expect(await scheduler.promoteDueScheduledRetries(new Date())).toMatchObject({ promoted: 0 });
expect(await scheduler.promoteDueScheduledRetries(retry!.scheduledRetryAt!)).toMatchObject({ promoted: 1, runIds: [retry!.id] });
expect(await taskInterruptionPresentation(db, issue!)).toMatchObject({ state: "working", canRetry: false, userAction: null });
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.runId, retry!.id))).toHaveLength(1);
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, retry!.id));
});
it("does not continue a turn a person stopped while it settled", async () => {
const issueId = randomUUID(), runId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Stopped task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: "receipt-owner" });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: "receipt-owner", contextSnapshot: { issueId } });
await stopAtReceiptLimit(runId, issueId, NATIVE_TOOL_RECEIPT_LIMIT_CODE, () => db.update(heartbeatRuns)
.set({ resultJson: { cancelledByActorType: "user", cancelledByUserId: "receipt-owner" } }).where(eq(heartbeatRuns.id, runId)));
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
expect(decision?.reasonCode).not.toBe(NATIVE_TOOL_RECEIPT_LIMIT_CODE);
});
it("settles the cancellation when a person stops the run after finalization first read it", async () => {
const issueId = randomUUID(), runId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Late stop task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: "receipt-owner" });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: "receipt-owner", contextSnapshot: { issueId } });
await stopAtReceiptLimit(runId, issueId, NATIVE_TOOL_RECEIPT_LIMIT_CODE, async () => {
receiptLimitRead.after = () => db.update(heartbeatRuns)
.set({ resultJson: { cancelledByActorType: "user", cancelledByUserId: "receipt-owner" } }).where(eq(heartbeatRuns.id, runId));
});
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
expect(coordinator).toMatchObject({ phase: "committed", failureCode: null });
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
expect(decision).toMatchObject({ reasonCode: "cancellation_run_only" });
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
});
it("does not continue when only an earlier turn hit the receipt limit", async () => {
const issueId = randomUUID(), runId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Earlier turn task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: "receipt-owner" });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: "receipt-owner", contextSnapshot: { issueId } });
const stopped = await stopAtReceiptLimit(runId, issueId, NATIVE_TOOL_RECEIPT_LIMIT_CODE, undefined, "earlier-provider-turn");
expect(await readPersistedNativeToolReceiptLimitStop(db, stopped, finalTurnId)).toBe(false);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
expect(decision).toMatchObject({ reasonCode: "cancellation_run_only" });
expect(await sentryReports(runId)).toEqual([[expect.objectContaining({ runStatus: "cancelled" })]]);
});
it("refuses to create the continuation under the run lock once a person's Stop is saved", async () => {
const issueId = randomUUID(), runId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Locked stop task", status: "in_progress", assigneeAgentId: agentId });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "cancelled", runtimeMode: "native", nativeIssueId: issueId,
contextSnapshot: { issueId }, resultJson: { cancelledByActorType: "user", cancelledByUserId: "receipt-owner" } });
await expect(db.transaction(tx => materializeNativeToolReceiptLimitContinuation({ tx: tx as never, companyId, issueId, runId, agentId })))
.rejects.toBeInstanceOf(NativeStatusRaceError);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.idempotencyKey, `native-tool-receipt-limit:${companyId}:${runId}`))).toHaveLength(0);
});
it("leaves another cancelled turn to its existing recovery", async () => {
const issueId = randomUUID(), runId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Other diagnostic task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: "receipt-owner" });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: "receipt-owner", contextSnapshot: { issueId } });
const stopped = await stopAtReceiptLimit(runId, issueId, "semantic_tool_denied");
expect(await readPersistedNativeToolReceiptLimitStop(db, stopped, finalTurnId)).toBe(false);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))).toHaveLength(0);
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
expect(decision).toMatchObject({ reasonCode: "cancellation_run_only" });
});
});