fix(sentry): keep provider-rejected Grok wakes out of run failure reports (#16297)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip Runner can run an external Grok bot. Paperclip offers
the bot an assignment and sends a webhook wake. The bot must accept the
assignment in 10 minutes.
> - When the Grok endpoint rejects every wake with HTTP 4xx, the bot
never sees the work. The offer expires and the run fails with `Dot
assignment acceptance or execution authority expired.`
> - Paperclip already records each wake in `grok_bot_dispatches` with
its HTTP status. The operator sees the rejection and can retry it. Thus
Paperclip operates correctly in this case.
> - But `reportRunFailure` sends this run to Sentry as an unexplained
backend error. This makes noise and hides real defects.
> - This pull request skips the Sentry report only when durable receipts
prove that the Grok endpoint rejected every wake.
> - The benefit is a cleaner Sentry signal. Local defects, uncertain
delivery, and accepted work stay visible.

## Linked Issues or Issue Description

Sentry issue `PAPERCLIP-BACKEND-3M` (staging, 1 event).

**What happened?**
A Grok bot run on staging failed after 10 minutes in the `preparing`
stage. Sentry got the error `Dot assignment acceptance or execution
authority expired.` with the code `native_provider_terminal_failed`. The
stored receipts show two `mailbox_updated` wakes for the assignment. The
Grok endpoint rejected both wakes with HTTP 400: the first wake and one
operator retry. The assignment was never accepted. Wakes with the same
shape got HTTP 200 before and after this run.

**Expected behavior**
The run fails, and the operator sees the rejected delivery. Sentry does
not get a backend error for a proven external rejection.

**Steps to reproduce**
1. Connect a Grok bot agent and verify its routine.
2. Make the Grok endpoint return HTTP 400 for work wakes.
3. Assign a task to the agent and wait 10 minutes.
4. See that the run fails and that a Sentry event is sent.

**Paperclip version or commit**
`f69c6a8` (master).

## What Changed

- `server/src/services/run-failure-report.ts`: Add
`isRejectedExternalBotWake`. It returns true only when all of these
conditions are true:
- The run failed in native mode with `native_provider_terminal_failed`.
- The finalization `failureDetail.originalFailureCode` is
`dot_assignment_expired`.
- The run has exactly one Dot assignment, and that assignment was never
accepted.
- The assignment has at least one `mailbox_updated` wake, and each wake
has the status `rejected` with an HTTP 4xx status.
- `captureTerminalRunFailure` skips the Sentry capture when the check is
true. If the receipt check itself fails, it logs a warning and the
failure stays reportable. The run status, the task, and the operator
recovery do not change.
- `packages/db`: Add the index `grok_dispatch_assignment_kind_idx` on
`grok_bot_dispatches (assignment_id, kind)` (migration `0346`). The
receipt lookup and the dispatcher's per-assignment checks use this
index.
- `server/src/services/__tests__/run-failure-report.test.ts`: Add one
test for the skipped case. Add nine tests for cases that must stay
reportable: no wake, a local rejection with no HTTP status, a delivered
wake, uncertain delivery, accepted work, a different provider failure
code, no finalization, more than one offer, and a receipt read that
fails.

## Verification

- `pnpm --dir server exec vitest run
src/services/__tests__/run-failure-report.test.ts`: 208 tests pass.
- I removed the new catch and ran the tests again. The
receipt-read-failure test failed.
- `pnpm --filter @paperclipai/db run check:migrations` and
`src/migration-snapshot-drift.test.ts` pass.
- I removed the source change and ran the new tests again. The
skipped-case test failed and the six reportable cases passed. This shows
that the test proves the change.
- I did not run the full `pnpm -r typecheck` locally. The server
`typecheck` script needs `cargo` to build the Runner binary, and the
sandbox stopped `tsc` with SIGKILL. CI runs the full typecheck.

## Risks

- Low risk. The change only removes one Sentry report. It does not
change run state, recovery, or what users see.
- The check uses durable receipts, not message text. A wake with no HTTP
status (local validation or credential failure) stays reportable, so a
Paperclip defect cannot hide behind this rule.
- For failed native runs with `native_provider_terminal_failed`, the
check adds up to three indexed reads. Other runs do no extra reads.
- Migration `0346` builds one index in a transaction.
`grok_bot_dispatches` was created in `0344` and is small, so the build
lock is short. The table is not on the migration gate's large-table
list.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic tool use with
shell, code search, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/__tests__/run-failure-report.test.ts

@@ -1,7 +1,9 @@
 import { randomUUID } from "node:crypto";
 import os from "node:os";
 import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
-import { agents, applyPendingMigrations, closeRegisteredClients, companies, createDb, heartbeatRuns, type Db } from "@paperclipai/db";
+import { agents, applyPendingMigrations, authUsers, closeRegisteredClients, companies, companySecrets, createDb, dotAgentBindings,
+  dotRunnerAssignments, grokBotDispatches, grokBotRoutines, heartbeatRuns, issues, mcpOauthClients, mcpOauthGrants, nativeRunFinalizations,
+  type Db } from "@paperclipai/db";
 import {
   getEmbeddedPostgresTestSupport,
   startEmbeddedPostgresTestDatabase,
@@ -67,9 +69,15 @@ describeEmbeddedPostgres("reportRunFailure", () => {
     vi.clearAllMocks();
     vi.unstubAllEnvs();
     process.env = inheritedEnv;
+    await db.delete(grokBotRoutines);
+    await db.delete(companySecrets);
+    await db.delete(nativeRunFinalizations);
     await db.delete(heartbeatRuns);
+    await db.delete(issues);
     await db.delete(agents);
     await db.delete(companies);
+    await db.delete(mcpOauthClients);
+    await db.delete(authUsers);
   });
 
   afterAll(async () => {
@@ -335,6 +343,80 @@ describeEmbeddedPostgres("reportRunFailure", () => {
     expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
   });
 
+  async function expiredGrokOffer(input: {
+    wakes: Array<{ status: "rejected" | "accepted" | "acknowledged" | "unknown"; httpStatus: number | null }>;
+    acceptedAt?: Date | null;
+    originalFailureCode?: string;
+    finalization?: boolean;
+    offers?: number;
+  }) {
+    await seedCompanyAndAgent({ adapterType: "paperclip_runner", adapterConfig: { provider: "grok_bot" } });
+    const userId = randomUUID();
+    await db.insert(authUsers).values({ id: userId, name: "Operator", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
+    await db.insert(mcpOauthClients).values({ id: userId, name: "Grok", redirectUris: ["https://client.example/callback"] });
+    const [grant] = await db.insert(mcpOauthGrants).values({ companyId, userId, clientId: userId, resource: "https://paperclip.example/mcp/runner",
+      purpose: "agent", agentId, scopes: ["paperclip:agent"] }).returning();
+    const [binding] = await db.insert(dotAgentBindings).values({ companyId, agentId, operatorId: userId, grantId: grant!.id,
+      providerKind: "grok_bot", status: "ready" }).returning();
+    const [secret] = await db.insert(companySecrets).values({ companyId, key: `grok-${binding!.id}`, name: "Grok routine" }).returning();
+    const [routine] = await db.insert(grokBotRoutines).values({ companyId, agentId, bindingId: binding!.id, bindingGeneration: 1,
+      botName: "Grok", secretId: secret!.id, secretVersion: 1 }).returning();
+    const [issue] = await db.insert(issues).values({ companyId, title: "Grok task", assigneeAgentId: agentId, status: "in_progress" }).returning();
+    const run = buildRun({ errorCode: "native_provider_terminal_failed", error: "Dot assignment acceptance or execution authority expired.",
+      nativeIssueId: issue!.id, runtimeMode: "native", executionStage: "preparing", nativePhase: "terminal_failure", exitCode: null, signal: null });
+    await db.insert(heartbeatRuns).values(run);
+    if (input.finalization !== false) {
+      await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId, issueId: run.nativeIssueId!, phase: "terminal_failure",
+        failureCode: "native_provider_terminal_failed", failureDetail: { originalFailureCode: input.originalFailureCode ?? "dot_assignment_expired" } });
+    }
+    const offers = await db.insert(dotRunnerAssignments).values(Array.from({ length: input.offers ?? 1 }, () => ({ companyId, agentId,
+      bindingId: binding!.id, bindingGeneration: 1, runId: run.id, normalizedSessionId: randomUUID(), turnId: randomUUID(), controllerGeneration: 1,
+      catalogDigest: "fixture", projection: {}, status: "fenced" as const, acceptBy: new Date(Date.now() - 1000),
+      expiresAt: new Date(Date.now() + 3_600_000), acceptedAt: input.acceptedAt ?? null }))).returning();
+    const assignment = offers[0];
+    for (const wake of input.wakes) {
+      await db.insert(grokBotDispatches).values({ companyId, routineId: routine!.id, bindingId: binding!.id, bindingGeneration: 1,
+        grantId: grant!.id, routineRevision: 1, assignmentId: assignment!.id, sourceEventId: randomUUID(), kind: "mailbox_updated",
+        payload: {}, status: wake.status, httpStatus: wake.httpStatus, expiresAt: new Date(Date.now() + 600_000) });
+    }
+    return run;
+  }
+
+  it("keeps a Grok offer local when the endpoint rejected every work wake with HTTP 4xx", async () => {
+    const run = await expiredGrokOffer({ wakes: [{ status: "rejected", httpStatus: 400 }, { status: "rejected", httpStatus: 400 }] });
+    await reportRunFailure(db, run, { phase: "execute" });
+    expect(mockCaptureRunFailure).not.toHaveBeenCalled();
+  });
+
+  it.each([
+    ["no wake was sent", { wakes: [] }],
+    ["a wake was rejected locally without HTTP evidence", { wakes: [{ status: "rejected", httpStatus: 400 }, { status: "rejected", httpStatus: null }] }],
+    ["the endpoint accepted a wake", { wakes: [{ status: "rejected", httpStatus: 400 }, { status: "acknowledged", httpStatus: 200 }] }],
+    ["delivery is uncertain", { wakes: [{ status: "unknown", httpStatus: 503 }] }],
+    ["the bot accepted the work", { wakes: [{ status: "rejected", httpStatus: 400 }], acceptedAt: new Date() }],
+    ["the run failed for another provider reason", { wakes: [{ status: "rejected", httpStatus: 400 }], originalFailureCode: "provider_failed" }],
+    ["the run has no finalization", { wakes: [{ status: "rejected", httpStatus: 400 }], finalization: false }],
+    ["the run has more than one offer", { wakes: [{ status: "rejected", httpStatus: 400 }], offers: 2 }],
+  ] as const)("keeps an expired Grok offer reportable when %s", async (_label, input) => {
+    const run = await expiredGrokOffer({ ...input, wakes: [...input.wakes] });
+    await reportRunFailure(db, run, { phase: "execute" });
+    expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
+  });
+
+  it("keeps an expired Grok offer reportable when its receipts cannot be read", async () => {
+    const run = await expiredGrokOffer({ wakes: [{ status: "rejected", httpStatus: 400 }] });
+    const select = db.select.bind(db);
+    // The agent lookup succeeds; the receipt reads fail.
+    const spy = vi.spyOn(db, "select").mockImplementationOnce(select as typeof db.select)
+      .mockImplementation(() => { throw new Error("fixture receipt read failed"); });
+    try {
+      await reportRunFailure(db, run, { phase: "execute" });
+    } finally {
+      spy.mockRestore();
+    }
+    expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
+  });
+
   function aiSelectionRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
     return buildRun({
       errorCode: "configuration_incomplete",

server/src/services/run-failure-report.ts

@@ -1,5 +1,5 @@
 import { and, eq } from "drizzle-orm";
-import { agents, heartbeatRuns, type Db } from "@paperclipai/db";
+import { agents, dotRunnerAssignments, grokBotDispatches, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
 import { captureRunFailure, type RunFailureStatus } from "../sentry.js";
 import {
   collectRunFailureDiagnostics,
@@ -174,6 +174,27 @@ function isProviderQuotaFailure(run: HeartbeatRun, options: RunFailureReportOpti
     options.phase !== "setup" && run.signal == null;
 }
 
+/**
+ * The Grok endpoint refused every work wake with an HTTP 4xx, so the offer
+ * expired unaccepted. Durable delivery receipts prove the external cause. A
+ * local rejection (no HTTP status), any delivered wake, or accepted work stays
+ * reportable.
+ */
+async function isRejectedExternalBotWake(db: Db, run: HeartbeatRun): Promise<boolean> {
+  if (run.status !== "failed" || run.runtimeMode !== "native" || run.errorCode !== "native_provider_terminal_failed") return false;
+  const [finalization] = await db.select({ failureDetail: nativeRunFinalizations.failureDetail }).from(nativeRunFinalizations)
+    .where(and(eq(nativeRunFinalizations.runId, run.id), eq(nativeRunFinalizations.companyId, run.companyId)));
+  if (finalization?.failureDetail?.originalFailureCode !== "dot_assignment_expired") return false;
+  const offers = await db.select({ id: dotRunnerAssignments.id, acceptedAt: dotRunnerAssignments.acceptedAt }).from(dotRunnerAssignments)
+    .where(and(eq(dotRunnerAssignments.runId, run.id), eq(dotRunnerAssignments.companyId, run.companyId)));
+  if (offers.length !== 1 || offers[0]!.acceptedAt) return false;
+  const wakes = await db.select({ status: grokBotDispatches.status, httpStatus: grokBotDispatches.httpStatus }).from(grokBotDispatches)
+    .where(and(eq(grokBotDispatches.assignmentId, offers[0]!.id), eq(grokBotDispatches.companyId, run.companyId),
+      eq(grokBotDispatches.kind, "mailbox_updated")));
+  return wakes.length > 0 && wakes.every((wake) =>
+    wake.status === "rejected" && wake.httpStatus != null && wake.httpStatus >= 400 && wake.httpStatus < 500);
+}
+
 /**
  * Report a terminal run failure to Sentry. Returns at once for any status
  * other than failures and unexpected started cancellations. Never throws — a Sentry failure or a
@@ -222,6 +243,12 @@ async function captureTerminalRunFailure(
       .then((rows) => rows[0] ?? null);
 
     if (isHermesConnectionFailure(run, options, agent?.adapterType)) return;
+    // An unreadable receipt proves nothing; keep the failure reportable.
+    const rejectedExternally = await isRejectedExternalBotWake(db, run).catch((err) => {
+      logger.warn({ err, runId: run.id }, "failed to read bot wake receipts for run failure report");
+      return false;
+    });
+    if (rejectedExternally) return;
 
     const taskId = readTaskId(run);
     if (!taskId) {