Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-22 00:30:19 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(sentry): keep provider-rejected Grok wakes out of run failure reports (#16297)

Browse Source 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip Runner can run an external Grok bot. Paperclip offers
the bot an assignment and sends a webhook wake. The bot must accept the
assignment in 10 minutes.
> - When the Grok endpoint rejects every wake with HTTP 4xx, the bot
never sees the work. The offer expires and the run fails with `Dot
assignment acceptance or execution authority expired.`
> - Paperclip already records each wake in `grok_bot_dispatches` with
its HTTP status. The operator sees the rejection and can retry it. Thus
Paperclip operates correctly in this case.
> - But `reportRunFailure` sends this run to Sentry as an unexplained
backend error. This makes noise and hides real defects.
> - This pull request skips the Sentry report only when durable receipts
prove that the Grok endpoint rejected every wake.
> - The benefit is a cleaner Sentry signal. Local defects, uncertain
delivery, and accepted work stay visible.

## Linked Issues or Issue Description

Sentry issue `PAPERCLIP-BACKEND-3M` (staging, 1 event).

**What happened?**
A Grok bot run on staging failed after 10 minutes in the `preparing`
stage. Sentry got the error `Dot assignment acceptance or execution
authority expired.` with the code `native_provider_terminal_failed`. The
stored receipts show two `mailbox_updated` wakes for the assignment. The
Grok endpoint rejected both wakes with HTTP 400: the first wake and one
operator retry. The assignment was never accepted. Wakes with the same
shape got HTTP 200 before and after this run.

**Expected behavior**
The run fails, and the operator sees the rejected delivery. Sentry does
not get a backend error for a proven external rejection.

**Steps to reproduce**
1. Connect a Grok bot agent and verify its routine.
2. Make the Grok endpoint return HTTP 400 for work wakes.
3. Assign a task to the agent and wait 10 minutes.
4. See that the run fails and that a Sentry event is sent.

**Paperclip version or commit**
`f69c6a8` (master).

## What Changed

- `server/src/services/run-failure-report.ts`: Add
`isRejectedExternalBotWake`. It returns true only when all of these
conditions are true:
- The run failed in native mode with `native_provider_terminal_failed`.
- The finalization `failureDetail.originalFailureCode` is
`dot_assignment_expired`.
- The run has exactly one Dot assignment, and that assignment was never
accepted.
- The assignment has at least one `mailbox_updated` wake, and each wake
has the status `rejected` with an HTTP 4xx status.
- `captureTerminalRunFailure` skips the Sentry capture when the check is
true. If the receipt check itself fails, it logs a warning and the
failure stays reportable. The run status, the task, and the operator
recovery do not change.
- `packages/db`: Add the index `grok_dispatch_assignment_kind_idx` on
`grok_bot_dispatches (assignment_id, kind)` (migration `0346`). The
receipt lookup and the dispatcher's per-assignment checks use this
index.
- `server/src/services/__tests__/run-failure-report.test.ts`: Add one
test for the skipped case. Add nine tests for cases that must stay
reportable: no wake, a local rejection with no HTTP status, a delivered
wake, uncertain delivery, accepted work, a different provider failure
code, no finalization, more than one offer, and a receipt read that
fails.

## Verification

- `pnpm --dir server exec vitest run
src/services/__tests__/run-failure-report.test.ts`: 208 tests pass.
- I removed the new catch and ran the tests again. The
receipt-read-failure test failed.
- `pnpm --filter @paperclipai/db run check:migrations` and
`src/migration-snapshot-drift.test.ts` pass.
- I removed the source change and ran the new tests again. The
skipped-case test failed and the six reportable cases passed. This shows
that the test proves the change.
- I did not run the full `pnpm -r typecheck` locally. The server
`typecheck` script needs `cargo` to build the Runner binary, and the
sandbox stopped `tsc` with SIGKILL. CI runs the full typecheck.

## Risks

- Low risk. The change only removes one Sentry report. It does not
change run state, recovery, or what users see.
- The check uses durable receipts, not message text. A wake with no HTTP
status (local validation or credential failure) stays reportable, so a
Paperclip defect cannot hide behind this rule.
- For failed native runs with `native_provider_terminal_failed`, the
check adds up to three indexed reads. Other runs do no extra reads.
- Migration `0346` builds one index in a transaction.
`grok_bot_dispatches` was created in `0344` and is small, so the build
lock is short. The table is not on the migration gate's large-table
list.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic tool use with
shell, code search, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Devin FoleydevinfoleyClaude Opus 5.5
 authored and GitHub committed 2026-10-14 08:32:48 -07:00
1 parent 14f961b698
commit bcfc03bc94
6 files changed
+763 -4
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

packages/db/src/migrations/0346_cloudy_whiplash.sql
+2
View File
@@ -0,0 +1,2 @@
-- Find an assignment's work wakes without scanning the company's dispatch history.
CREATE INDEX "grok_dispatch_assignment_kind_idx" ON "grok_bot_dispatches" USING btree ("assignment_id","kind");
packages/db/src/migrations/meta/0341_snapshot.json → packages/db/src/migrations/meta/0346_snapshot.json
Generated
+642 -2
View File
@@ -1,6 +1,6 @@
{
"id": "927d7532-059b-4a4f-846a-5e3445a81891",
"prevId": "b32fa655-79c7-4793-a021-22258b6c369c",
"id": "40048887-b6ac-4334-b77b-6b8fc4b52701",
"prevId": "3bd7b1df-1434-4fac-87d6-53b90a9eb9e1",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -4098,6 +4098,12 @@
"primaryKey": false,
"notNull": false
},
"created_by_user_id": {
"name": "created_by_user_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
@@ -16852,6 +16858,53 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.company_skill_import_budgets": {
"name": "company_skill_import_budgets",
"schema": "",
"columns": {
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"available_bytes": {
"name": "available_bytes",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 536870912
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {
"company_skill_import_budgets_company_id_companies_id_fk": {
"name": "company_skill_import_budgets_company_id_companies_id_fk",
"tableFrom": "company_skill_import_budgets",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.company_skill_source_entries": {
"name": "company_skill_source_entries",
"schema": "",
@@ -24927,6 +24980,13 @@
"primaryKey": false,
"notNull": false
},
"provider_kind": {
"name": "provider_kind",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'openai_dot'"
},
"generation": {
"name": "generation",
"type": "integer",
@@ -25554,6 +25614,18 @@
"primaryKey": false,
"notNull": false
},
"pending_returned_at": {
"name": "pending_returned_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"observed_at": {
"name": "observed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"source_event_id": {
"name": "source_event_id",
"type": "text",
@@ -30638,6 +30710,495 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.grok_bot_dispatches": {
"name": "grok_bot_dispatches",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"routine_id": {
"name": "routine_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"binding_id": {
"name": "binding_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"binding_generation": {
"name": "binding_generation",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"grant_id": {
"name": "grant_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"routine_revision": {
"name": "routine_revision",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"assignment_id": {
"name": "assignment_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"source_event_id": {
"name": "source_event_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"kind": {
"name": "kind",
"type": "text",
"primaryKey": false,
"notNull": true
},
"payload": {
"name": "payload",
"type": "jsonb",
"primaryKey": false,
"notNull": true
},
"challenge_hash": {
"name": "challenge_hash",
"type": "text",
"primaryKey": false,
"notNull": false
},
"status": {
"name": "status",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'prepared'"
},
"http_status": {
"name": "http_status",
"type": "integer",
"primaryKey": false,
"notNull": false
},
"expires_at": {
"name": "expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"sent_at": {
"name": "sent_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"acknowledged_at": {
"name": "acknowledged_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"grok_dispatch_event_uq": {
"name": "grok_dispatch_event_uq",
"columns": [
{
"expression": "binding_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "binding_generation",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "routine_revision",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "source_event_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"grok_dispatch_company_status_idx": {
"name": "grok_dispatch_company_status_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "status",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"grok_dispatch_assignment_kind_idx": {
"name": "grok_dispatch_assignment_kind_idx",
"columns": [
{
"expression": "assignment_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "kind",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"grok_bot_dispatches_company_id_companies_id_fk": {
"name": "grok_bot_dispatches_company_id_companies_id_fk",
"tableFrom": "grok_bot_dispatches",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"grok_bot_dispatches_routine_id_grok_bot_routines_id_fk": {
"name": "grok_bot_dispatches_routine_id_grok_bot_routines_id_fk",
"tableFrom": "grok_bot_dispatches",
"tableTo": "grok_bot_routines",
"columnsFrom": [
"routine_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"grok_bot_dispatches_binding_id_dot_agent_bindings_id_fk": {
"name": "grok_bot_dispatches_binding_id_dot_agent_bindings_id_fk",
"tableFrom": "grok_bot_dispatches",
"tableTo": "dot_agent_bindings",
"columnsFrom": [
"binding_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"grok_bot_dispatches_grant_id_mcp_oauth_grants_id_fk": {
"name": "grok_bot_dispatches_grant_id_mcp_oauth_grants_id_fk",
"tableFrom": "grok_bot_dispatches",
"tableTo": "mcp_oauth_grants",
"columnsFrom": [
"grant_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"grok_bot_dispatches_assignment_id_dot_runner_assignments_id_fk": {
"name": "grok_bot_dispatches_assignment_id_dot_runner_assignments_id_fk",
"tableFrom": "grok_bot_dispatches",
"tableTo": "dot_runner_assignments",
"columnsFrom": [
"assignment_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.grok_bot_routines": {
"name": "grok_bot_routines",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"agent_id": {
"name": "agent_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"binding_id": {
"name": "binding_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"binding_generation": {
"name": "binding_generation",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"bot_name": {
"name": "bot_name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"secret_id": {
"name": "secret_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"secret_version": {
"name": "secret_version",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"revision": {
"name": "revision",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 1
},
"verified_grant_id": {
"name": "verified_grant_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"verified_at": {
"name": "verified_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"last_contact_at": {
"name": "last_contact_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"recovery_required_at": {
"name": "recovery_required_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"recovery_reason": {
"name": "recovery_reason",
"type": "text",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"grok_routines_binding_uq": {
"name": "grok_routines_binding_uq",
"columns": [
{
"expression": "binding_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"grok_routines_company_agent_idx": {
"name": "grok_routines_company_agent_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "agent_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"grok_bot_routines_company_id_companies_id_fk": {
"name": "grok_bot_routines_company_id_companies_id_fk",
"tableFrom": "grok_bot_routines",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"grok_bot_routines_agent_id_agents_id_fk": {
"name": "grok_bot_routines_agent_id_agents_id_fk",
"tableFrom": "grok_bot_routines",
"tableTo": "agents",
"columnsFrom": [
"agent_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"grok_bot_routines_binding_id_dot_agent_bindings_id_fk": {
"name": "grok_bot_routines_binding_id_dot_agent_bindings_id_fk",
"tableFrom": "grok_bot_routines",
"tableTo": "dot_agent_bindings",
"columnsFrom": [
"binding_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"grok_bot_routines_secret_id_company_secrets_id_fk": {
"name": "grok_bot_routines_secret_id_company_secrets_id_fk",
"tableFrom": "grok_bot_routines",
"tableTo": "company_secrets",
"columnsFrom": [
"secret_id"
],
"columnsTo": [
"id"
],
"onDelete": "no action",
"onUpdate": "no action"
},
"grok_bot_routines_verified_grant_id_mcp_oauth_grants_id_fk": {
"name": "grok_bot_routines_verified_grant_id_mcp_oauth_grants_id_fk",
"tableFrom": "grok_bot_routines",
"tableTo": "mcp_oauth_grants",
"columnsFrom": [
"verified_grant_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.heartbeat_run_events": {
"name": "heartbeat_run_events",
"schema": "",
@@ -31967,6 +32528,54 @@
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_runs_company_ctx_paperclip_issue_created_idx": {
"name": "heartbeat_runs_company_ctx_paperclip_issue_created_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "(\"context_snapshot\" -> 'paperclipIssue' ->> 'id')",
"asc": true,
"isExpression": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": false,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": true,
"method": "btree",
"with": {}
},
"heartbeat_runs_company_origin_issue_idx": {
"name": "heartbeat_runs_company_origin_issue_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "coalesce(\n \"native_issue_id\"::text,\n nullif(\"context_snapshot\"->>'issueId', ''),\n nullif(\"context_snapshot\"->>'taskId', ''),\n nullif(\"context_snapshot\"->>'taskKey', '')\n )",
"asc": true,
"isExpression": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": true,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
@@ -58385,6 +58994,18 @@
"name": "mcp_attachment_uploads",
"schema": "",
"columns": {
"grant_id": {
"name": "grant_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"work_session_id": {
"name": "work_session_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"id": {
"name": "id",
"type": "uuid",
@@ -58529,6 +59150,19 @@
}
},
"foreignKeys": {
"mcp_attachment_uploads_grant_id_mcp_oauth_grants_id_fk": {
"name": "mcp_attachment_uploads_grant_id_mcp_oauth_grants_id_fk",
"tableFrom": "mcp_attachment_uploads",
"tableTo": "mcp_oauth_grants",
"columnsFrom": [
"grant_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"mcp_attachment_uploads_company_id_companies_id_fk": {
"name": "mcp_attachment_uploads_company_id_companies_id_fk",
"tableFrom": "mcp_attachment_uploads",
@@ -59005,6 +59639,12 @@
"name": "mcp_file_tickets",
"schema": "",
"columns": {
"work_session_id": {
"name": "work_session_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"token_hash": {
"name": "token_hash",
"type": "text",
packages/db/src/migrations/meta/_journal.json
Generated
+7
View File
@@ -2409,6 +2409,13 @@
"when": 1791978393174,
"tag": "0345_luxuriant_carmella_unuscione",
"breakpoints": true
},
{
"idx": 346,
"version": "7",
"when": 1791989860950,
"tag": "0346_cloudy_whiplash",
"breakpoints": true
}
]
}
packages/db/src/schema/grok_bot.ts
+1
View File
@@ -50,4 +50,5 @@ export const grokBotDispatches = pgTable("grok_bot_dispatches", {
}, t => [
uniqueIndex("grok_dispatch_event_uq").on(t.bindingId, t.bindingGeneration, t.routineRevision, t.sourceEventId),
index("grok_dispatch_company_status_idx").on(t.companyId, t.status),
index("grok_dispatch_assignment_kind_idx").on(t.assignmentId, t.kind),
]);
server/src/services/__tests__/run-failure-report.test.ts
+83 -1
View File
@@ -1,7 +1,9 @@
import { randomUUID } from "node:crypto";
import os from "node:os";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { agents, applyPendingMigrations, closeRegisteredClients, companies, createDb, heartbeatRuns, type Db } from "@paperclipai/db";
import { agents, applyPendingMigrations, authUsers, closeRegisteredClients, companies, companySecrets, createDb, dotAgentBindings,
dotRunnerAssignments, grokBotDispatches, grokBotRoutines, heartbeatRuns, issues, mcpOauthClients, mcpOauthGrants, nativeRunFinalizations,
type Db } from "@paperclipai/db";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
@@ -67,9 +69,15 @@ describeEmbeddedPostgres("reportRunFailure", () => {
vi.clearAllMocks();
vi.unstubAllEnvs();
process.env = inheritedEnv;
await db.delete(grokBotRoutines);
await db.delete(companySecrets);
await db.delete(nativeRunFinalizations);
await db.delete(heartbeatRuns);
await db.delete(issues);
await db.delete(agents);
await db.delete(companies);
await db.delete(mcpOauthClients);
await db.delete(authUsers);
});
afterAll(async () => {
@@ -335,6 +343,80 @@ describeEmbeddedPostgres("reportRunFailure", () => {
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
});
async function expiredGrokOffer(input: {
wakes: Array<{ status: "rejected" | "accepted" | "acknowledged" | "unknown"; httpStatus: number | null }>;
acceptedAt?: Date | null;
originalFailureCode?: string;
finalization?: boolean;
offers?: number;
}) {
await seedCompanyAndAgent({ adapterType: "paperclip_runner", adapterConfig: { provider: "grok_bot" } });
const userId = randomUUID();
await db.insert(authUsers).values({ id: userId, name: "Operator", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
await db.insert(mcpOauthClients).values({ id: userId, name: "Grok", redirectUris: ["https://client.example/callback"] });
const [grant] = await db.insert(mcpOauthGrants).values({ companyId, userId, clientId: userId, resource: "https://paperclip.example/mcp/runner",
purpose: "agent", agentId, scopes: ["paperclip:agent"] }).returning();
const [binding] = await db.insert(dotAgentBindings).values({ companyId, agentId, operatorId: userId, grantId: grant!.id,
providerKind: "grok_bot", status: "ready" }).returning();
const [secret] = await db.insert(companySecrets).values({ companyId, key: `grok-${binding!.id}`, name: "Grok routine" }).returning();
const [routine] = await db.insert(grokBotRoutines).values({ companyId, agentId, bindingId: binding!.id, bindingGeneration: 1,
botName: "Grok", secretId: secret!.id, secretVersion: 1 }).returning();
const [issue] = await db.insert(issues).values({ companyId, title: "Grok task", assigneeAgentId: agentId, status: "in_progress" }).returning();
const run = buildRun({ errorCode: "native_provider_terminal_failed", error: "Dot assignment acceptance or execution authority expired.",
nativeIssueId: issue!.id, runtimeMode: "native", executionStage: "preparing", nativePhase: "terminal_failure", exitCode: null, signal: null });
await db.insert(heartbeatRuns).values(run);
if (input.finalization !== false) {
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId, issueId: run.nativeIssueId!, phase: "terminal_failure",
failureCode: "native_provider_terminal_failed", failureDetail: { originalFailureCode: input.originalFailureCode ?? "dot_assignment_expired" } });
}
const offers = await db.insert(dotRunnerAssignments).values(Array.from({ length: input.offers ?? 1 }, () => ({ companyId, agentId,
bindingId: binding!.id, bindingGeneration: 1, runId: run.id, normalizedSessionId: randomUUID(), turnId: randomUUID(), controllerGeneration: 1,
catalogDigest: "fixture", projection: {}, status: "fenced" as const, acceptBy: new Date(Date.now() - 1000),
expiresAt: new Date(Date.now() + 3_600_000), acceptedAt: input.acceptedAt ?? null }))).returning();
const assignment = offers[0];
for (const wake of input.wakes) {
await db.insert(grokBotDispatches).values({ companyId, routineId: routine!.id, bindingId: binding!.id, bindingGeneration: 1,
grantId: grant!.id, routineRevision: 1, assignmentId: assignment!.id, sourceEventId: randomUUID(), kind: "mailbox_updated",
payload: {}, status: wake.status, httpStatus: wake.httpStatus, expiresAt: new Date(Date.now() + 600_000) });
}
return run;
}
it("keeps a Grok offer local when the endpoint rejected every work wake with HTTP 4xx", async () => {
const run = await expiredGrokOffer({ wakes: [{ status: "rejected", httpStatus: 400 }, { status: "rejected", httpStatus: 400 }] });
await reportRunFailure(db, run, { phase: "execute" });
expect(mockCaptureRunFailure).not.toHaveBeenCalled();
});
it.each([
["no wake was sent", { wakes: [] }],
["a wake was rejected locally without HTTP evidence", { wakes: [{ status: "rejected", httpStatus: 400 }, { status: "rejected", httpStatus: null }] }],
["the endpoint accepted a wake", { wakes: [{ status: "rejected", httpStatus: 400 }, { status: "acknowledged", httpStatus: 200 }] }],
["delivery is uncertain", { wakes: [{ status: "unknown", httpStatus: 503 }] }],
["the bot accepted the work", { wakes: [{ status: "rejected", httpStatus: 400 }], acceptedAt: new Date() }],
["the run failed for another provider reason", { wakes: [{ status: "rejected", httpStatus: 400 }], originalFailureCode: "provider_failed" }],
["the run has no finalization", { wakes: [{ status: "rejected", httpStatus: 400 }], finalization: false }],
["the run has more than one offer", { wakes: [{ status: "rejected", httpStatus: 400 }], offers: 2 }],
] as const)("keeps an expired Grok offer reportable when %s", async (_label, input) => {
const run = await expiredGrokOffer({ ...input, wakes: [...input.wakes] });
await reportRunFailure(db, run, { phase: "execute" });
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
});
it("keeps an expired Grok offer reportable when its receipts cannot be read", async () => {
const run = await expiredGrokOffer({ wakes: [{ status: "rejected", httpStatus: 400 }] });
const select = db.select.bind(db);
// The agent lookup succeeds; the receipt reads fail.
const spy = vi.spyOn(db, "select").mockImplementationOnce(select as typeof db.select)
.mockImplementation(() => { throw new Error("fixture receipt read failed"); });
try {
await reportRunFailure(db, run, { phase: "execute" });
} finally {
spy.mockRestore();
}
expect(mockCaptureRunFailure).toHaveBeenCalledOnce();
});
function aiSelectionRun(overrides: Partial<typeof heartbeatRuns.$inferSelect> = {}) {
return buildRun({
errorCode: "configuration_incomplete",
server/src/services/run-failure-report.ts
+28 -1
View File
@@ -1,5 +1,5 @@
import { and, eq } from "drizzle-orm";
import { agents, heartbeatRuns, type Db } from "@paperclipai/db";
import { agents, dotRunnerAssignments, grokBotDispatches, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { captureRunFailure, type RunFailureStatus } from "../sentry.js";
import {
collectRunFailureDiagnostics,
@@ -174,6 +174,27 @@ function isProviderQuotaFailure(run: HeartbeatRun, options: RunFailureReportOpti
options.phase !== "setup" && run.signal == null;
}
/**
* The Grok endpoint refused every work wake with an HTTP 4xx, so the offer
* expired unaccepted. Durable delivery receipts prove the external cause. A
* local rejection (no HTTP status), any delivered wake, or accepted work stays
* reportable.
*/
async function isRejectedExternalBotWake(db: Db, run: HeartbeatRun): Promise<boolean> {
if (run.status !== "failed" || run.runtimeMode !== "native" || run.errorCode !== "native_provider_terminal_failed") return false;
const [finalization] = await db.select({ failureDetail: nativeRunFinalizations.failureDetail }).from(nativeRunFinalizations)
.where(and(eq(nativeRunFinalizations.runId, run.id), eq(nativeRunFinalizations.companyId, run.companyId)));
if (finalization?.failureDetail?.originalFailureCode !== "dot_assignment_expired") return false;
const offers = await db.select({ id: dotRunnerAssignments.id, acceptedAt: dotRunnerAssignments.acceptedAt }).from(dotRunnerAssignments)
.where(and(eq(dotRunnerAssignments.runId, run.id), eq(dotRunnerAssignments.companyId, run.companyId)));
if (offers.length !== 1 || offers[0]!.acceptedAt) return false;
const wakes = await db.select({ status: grokBotDispatches.status, httpStatus: grokBotDispatches.httpStatus }).from(grokBotDispatches)
.where(and(eq(grokBotDispatches.assignmentId, offers[0]!.id), eq(grokBotDispatches.companyId, run.companyId),
eq(grokBotDispatches.kind, "mailbox_updated")));
return wakes.length > 0 && wakes.every((wake) =>
wake.status === "rejected" && wake.httpStatus != null && wake.httpStatus >= 400 && wake.httpStatus < 500);
}
/**
* Report a terminal run failure to Sentry. Returns at once for any status
* other than failures and unexpected started cancellations. Never throws — a Sentry failure or a
@@ -222,6 +243,12 @@ async function captureTerminalRunFailure(
.then((rows) => rows[0] ?? null);
if (isHermesConnectionFailure(run, options, agent?.adapterType)) return;
// An unreadable receipt proves nothing; keep the failure reportable.
const rejectedExternally = await isRejectedExternalBotWake(db, run).catch((err) => {
logger.warn({ err, runId: run.id }, "failed to read bot wake receipts for run failure report");
return false;
});
if (rejectedExternally) return;
const taskId = readTaskId(run);
if (!taskId) {