fix(sentry): preserve run failure stacks and diagnostic context (#14585)

Builds on merged #14575 and targets `master`.

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators use optional Sentry reports to diagnose failed agent runs.
> - The shared reporter currently converts each saved error message into
a new exception.
> - This loses the original stack and cause chain, and omits available
adapter and provider diagnostics.
> - This pull request selects, redacts, and bounds useful diagnostics
before it sends them to Sentry.
> - Operators can locate the failing operation and correlate upstream
failures without copying arbitrary run data.

## Linked Issues or Issue Description

Refs #14573. That change preserves ACP provider failure details in the
run record.
Refs #14575. That merged PR adds recorded exit codes and signals. This
PR covers stacks, causes, and structured diagnostics without duplicating
those fields.

A thrown setup or adapter error currently appears in Sentry with the
reporter's stack. A saved provider failure can contain useful details
that never reach the Sentry event. Both cases use the same shared
reporting path.

## What Changed

- Pass caught setup and execution exceptions, and structured adapter
error metadata, through successful terminal status transitions.
- Snapshot a fixed set of execution, adapter, provider, and exception
fields. Preserve up to four exceptions in the cause chain.
- Remove registered secret values, declared runtime environment
credentials, unknown inherited environment values and encoded credential
forms, and credential patterns before truncation. Mark truncated fields
and bound provider details and stacks.
- Rebuild sanitized Sentry exceptions with the original stacks and
causes. Use an adapter stack preview when available. Omit a fabricated
reporting stack when the source has no stack.
- Keep contexts local to each event. Preserve the optional DSN gate and
existing error-code/adapter fingerprint.
- Document the fields, limits, and omitted data.
- Settle leftover chat fixture outbox rows only after assertions and
worker shutdown, so subsequent tests cannot claim earlier cases’ pending
actions or provider I/O. This fixes the CI shard contamination exposed
during verification; production chat behavior and test timeouts are
unchanged.

## Verification

- `pnpm -r typecheck` passed.
- `pnpm build` passed.
- 86 focused diagnostic, Sentry, and startup tests passed after the
rebase, including the real `@sentry/node@10.71.0` SDK with an in-memory
transport.
- Tests cover cause chains, HTTP status and request IDs, long provider
details, secret redaction, failed secret resolution, cyclic causes, size
limits, and context isolation.
- Five targeted heartbeat integration cases passed, including thrown and
returned errors containing an opaque environment-bound credential. The
earlier full heartbeat integration run also passed its integration
cases.
- The focused suite also passed with an unknown inherited environment
value set to `1`; reporting tests use a controlled environment and
separately verify short-secret redaction.
- Before the fixture cleanup, the full chat shard reproduced the CI
Telegram timeout at `pending recovery before restart` (354 passed, 1
failed). After the cleanup, the same shard passed all 355 tests. No
timeout or production behavior changed.
- Latest-head CI (`0ae9e70df321d66dda025c3a7ba4787e169e0a4f`) passed
typecheck, build, all 12 server shards, all 3 chat shards, workspace and
serialized suites, browser shards, Runner checks, canary validation, the
real Sentry SDK contract, and security checks. Required `ci / verify`
and `ci / e2e` passed.
- The branch has been rebased onto `master` after #14575 merged. All 55
applicable checks passed on this head (2 unrelated Storybook checks
skipped). Greptile re-reviewed the current 11-file diff at 5/5 with no
findings or unresolved review threads.
- Local monolithic full-suite attempts were interrupted to apply fixes;
full-suite success is not claimed from those runs.

## Risks

- Error messages and stacks can contain credentials. The reporter uses
existing redactors and the run's encrypted secret registry, reads only
known fields, and skips capture if registered-secret resolution fails.
- Unknown environment values remain private by default. Unrecognized
short values can mask benign matches; known public settings are
explicitly allowed.
- Diagnostic text is bounded and can be truncated. Truncation is
explicit. Data discarded upstream cannot be recovered.
- These additional fields go to the operator's configured Sentry
endpoint. Arbitrary request/response objects, headers, configuration,
prompts, and stdout/stderr are not copied.

## Model Used

OpenAI GPT-6 through Codex, with tool use and code execution. The exact
serving model ID, context window, and configured reasoning level are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-09-29 12:54:12 -05:00
1 parent 2de43fc909
commit b3eb03fcba
11 files changed
+667 -36

No files matched your search

+31 -1
View File
@@ -56,6 +56,7 @@
// `instrumentation.ts`.
import os from "node:os";
import type { RunFailureDiagnostics } from "./services/run-failure-diagnostics.js";
import { readBuildCommit } from "./build-commit.js";
import { checkExactPeerVersions } from "./peer-version-check.js";
import { resolveSentryDsns } from "./sentry-dsn.js";
@@ -132,6 +133,8 @@ export interface RunFailureEvent {
agentAdapter: string;
/** The run status that triggered this report. */
runStatus: RunFailureStatus;
/** Bounded, redacted diagnostics selected by the run failure reporter. */
diagnostics?: RunFailureDiagnostics;
/** Recorded process exit evidence, when available. Validated before capture. */
exitCode?: number | null;
signal?: string | null;
@@ -174,7 +177,33 @@ export function captureRunFailure(event: RunFailureEvent): void {
// Sentry's async scope isolation is absent when OTel setup is skipped.
// A withScope mutation can then persist into unrelated later captures.
// Pass these fields on this event only; do not mutate the ambient scope.
handle.captureException(new Error(event.errorMessage), {
const diagnostics = event.diagnostics;
const contexts: Record<string, Record<string, unknown>> = {};
if (diagnostics) {
contexts.run_execution = { ...diagnostics.execution, truncatedFields: diagnostics.truncatedFields };
if (Object.keys(diagnostics.adapter).length) contexts.adapter_failure = diagnostics.adapter;
if (Object.keys(diagnostics.provider).length) contexts.provider_failure = diagnostics.provider;
diagnostics.exceptions.forEach(({ name, code, status, requestId }, index) => {
contexts[`run_exception_${index}`] = { name, code, status, requestId };
});
}
// Rebuild only sanitized exception fields. Passing a raw SDK Error can
// serialize its request/response, headers, or other enumerable properties.
let cause: Error | undefined;
for (const entry of [...(diagnostics?.exceptions ?? [])].reverse()) {
const error: Error = new Error(entry.message ?? event.errorMessage, cause ? { cause } : undefined);
error.name = entry.name ?? "Error";
error.stack = entry.stack;
cause = error;
}
const exception = cause ?? new Error(event.errorMessage);
if (!cause) {
// A saved adapter result is not a thrown Error. Do not pretend that the
// reporter's own stack is the failure location.
exception.stack = typeof diagnostics?.adapter.stackPreview === "string"
? diagnostics.adapter.stackPreview : undefined;
}
handle.captureException(exception, {
tags: {
run_id: event.runId,
task_id: event.taskId,
@@ -183,6 +212,7 @@ export function captureRunFailure(event: RunFailureEvent): void {
run_status: event.runStatus,
},
contexts: {
...contexts,
run_failure: {
taskId: event.taskId,
runId: event.runId,