Files
PaperClipAI/server/src/services/issue-continuation-summary.ts
T
DottaandClaude 1e13c58a4e fix(heartbeat): name the cause of generic adapter failures and keep SQL text off the board (#20483)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Heartbeat run settlement records why a run failed, and the board
shows that record.
> - Runs that fail without a named code settle as `adapter_failed` (or
`setup_failed`) with no cause.
> - A user, an operator, and the restart chaos harness cannot tell what
went wrong from such a run.
> - Some run errors also show raw SQL statement text to users.
> - This pull request records a bounded `failureCause` code for generic
failures, shows it on the board, and keeps SQL text out of run errors.
> - The benefit is that a generic failure now names its cause, for
example "adapter failed (40001)", without leaking messages, SQL, or
secrets.

## Linked Issues or Issue Description

Refs #20385 (the nested-cause extraction this change reuses). Related:
#20421 (restart chaos universal checks U1-U4).

**What happened?**

The restart chaos audit of 79 saved waves found 30 failed runs with
`errorCode: "adapter_failed"` and a null or generic error. Every U3
("diagnosable") warning was `adapter_failed`. Nobody could tell what
failed. Some board run errors also showed raw SQL statement text from
wrapped database errors (Drizzle `Failed query: ... params: ...`).

**Expected behavior**

A run that fails with a generic code names a short, safe cause. The
board shows the cause next to the stop reason. Run error text never
shows SQL statements or their parameters.

**Steps to reproduce**

Make an adapter throw a wrapped database error (an `Error` whose nested
`cause` has `code: "40001"` and whose message has a Drizzle `Failed
query:` statement). Before this change, the run settles as
`adapter_failed` with no cause, and `heartbeat_runs.error` contains the
SQL statement and parameters. The new tests in
`server/src/services/heartbeat/run-completion.test.ts` reproduce this.

## What Changed

- Runner: move the #20385 event-commit cause extraction into
`safeErrorCauseCode`
(`packages/paperclip-runner/src/control-plane/safe-error-cause.ts`) and
export it. The Runner's own diagnostics use the same function, unchanged
in behavior.
- Shared: add the `failureCause` contract
(`packages/shared/src/run-failure-cause.ts`): one `[A-Za-z0-9_.:-]`
token of at most 80 characters, `normalizeRunFailureCause`,
`readRunFailureCause`, and `withoutSqlStatementText`.
- Server: add `server/src/services/heartbeat/run-failure-cause.ts`.
Settlement records `resultJson.failureCause` only when the code is
generic (`adapter_failed`, `setup_failed`). `errorCode` stays the same.
- Thrown errors (`failRun`, `failRunSetup`): `provider_http_<status>`,
else `provider_<category>`, else the first safe code in the error and up
to three nested causes (`40001`, `ECONNRESET`), else the error class
name, else `unknown_error`.
- Adapter results without a code (`completeRun`): the error family,
`signal_<name>`, `process_cancellation_failed`, `exit_code_<n>`,
`adapter_reported_error`, or `unknown_error`.
- The failure log lines (`heartbeat execution failed`, `heartbeat
execution setup failed`, and a new `heartbeat run failed` warning for
generic adapter results) include `failureCause` and `errorCode`.
- Server: replace SQL statement text in run error text with "Database
query failed (<cause>)" at all three settlement paths, including the
cancelled branch. The execution blocker's `runError`, the attention
inbox (failed-run excerpt and metadata), and continuation summaries
apply the same replacement on read, so older rows also stop showing SQL
there.
- Server: Retry admission and failed-preparation replay accept only a
closed set of `resultJson` keys. `failureCause` is added to each of
those lists (`task-retry.ts`, `chat-channels/failed-run-retry.ts`,
`cancelled-native-startup.ts`, `fresh-workspace-continuation.ts`). It is
a diagnostic code with no execution or replay authority, so a
failed-preparation run stays retryable.
- Server: `runsForIssue` returns a validated top-level `failureCause`.
It is null for older rows, non-string JSON values, and malformed values.
- UI: `RunForIssue.failureCause`; `IssueRunLedger` shows "adapter failed
(40001)". Older runs keep "adapter failed". The Storybook run-ledger
fixture has a failed run with a cause.
- Restart chaos harness: run samples keep `failureCause`. U3 treats a
generic code with a specific `failureCause` as diagnosable. A bare
`Error`, `unknown_error`, or the no-code fallback
`adapter_reported_error` still counts as vague. U3 stays a warning.
- Docs: `doc/run-log-events.md` and the harness README describe the
field and the SQL replacement.

## Verification

- Red before the fix: the four new `run-completion.test.ts` cases failed
against `origin/master`'s `run-completion.ts` (no `failureCause`, SQL
text in `error`).
- Green:
- `server/src/services/heartbeat/run-completion.test.ts`: all tests
passed (real embedded PostgreSQL). New cases cover a nested `40001`
three levels deep for execution and setup failures, SQL and a secret
parameter absent from `error` and `resultJson`, the bound (an
81-character code and a SQL "code" fall back to the class name),
`exit_code_137` for a code-less adapter result, a cancelled result with
SQL text, no cause for a named code, and the log line. On this loaded
host the suite's `afterAll` database teardown hit its 30 s hook limit
after every test passed (the known local teardown limit, also seen in
#20385). CI runs the suite.
- `server/src/services/heartbeat/run-failure-cause.test.ts` (4),
`server/src/__tests__/activity-service.test.ts` (recorded, legacy,
malformed, overlong, boolean, and number cause projections),
`server/src/__tests__/issue-continuation-summary.test.ts` (legacy SQL
row).
- `packages/shared/src/run-failure-cause.test.ts` (6),
`ui/src/components/IssueRunLedger.test.tsx` (21), restart chaos
`universal.test.ts` and `observer.test.ts`.
-
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`:
207 passed (the stall-diagnostic cases still read `40001` and
`ECONNRESET`).
- Typecheck: shared, runner, server (all layers), UI, restart chaos
harness. `pnpm check:token-gates`: all gates clean.
- `pnpm verify` returned exit 2 (global doc input changed); required CI
supplies broad coverage.
- CI on `6295f8d` found that the new key made failed-preparation runs
ineligible for Retry (`chat-channels.integration.test.ts` saved
board-message Retry got 409;
`workspace-recovery-preparation-dispatch.test.ts` dispatched 0 Retries).
Both reproduced locally and pass after the allowlist fix, with the other
459 Retry cases in `chat-channels.integration.test.ts` and all 25 in
`workspace-recovery-preparation-dispatch.test.ts`.
- No chaos waves were run. Storybook was not opened in a browser; the
fixture typechecks and the label is covered by the UI test.

## Risks

- Low risk. `errorCode` values do not change, so retry, recovery, and
comparison logic is unaffected.
- No schema change and no migration. `failureCause` is a new optional
`resultJson` key. Rows saved by the previous release have no key and
render as before; a new row read by an old server is ignored.
- The SQL replacement only acts on a Drizzle `Failed query:` marker, or
on `$n` placeholder text with a statement shape. Ordinary error text
does not change. When it acts, it also drops the text after the
statement.
- Code that checks `resultJson` against a closed key list must allow
`failureCause`. I found and updated the four lists on the Retry path; a
list I missed would refuse a Retry for a run with a generic failure.
- This is the run log / instance database path. It adds no Telemetry
event.
- The server failure log line still logs the raw error message
(unchanged behavior); only the board-facing text is changed.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file editing) and code execution for
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:14:53 -06:00

294 lines
11 KiB
TypeScript

import { and, eq } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { documents, issueDocuments, issues } from "@paperclipai/db";
import { ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY, readRunFailureCause, withoutSqlStatementText, type SourceTrustMetadata } from "@paperclipai/shared";
import { documentService } from "./documents.js";
import { summarizeRunErrorForModel } from "./heartbeat-run-summary.js";
export { ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY };
export const ISSUE_CONTINUATION_SUMMARY_TITLE = "Continuation Summary";
export const ISSUE_CONTINUATION_SUMMARY_MAX_BODY_CHARS = 8_000;
const SUMMARY_SECTION_MAX_CHARS = 1_200;
const PATH_CANDIDATE_RE = /(?:^|[\s`"'(])((?:server|ui|packages|doc|scripts|\.github)\/[A-Za-z0-9._/-]+)/g;
const WAITING_FOR_REVIEW_OR_APPROVAL_RE =
/\bwait(?:ing)? for\b.{0,160}\b(?:review(?:er)?(?: feedback)?|approval|board|human|user|operator)\b/i;
type IssueSummaryInput = {
id: string;
identifier: string | null;
title: string;
description: string | null;
status: string;
priority: string;
};
type RunSummaryInput = {
id: string;
status: string;
error: string | null;
errorCode?: string | null;
resultJson?: Record<string, unknown> | null;
stdoutExcerpt?: string | null;
stderrExcerpt?: string | null;
finishedAt?: Date | null;
};
type AgentSummaryInput = {
id: string;
name: string;
adapterType: string | null;
};
export type IssueContinuationSummaryDocument = {
key: typeof ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY;
title: string | null;
body: string;
latestRevisionId: string | null;
latestRevisionNumber: number;
sourceTrust: SourceTrustMetadata | null;
updatedAt: Date;
};
function truncateText(value: string, maxChars: number) {
const trimmed = value.trim();
if (trimmed.length <= maxChars) return trimmed;
return `${trimmed.slice(0, Math.max(0, maxChars - 20)).trimEnd()}\n[truncated]`;
}
function asNonEmptyString(value: unknown) {
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
}
function readResultSummary(resultJson: Record<string, unknown> | null | undefined) {
if (!resultJson || typeof resultJson !== "object" || Array.isArray(resultJson)) return null;
return (
asNonEmptyString(resultJson.summary) ??
asNonEmptyString(resultJson.result) ??
asNonEmptyString(resultJson.message) ??
asNonEmptyString(resultJson.error) ??
null
);
}
function extractMarkdownSection(markdown: string | null | undefined, heading: string) {
if (!markdown) return null;
const escaped = heading.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const re = new RegExp(`^##\\s+${escaped}\\s*$([\\s\\S]*?)(?=^##\\s+|(?![\\s\\S]))`, "im");
const match = re.exec(markdown);
const section = match?.[1]?.trim();
return section ? truncateText(section, SUMMARY_SECTION_MAX_CHARS) : null;
}
function extractPathCandidates(...texts: Array<string | null | undefined>) {
const seen = new Set<string>();
for (const text of texts) {
if (!text) continue;
for (const match of text.matchAll(PATH_CANDIDATE_RE)) {
const path = match[1]?.replace(/[),.;:]+$/, "");
if (path) seen.add(path);
if (seen.size >= 12) break;
}
if (seen.size >= 12) break;
}
return [...seen];
}
function inferMode(issue: IssueSummaryInput, run: RunSummaryInput) {
if (issue.status === "done" || issue.status === "in_review") return "review";
if (run.status === "failed" || run.status === "timed_out" || run.status === "cancelled" || run.status === "interrupted") return "implementation";
if (issue.status === "backlog" || issue.status === "todo") return "plan";
return "implementation";
}
function inferNextAction(issue: IssueSummaryInput, run: RunSummaryInput, previousNextAction: string | null) {
if (issue.status === "done") return "Review the completed issue output and close any remaining follow-up comments.";
if (issue.status === "in_review") return "Wait for reviewer feedback or approval before continuing executor work.";
if (run.status === "failed" || run.status === "timed_out") {
return "Inspect the failed run, fix the cause, and resume from the most recent concrete action above.";
}
if (run.status === "cancelled") return "Confirm the cancellation reason before starting another run.";
return previousNextAction ?? "Resume implementation from the acceptance criteria, latest comments, and this summary.";
}
function bulletList(items: string[], empty: string) {
if (items.length === 0) return `- ${empty}`;
return items.map((item) => `- ${item}`).join("\n");
}
function extractPreviousNextAction(previousBody: string | null | undefined) {
const section = extractMarkdownSection(previousBody, "Next Action");
if (!section) return null;
return section
.split(/\r?\n/)
.map((line) => line.replace(/^[-*]\s+/, "").trim())
.find(Boolean) ?? null;
}
export function extractContinuationSummaryNextAction(body: string | null | undefined) {
return extractPreviousNextAction(body);
}
export function continuationSummaryParksExecutor(body: string | null | undefined) {
const nextAction = extractContinuationSummaryNextAction(body);
if (!nextAction) return false;
return WAITING_FOR_REVIEW_OR_APPROVAL_RE.test(nextAction);
}
export function buildContinuationSummaryMarkdown(input: {
issue: IssueSummaryInput;
run: RunSummaryInput;
agent: AgentSummaryInput;
previousSummaryBody?: string | null;
}) {
const { issue, run, agent } = input;
const resultSummary = readResultSummary(run.resultJson);
const terminalFailure = run.resultJson?.terminalSessionFailure;
const terminalFailureCategory = terminalFailure && typeof terminalFailure === "object" && !Array.isArray(terminalFailure)
? ((terminalFailure as Record<string, unknown>).category ?? "unknown")
: null;
// Rows saved before SQL text was kept out of run errors still carry it.
const modelError = summarizeRunErrorForModel(
run.error ? withoutSqlStatementText(run.error, readRunFailureCause(run.resultJson)) : run.error, terminalFailureCategory);
const recentActions = [
`Run \`${run.id}\` finished with status \`${run.status}\`${run.finishedAt ? ` at ${run.finishedAt.toISOString()}` : ""}.`,
resultSummary ? truncateText(resultSummary, SUMMARY_SECTION_MAX_CHARS) : "No adapter-provided result summary was captured for this run.",
];
if (modelError) {
recentActions.push(`Latest run error${run.errorCode ? ` (${run.errorCode})` : ""}: ${truncateText(modelError, 500)}`);
}
const paths = extractPathCandidates(resultSummary, run.stdoutExcerpt, run.stderrExcerpt, input.previousSummaryBody);
const objective = extractMarkdownSection(issue.description, "Objective") ?? issue.description?.trim() ?? "No objective captured.";
const acceptanceCriteria = extractMarkdownSection(issue.description, "Acceptance Criteria") ?? "No explicit acceptance criteria captured.";
const mode = inferMode(issue, run);
const nextAction = inferNextAction(issue, run, extractPreviousNextAction(input.previousSummaryBody));
const body = [
"# Continuation Summary",
"",
`- Issue: ${issue.identifier ?? issue.id} — ${issue.title}`,
`- Status: ${issue.status}`,
`- Priority: ${issue.priority}`,
`- Current mode: ${mode}`,
`- Last updated by run: ${run.id}`,
`- Agent: ${agent.name} (${agent.adapterType ?? "unknown"})`,
"",
"## Objective",
"",
truncateText(objective, SUMMARY_SECTION_MAX_CHARS),
"",
"## Acceptance Criteria",
"",
acceptanceCriteria,
"",
"## Recent Concrete Actions",
"",
bulletList(recentActions, "No recent actions captured."),
"",
"## Files / Routes Touched",
"",
bulletList(paths.map((path) => `\`${path}\``), "No file or route paths were detected in the captured run summary."),
"",
"## Commands Run",
"",
bulletList(
[
`Heartbeat run \`${run.id}\` invoked adapter \`${agent.adapterType ?? "unknown"}\`.`,
"Detailed shell/tool commands remain in the run log and transcript.",
],
"No command metadata captured.",
),
"",
"## Blockers / Decisions",
"",
bulletList(
run.error
? [`Latest run ended with \`${run.status}\`; inspect the error before continuing.`]
: ["No new blocker was recorded by the latest run."],
"No blockers or decisions captured.",
),
"",
"## Next Action",
"",
`- ${nextAction}`,
].join("\n");
return truncateText(body, ISSUE_CONTINUATION_SUMMARY_MAX_BODY_CHARS);
}
export async function getIssueContinuationSummaryDocument(
db: Db,
issueId: string,
): Promise<IssueContinuationSummaryDocument | null> {
const row = await db
.select({
key: issueDocuments.key,
title: documents.title,
body: documents.latestBody,
latestRevisionId: documents.latestRevisionId,
latestRevisionNumber: documents.latestRevisionNumber,
sourceTrust: documents.sourceTrust,
updatedAt: documents.updatedAt,
})
.from(issueDocuments)
.innerJoin(documents, eq(issueDocuments.documentId, documents.id))
.where(and(eq(issueDocuments.issueId, issueId), eq(issueDocuments.key, ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY)))
.then((rows) => rows[0] ?? null);
if (!row) return null;
return {
key: ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY,
title: row.title,
body: row.body,
latestRevisionId: row.latestRevisionId,
latestRevisionNumber: row.latestRevisionNumber,
sourceTrust: row.sourceTrust ?? null,
updatedAt: row.updatedAt,
};
}
export async function refreshIssueContinuationSummary(input: {
db: Db;
issueId: string;
run: RunSummaryInput;
agent: AgentSummaryInput;
}) {
const { db, issueId, run, agent } = input;
const [issue, existing] = await Promise.all([
db
.select({
id: issues.id,
conversationAgentId: issues.conversationAgentId,
identifier: issues.identifier,
title: issues.title,
description: issues.description,
status: issues.status,
priority: issues.priority,
})
.from(issues)
.where(eq(issues.id, issueId))
.then((rows) => rows[0] ?? null),
getIssueContinuationSummaryDocument(db, issueId),
]);
if (!issue || issue.conversationAgentId) return null;
const body = buildContinuationSummaryMarkdown({
issue,
run,
agent,
previousSummaryBody: existing?.body ?? null,
});
const result = await documentService(db).upsertIssueDocument({
issueId,
key: ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY,
title: ISSUE_CONTINUATION_SUMMARY_TITLE,
format: "markdown",
body,
baseRevisionId: existing?.latestRevisionId ?? null,
changeSummary: `Refresh continuation summary after run ${run.id}`,
createdByAgentId: agent.id,
createdByRunId: run.id,
});
return result.document;
}