feat: structure and humanize recovery notices (#11075)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip posts system comments when automatic run recovery cannot
continue
> - These comments currently mix the main event with recovery
identifiers and routing details
> - The task chat shell also renders these comments as large raw text
blocks
> - Operators need a short explanation first and inspectable evidence on
demand
> - This pull request emits structured recovery notices and renders them
as compact humanized rows
> - The benefit is a quieter task thread that keeps the full recovery
evidence available

## Linked Issues or Issue Description

Related prior extraction source: #11070. This pull request replaces only
its structured recovery notice slice with a focused branch based on
current master.

**What existing behavior does this improve?**

Paperclip recovery escalations and the experimental task chat
system-comment renderer.

**Current behavior**

Recovery escalation comments put action identifiers, owner details, run
details, and failure codes into the visible markdown body. The task chat
shell renders the complete system comment as a large text block.

**Proposed behavior**

The server emits a short system notice with typed metadata sections. The
task chat shell classifies known recovery families and renders one
compact row. An operator can expand the row to inspect the full body and
metadata.

**Reason and benefit**

The main thread stays readable during repeated recovery activity. Typed
links and evidence remain available without exposing raw failure text in
the default view.

**Breaking changes**

The visible recovery comment body is shorter. Recovery action
deduplication now reads the structured metadata and still recognizes
legacy body markers. No API schema or database migration changes.

## What Changed

- Emit stranded recovery escalations with `system_notice` presentation
and typed recovery, owner, run, and failure-code metadata.
- Share bounded metadata row builders across recovery notice producers
and preserve legacy deduplication compatibility.
- Humanize known recovery notice families and render compact expandable
task-chat rows.
- Route system-authored comments ahead of derived agent authorship so
recovery notices do not appear as agent bubbles.
- Add focused server and UI regression coverage.

## Verification

- `pnpm check:token-gates` — 3/3 clean.
- `pnpm --filter @paperclipai/server typecheck` — passed.
- `pnpm --filter @paperclipai/ui typecheck` — passed.
- `pnpm --filter @paperclipai/shared typecheck` — passed.
- `pnpm --filter @paperclipai/shared exec vitest run
src/validators/issue.test.ts` — 32 tests passed.
- `pnpm --filter @paperclipai/server exec vitest run
src/services/recovery/stranded-notice.test.ts
src/__tests__/issue-recovery-actions.test.ts` — 57 tests passed.
- `pnpm --filter @paperclipai/server exec vitest run
src/services/recovery/successful-run-handoff.test.ts
src/services/recovery/stranded-notice.test.ts` — 39 tests passed.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t 'escalates an
exhausted failed successful-run handoff without using generic
continuation recovery first|escalates an exhausted successful handoff
run that still leaves no disposition'` — 2 tests passed.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t 'blocks assigned
todo work after the one automatic dispatch recovery was already used'` —
passed.
- `pnpm --filter @paperclipai/ui exec vitest run
src/lib/system-notice-humanizer.test.ts
src/components/task-chat/TaskChatSystemNotice.test.tsx
src/components/task-chat/task-chat-adapter.test.ts` — 15 tests passed.
- Storybook visual baselines were not updated because this chat-shell
path has no affected snapshot baseline. Focused rendering tests and
token gates cover this change.

## Risks

- Consumers that parse recovery action identifiers from comment markdown
must move to structured metadata. Server deduplication remains backward
compatible with legacy comments.
- The humanizer uses stable recovery-family phrases. Unknown notices use
a generic truncated first-sentence fallback.
- The UI changes only the experimental task chat presentation. The
stored comment body and expanded metadata remain available.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex with GPT-5, reasoning mode, repository tools, shell
execution, and GitHub integration. The runtime did not expose a
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
scotttongandPaperclip authored and GitHub committed 2026-08-07 18:41:52 -07:00
1 parent 4e9a78db58
commit cc35c3c395
25 files changed
+1166 -308

No files matched your search

+16 -102
View File
@@ -224,6 +224,12 @@ import {
readContinuationAttempt,
} from "./recovery/index.js";
import { isAutomaticRecoverySuppressedByPauseHold } from "./recovery/pause-hold-guard.js";
import {
buildConfigurationIncompleteRecoveryNoticeSeed,
buildExecutionReviewParticipantRecoveryNoticeSeed,
buildImmediateExecutionPathRecoveryNoticeSeed,
buildWorkspaceValidationRecoveryNoticeSeed,
} from "./recovery/stranded-notice.js";
import {
recoveryAssigneeAdapterOverrides,
withRecoveryModelProfileHint,
@@ -3616,30 +3622,6 @@ export function summarizeHeartbeatRunListResultJson(input: {
return Object.keys(summary).length > 0 ? summary : null;
}
function summarizeRunFailureForIssueComment(
run: Pick<typeof heartbeatRuns.$inferSelect, "error" | "errorCode"> | null | undefined,
) {
if (!run) return null;
const errorCode = readNonEmptyString(run.errorCode)?.trim() ?? null;
const rawError = readNonEmptyString(run.error)?.trim() ?? null;
const apiMessageMatch = rawError?.match(/"message"\s*:\s*"([^"]+)"/);
const firstLine = rawError
?.split(/\r?\n/)
.map((line) => line.trim())
.find(Boolean) ?? null;
const summarySource = apiMessageMatch?.[1] ?? firstLine;
const summary =
summarySource && summarySource.length > 240
? `${summarySource.slice(0, 237)}...`
: summarySource;
if (errorCode && summary) return ` Latest retry failure: \`${errorCode}\` - ${summary}.`;
if (errorCode) return ` Latest retry failure: \`${errorCode}\`.`;
if (summary) return ` Latest retry failure: ${summary}.`;
return null;
}
function didAutomaticRecoveryFail(
latestRun: Pick<typeof heartbeatRuns.$inferSelect, "status" | "contextSnapshot"> | null,
expectedRetryReason:
@@ -16379,73 +16361,6 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
}
}
function buildImmediateExecutionPathRecoveryComment(input: {
status: "todo" | "in_progress";
latestRun: Pick<typeof heartbeatRuns.$inferSelect, "error" | "errorCode"> | null | undefined;
}) {
const failureSummary = summarizeRunFailureForIssueComment(input.latestRun);
if (input.status === "todo") {
return (
"Paperclip automatically retried dispatch for this assigned `todo` issue during terminal run recovery, " +
`but it still has no live execution path.${failureSummary ?? ""} ` +
"Moving it to `blocked` so it is visible for intervention."
);
}
return (
"Paperclip automatically retried continuation for this assigned `in_progress` issue during terminal run " +
`recovery, but it still has no live execution path.${failureSummary ?? ""} ` +
"Moving it to `blocked` so it is visible for intervention."
);
}
function buildWorkspaceValidationRecoveryComment(input: {
latestRun:
| Pick<typeof heartbeatRuns.$inferSelect, "error" | "errorCode" | "resultJson">
| null
| undefined;
}) {
const failureSummary = summarizeRunFailureForIssueComment(input.latestRun);
const validationReason = readNonEmptyString(
readWorkspaceValidationPayloadFromRun(input.latestRun).reason,
);
if (validationReason === "git_worktree_base_materialization_failed") {
return (
"Paperclip stopped before launching the local adapter because the project workspace checkout could not be prepared " +
`(for example the repository clone failed).${failureSummary ?? ""} ` +
"Moving it to `blocked` with a source-scoped recovery action so the repository URL, clone access, or configured local cwd can be repaired before resuming."
);
}
return (
"Paperclip stopped before launching the local adapter because the issue workspace failed validation. " +
`This prevents git-sensitive adapters from running in an unrelated fallback cwd.${failureSummary ?? ""} ` +
"Moving it to `blocked` with a source-scoped recovery action so the workspace link, cwd, or git checkout can be repaired before resuming."
);
}
function buildConfigurationIncompleteRecoveryComment(input: {
latestRun: Pick<typeof heartbeatRuns.$inferSelect, "error" | "errorCode"> | null | undefined;
}) {
const failureSummary = summarizeRunFailureForIssueComment(input.latestRun);
return (
"Paperclip stopped before dispatching the adapter because required secret/env bindings are missing. " +
`Resolving them as a runtime failure would only produce repeated opaque setup failures.${failureSummary ?? ""} ` +
"Moving it to `blocked` with a source-scoped recovery action so an operator can bind the missing secret(s) before resuming."
);
}
function buildExecutionReviewParticipantRecoveryComment(input: {
latestRun: Pick<typeof heartbeatRuns.$inferSelect, "error" | "errorCode"> | null | undefined;
}) {
const failureSummary = summarizeRunFailureForIssueComment(input.latestRun);
return (
"Paperclip retried the pending execution-review participant once, but the review stage still has no completed decision " +
`or live reviewer run.${failureSummary ?? ""} ` +
"Moving it to `blocked` with a source-scoped recovery action so the recovery owner can repair the reviewer runtime, " +
"restore the review stage, or record an intentional manual resolution."
);
}
async function releaseIssueExecutionAndPromote(
run: typeof heartbeatRuns.$inferSelect,
options: { suppressImmediateRecovery?: boolean } = {},
@@ -16579,9 +16494,9 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
kind: "blocked" as const,
issue,
previousStatus: issue.status,
comment: configurationIncomplete
? buildConfigurationIncompleteRecoveryComment({ latestRun: run })
: buildWorkspaceValidationRecoveryComment({ latestRun: run }),
notice: configurationIncomplete
? buildConfigurationIncompleteRecoveryNoticeSeed()
: buildWorkspaceValidationRecoveryNoticeSeed(),
recoveryCause: configurationIncomplete
? CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE
: WORKSPACE_VALIDATION_RECOVERY_CAUSE,
@@ -16925,7 +16840,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
kind: "blocked" as const,
issue,
previousStatus: issue.status,
comment: buildExecutionReviewParticipantRecoveryComment({ latestRun: run }),
notice: buildExecutionReviewParticipantRecoveryNoticeSeed(),
recoveryCause: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_CAUSE,
recoveryOwnerAgentId: currentParticipant.agentId,
};
@@ -17046,19 +16961,18 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
if (shouldBlockImmediately) {
const workspaceValidationFailure = isWorkspaceValidationFailedRun(run);
const configurationIncompleteFailure = isConfigurationIncompleteFailedRun(run);
const comment = workspaceValidationFailure
? buildWorkspaceValidationRecoveryComment({ latestRun: run })
const notice = workspaceValidationFailure
? buildWorkspaceValidationRecoveryNoticeSeed()
: configurationIncompleteFailure
? buildConfigurationIncompleteRecoveryComment({ latestRun: run })
: buildImmediateExecutionPathRecoveryComment({
? buildConfigurationIncompleteRecoveryNoticeSeed()
: buildImmediateExecutionPathRecoveryNoticeSeed({
status: issue.status as "todo" | "in_progress",
latestRun: run,
});
return {
kind: "blocked" as const,
issue,
previousStatus: issue.status,
comment,
notice,
recoveryCause: workspaceValidationFailure
? WORKSPACE_VALIDATION_RECOVERY_CAUSE
: configurationIncompleteFailure
@@ -17168,7 +17082,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
issue: promotionResult.issue,
previousStatus: promotionResult.previousStatus as "todo" | "in_progress" | "in_review",
latestRun: run,
comment: promotionResult.comment,
notice: promotionResult.notice,
recoveryCause:
promotionResult.recoveryCause === WORKSPACE_VALIDATION_RECOVERY_CAUSE
? WORKSPACE_VALIDATION_RECOVERY_CAUSE
+14
View File
@@ -58,8 +58,22 @@ export {
findExistingFinishSuccessfulRunHandoffWake,
isSuccessfulRunHandoffValidPathSkip,
isSuccessfulRunHandoffRequiredNoticeBody,
noticeMetadataReferencesRecoveryAction,
} from "./successful-run-handoff.js";
export type {
SuccessfulRunHandoffNotice,
SuccessfulRunHandoffDecision,
} from "./successful-run-handoff.js";
export {
DEFAULT_STRANDED_RECOVERY_NOTICE_BODY,
buildConfigurationIncompleteRecoveryNoticeSeed,
buildExecutionReviewParticipantRecoveryNoticeSeed,
buildExecutionReviewParticipantUnavailableNoticeSeed,
buildImmediateExecutionPathRecoveryNoticeSeed,
buildStrandedRecoveryEscalationNotice,
buildWorkspaceValidationRecoveryNoticeSeed,
} from "./stranded-notice.js";
export type {
StrandedRecoveryEscalationNotice,
StrandedRecoveryNoticeSeed,
} from "./stranded-notice.js";
@@ -0,0 +1,68 @@
import type { IssueCommentMetadata, IssueCommentPresentation } from "@paperclipai/shared";
export type NoticeMetadataRow = IssueCommentMetadata["sections"][number]["rows"][number];
export type NoticeMetadataSection = IssueCommentMetadata["sections"][number];
export function metadataText(value: unknown, fallback = "unknown") {
const text = typeof value === "string" ? value.trim() : value == null ? "" : String(value).trim();
const resolved = text.length > 0 ? text : fallback;
return resolved.length > 2000 ? `${resolved.slice(0, 1997)}...` : resolved;
}
export function keyValueRow(label: string, value: unknown): NoticeMetadataRow {
return { type: "key_value", label, value: metadataText(value) };
}
export function issueLinkRow(
label: string,
issue: { id: string; identifier: string | null; title: string } | null | undefined,
): NoticeMetadataRow {
if (!issue) return keyValueRow(label, "unknown");
return {
type: "issue_link",
label,
issueId: issue.id,
identifier: issue.identifier,
title: issue.title,
};
}
export function runLinkRow(
label: string,
run: { id: string; status: string; agentId?: string | null } | null | undefined,
): NoticeMetadataRow {
if (!run) return keyValueRow(label, "unknown");
return {
type: "run_link",
label,
runId: run.id,
...(run.agentId ? { agentId: run.agentId } : {}),
title: run.status,
};
}
export function agentLinkRow(
label: string,
agent: { id: string; name: string | null } | null | undefined,
): NoticeMetadataRow {
if (!agent) return keyValueRow(label, "unknown");
return {
type: "agent_link",
label,
agentId: agent.id,
// Issue-comment metadata constrains link names to 160 characters.
name: agent.name?.slice(0, 160) ?? null,
};
}
export function systemNoticePresentation(input: {
tone: IssueCommentPresentation["tone"];
title: string;
}): IssueCommentPresentation {
return {
kind: "system_notice",
tone: input.tone,
title: input.title,
detailsDefaultOpen: false,
};
}
+72 -69
View File
@@ -61,6 +61,12 @@ import {
noticeMetadataReferencesRecoveryAction,
type SuccessfulRunHandoffNotice,
} from "./successful-run-handoff.js";
import {
buildExecutionReviewParticipantRecoveryNoticeSeed,
buildExecutionReviewParticipantUnavailableNoticeSeed,
buildStrandedRecoveryEscalationNotice,
type StrandedRecoveryNoticeSeed,
} from "./stranded-notice.js";
import {
RECOVERY_ORIGIN_KINDS,
buildIssueGraphLivenessLeafKey,
@@ -312,25 +318,6 @@ function summarizeRunFailureForIssueComment(run: LatestIssueRun) {
return null;
}
function buildExecutionReviewParticipantRecoveryComment(latestRun: LatestIssueRun) {
const failureSummary = summarizeRunFailureForIssueComment(latestRun);
return (
"Paperclip retried the pending execution-review participant once, but the review stage still has no completed decision " +
`or live reviewer run.${failureSummary ?? ""} ` +
"Moving it to `blocked` with a source-scoped recovery action so the recovery owner can repair the reviewer runtime, " +
"restore the review stage, or record an intentional manual resolution."
);
}
function buildExecutionReviewParticipantUnavailableComment(latestRun: LatestIssueRun) {
const failureSummary = summarizeRunFailureForIssueComment(latestRun);
return (
"Paperclip cannot continue the pending execution-review participant because the participant is not invokable " +
`and the review stage has no completed decision or live reviewer run.${failureSummary ?? ""} ` +
"Moving it to `blocked` with a source-scoped recovery action so the recovery owner can repair the reviewer runtime, " +
"restore the review stage, or record an intentional manual resolution."
);
}
function didAutomaticRecoveryFail(
latestRun: LatestIssueRun,
@@ -3315,6 +3302,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
previousStatus: StrandedPreviousStatus;
latestRun: LatestIssueRun;
comment?: string;
notice?: StrandedRecoveryNoticeSeed | null;
recoveryCause?: StrandedRecoveryCause;
recoveryOwnerAgentId?: string | null;
successfulRunHandoffEvidence?: SuccessfulRunHandoffRecoveryEvidence | null;
@@ -3356,17 +3344,30 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
if (!updated) return null;
if (isProviderQuotaWait) return updated;
const prefix = await getCompanyIssuePrefix(input.issue.companyId);
const recoveryOwner = recoveryAction.ownerAgentId ? await getAgent(recoveryAction.ownerAgentId) : null;
const sourceAssignee = input.issue.assigneeAgentId ? await getAgent(input.issue.assigneeAgentId) : null;
let notice: SuccessfulRunHandoffNotice | null = null;
if (input.recoveryCause === SUCCESSFUL_RUN_MISSING_STATE_REASON && input.successfulRunHandoffEvidence) {
const [sourceRun] = input.successfulRunHandoffEvidence.sourceRunId
? await db
.select({
id: heartbeatRuns.id,
status: heartbeatRuns.status,
agentId: heartbeatRuns.agentId,
})
.from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.id, input.successfulRunHandoffEvidence.sourceRunId),
eq(heartbeatRuns.companyId, input.issue.companyId),
))
.limit(1)
: [];
notice = buildSuccessfulRunHandoffExhaustedNotice({
issue: input.issue,
sourceRun: input.successfulRunHandoffEvidence.sourceRunId
? { id: input.successfulRunHandoffEvidence.sourceRunId, status: "succeeded" }
sourceRun: sourceRun ?? null,
correctiveRun: input.latestRun
? { id: input.latestRun.id, status: input.latestRun.status, agentId: input.latestRun.agentId }
: null,
correctiveRun: input.latestRun ? { id: input.latestRun.id, status: input.latestRun.status } : null,
sourceAssignee,
recoveryIssue: null,
recoveryActionId: recoveryAction.id,
@@ -3376,19 +3377,24 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
missingDisposition: input.successfulRunHandoffEvidence.missingDisposition,
});
}
const recoveryLine = recoveryAction.ownerAgentId
? [
"",
`- Recovery action: \`${recoveryAction.id}\``,
`- Recovery owner: ${agentUiLink(recoveryOwner, prefix)}`,
"- Next action: the recovery owner should either restore a live execution path or record the manual resolution on the source issue.",
].join("\n")
: [
"",
`- Recovery action: \`${recoveryAction.id}\``,
"- Recovery owner: board escalation, because Paperclip could not find an invokable manager, creator, or executive owner with budget available.",
"- Next action: a board operator should assign an invokable recovery owner, fix the agent/runtime state, or record an intentional manual resolution.",
].join("\n");
const escalationNotice = buildStrandedRecoveryEscalationNotice({
seed: input.notice,
fallbackBody: input.comment,
recoveryCause,
recoveryActionId: recoveryAction.id,
recoveryOwner: recoveryAction.ownerAgentId && recoveryOwner
? { id: recoveryOwner.id, name: recoveryOwner.name }
: null,
sourceRun: input.latestRun
? {
id: input.latestRun.id,
agentId: input.latestRun.agentId,
status: input.latestRun.status,
errorCode: input.latestRun.errorCode,
errorSummary: input.latestRun.error ? redactSensitiveText(input.latestRun.error) : null,
}
: null,
});
const shouldPostEscalationComment =
recoveryAction.attemptCount === 1 ||
@@ -3421,19 +3427,10 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
metadata: notice.metadata,
});
} else {
await issuesSvc.addComment(input.issue.id, `${input.comment ?? ""}${recoveryLine}`, {}, {
await issuesSvc.addComment(input.issue.id, escalationNotice.body, {}, {
authorType: "system",
presentation: compactRecoveryPresentation(
`Recovery: ${recoveryCauseTitle(recoveryCause)} — moved to blocked ` +
`(owner: ${recoveryOwner?.name ?? "board"})`,
),
metadata: recoveryNoticeMetadata({
cause: recoveryCause,
latestRun: input.latestRun,
recoveryActionId: recoveryAction.id,
previousStatus: input.previousStatus,
recoveryOwner,
}),
presentation: escalationNotice.presentation,
metadata: escalationNotice.metadata,
});
}
}
@@ -3907,7 +3904,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
issue,
previousStatus: "in_review",
latestRun: participantLatestRun,
comment: buildExecutionReviewParticipantUnavailableComment(participantLatestRun),
notice: buildExecutionReviewParticipantUnavailableNoticeSeed(),
recoveryCause: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON,
recoveryOwnerAgentId: participantAgentId,
});
@@ -3974,7 +3971,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
issue,
previousStatus: "in_review",
latestRun: participantLatestRun,
comment: buildExecutionReviewParticipantUnavailableComment(participantLatestRun),
notice: buildExecutionReviewParticipantUnavailableNoticeSeed(),
recoveryCause: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON,
recoveryOwnerAgentId: participantAgentId,
});
@@ -3992,7 +3989,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
issue,
previousStatus: "in_review",
latestRun: participantLatestRun,
comment: buildExecutionReviewParticipantRecoveryComment(participantLatestRun),
notice: buildExecutionReviewParticipantRecoveryNoticeSeed(),
recoveryCause: EXECUTION_REVIEW_PARTICIPANT_RECOVERY_REASON,
recoveryOwnerAgentId: participantAgentId,
});
@@ -4069,15 +4066,18 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
}
if (didAutomaticRecoveryFail(latestRun, "assignment_recovery")) {
const failureSummary = summarizeRunFailureForIssueComment(latestRun);
const updated = await escalateStrandedAssignedIssue({
issue,
previousStatus: "todo",
latestRun,
comment:
"Paperclip automatically retried dispatch for this assigned `todo` issue after a lost wake/run, " +
`but it still has no live execution path.${failureSummary ?? ""} ` +
"Moving it to `blocked` so it is visible for intervention.",
notice: {
body:
"Paperclip automatically retried dispatch for this assigned `todo` issue after a lost wake/run, " +
"but it still has no live execution path. " +
"Moving it to `blocked` so it is visible for intervention.",
title: "No live execution path",
tone: "danger",
},
});
if (updated) {
result.escalated += 1;
@@ -4210,15 +4210,18 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
}
if (classification.kind === "non_retryable") {
const failureSummary = summarizeRunFailureForIssueComment(latestRun);
const updated = await escalateStrandedAssignedIssue({
issue,
previousStatus: "in_progress",
latestRun,
comment:
"Paperclip detected a non-retryable failure on this issue's continuation run " +
`(\`${classification.errorCode}\`). Skipping automatic retries and moving it to \`blocked\` ` +
`so it is visible for intervention.${failureSummary ?? ""}`,
notice: {
body:
"Paperclip detected a non-retryable failure on this issue's continuation run " +
`(\`${classification.errorCode}\`). Skipping automatic retries and moving it to \`blocked\` ` +
"so it is visible for intervention.",
title: "Continuation failed",
tone: "danger",
},
});
if (updated) {
result.escalated += 1;
@@ -4237,19 +4240,19 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
classification.errorCode,
);
if (consecutive >= classification.maxAttempts) {
const failureSummary = summarizeRunFailureForIssueComment(latestRun);
const attemptCopy = consecutive <= 1 ? "" : ` (${consecutive}× attempts)`;
const causeCopy = classification.errorCode
? ` Latest cause: \`${classification.errorCode}\`.`
: "";
const updated = await escalateStrandedAssignedIssue({
issue,
previousStatus: "in_progress",
latestRun,
comment:
"Paperclip automatically retried continuation for this assigned `in_progress` issue after its live " +
`execution disappeared, but it still has no live execution path${attemptCopy}.${causeCopy}${failureSummary ?? ""} ` +
"Moving it to `blocked` so it is visible for intervention.",
notice: {
body:
"Paperclip automatically retried continuation for this assigned `in_progress` issue after its live " +
`execution disappeared, but it still has no live execution path${attemptCopy}. ` +
"Moving it to `blocked` so it is visible for intervention.",
title: "No live execution path",
tone: "danger",
},
});
if (updated) {
result.escalated += 1;
@@ -0,0 +1,176 @@
import { describe, expect, it } from "vitest";
import { noticeMetadataReferencesRecoveryAction } from "./successful-run-handoff.js";
import {
DEFAULT_STRANDED_RECOVERY_NOTICE_BODY,
buildConfigurationIncompleteRecoveryNoticeSeed,
buildExecutionReviewParticipantRecoveryNoticeSeed,
buildExecutionReviewParticipantUnavailableNoticeSeed,
buildImmediateExecutionPathRecoveryNoticeSeed,
buildStrandedRecoveryEscalationNotice,
buildWorkspaceValidationRecoveryNoticeSeed,
} from "./stranded-notice.js";
function allRows(metadata: { sections: Array<{ rows: unknown[] }> }) {
return metadata.sections.flatMap((section) => section.rows) as Array<Record<string, unknown>>;
}
describe("stranded recovery notice seeds", () => {
it.each([
["todo dispatch", buildImmediateExecutionPathRecoveryNoticeSeed({ status: "todo" }), "No live execution path"],
["in_progress continuation", buildImmediateExecutionPathRecoveryNoticeSeed({ status: "in_progress" }), "No live execution path"],
["workspace validation", buildWorkspaceValidationRecoveryNoticeSeed(), "Workspace validation failed"],
["configuration incomplete", buildConfigurationIncompleteRecoveryNoticeSeed(), "Configuration incomplete"],
["review participant recovery", buildExecutionReviewParticipantRecoveryNoticeSeed(), "Review recovery stalled"],
["review participant unavailable", buildExecutionReviewParticipantUnavailableNoticeSeed(), "Review recovery stalled"],
])("%s seed has a short body plus title and tone", (_label, seed, expectedTitle) => {
expect(seed.title).toBe(expectedTitle);
expect(seed.tone).toBe("danger");
expect(seed.body.length).toBeGreaterThan(0);
// Failure details now live in metadata rows, never inline in the body.
expect(seed.body).not.toContain("Latest retry failure");
expect(seed.body).not.toContain("Recovery action:");
});
it("distinguishes todo dispatch from in_progress continuation copy", () => {
expect(buildImmediateExecutionPathRecoveryNoticeSeed({ status: "todo" }).body).toContain("retried dispatch");
expect(buildImmediateExecutionPathRecoveryNoticeSeed({ status: "in_progress" }).body).toContain(
"retried continuation",
);
});
});
describe("buildStrandedRecoveryEscalationNotice", () => {
const actionId = "6a2f8e64-6f5e-4b58-b7fd-111111111111";
const owner = { id: "9b1c2d3e-4f50-4a61-8b72-222222222222", name: "CTO" };
const sourceRun = {
id: "0c1d2e3f-4a5b-4c6d-8e7f-333333333333",
agentId: "4d5e6f70-8a9b-4c1d-8e2f-444444444444",
status: "failed",
errorCode: "workspace_validation_failed",
errorSummary: "Expected project worktree but resolved the agent fallback directory.",
};
it("emits system_notice presentation and the required metadata rows", () => {
const seed = buildWorkspaceValidationRecoveryNoticeSeed();
const notice = buildStrandedRecoveryEscalationNotice({
seed,
recoveryActionId: actionId,
recoveryOwner: owner,
sourceRun,
});
expect(notice.body).toBe(seed.body);
expect(notice.presentation).toMatchObject({
kind: "system_notice",
tone: "danger",
title: "Workspace validation failed",
});
expect(notice.metadata.version).toBe(1);
expect(notice.metadata.sourceRunId).toBe(sourceRun.id);
const rows = allRows(notice.metadata);
expect(rows).toContainEqual({ type: "key_value", label: "Recovery action", value: actionId });
expect(rows).toContainEqual({ type: "agent_link", label: "Recovery owner", agentId: owner.id, name: owner.name });
expect(rows).toContainEqual({
type: "run_link",
label: "Source run",
runId: sourceRun.id,
agentId: sourceRun.agentId,
title: "failed",
});
expect(rows).toContainEqual({
type: "key_value",
label: "Failure code",
value: "workspace_validation_failed",
});
expect(rows).toContainEqual({
type: "key_value",
label: "Failure summary",
value: sourceRun.errorSummary,
});
expect(rows.some((row) => row.label === "Next action")).toBe(true);
});
it("records board escalation when there is no invokable recovery owner", () => {
const notice = buildStrandedRecoveryEscalationNotice({
seed: buildConfigurationIncompleteRecoveryNoticeSeed(),
recoveryActionId: actionId,
recoveryOwner: null,
sourceRun,
});
const ownerRow = allRows(notice.metadata).find((row) => row.label === "Recovery owner");
expect(ownerRow?.type).toBe("key_value");
expect(String(ownerRow?.value)).toContain("Board escalation");
});
it("derives title from the recovery cause and body from the plain-comment fallback", () => {
const notice = buildStrandedRecoveryEscalationNotice({
fallbackBody: "Legacy plain comment body.",
recoveryCause: "configuration_incomplete",
recoveryActionId: actionId,
recoveryOwner: owner,
sourceRun: null,
});
expect(notice.body).toBe("Legacy plain comment body.");
expect(notice.presentation.title).toBe("Configuration incomplete");
expect(notice.metadata.sourceRunId).toBeNull();
expect(notice.metadata.sections).toHaveLength(1);
});
it("falls back to the default body and title when no seed or comment is given", () => {
const notice = buildStrandedRecoveryEscalationNotice({
recoveryActionId: actionId,
recoveryOwner: null,
sourceRun: null,
});
expect(notice.body).toBe(DEFAULT_STRANDED_RECOVERY_NOTICE_BODY);
expect(notice.presentation.title).toBe("Automatic recovery blocked");
});
it("omits the failure code row when the source run has no error code", () => {
const notice = buildStrandedRecoveryEscalationNotice({
seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "in_progress" }),
recoveryActionId: actionId,
recoveryOwner: owner,
sourceRun: {
id: sourceRun.id,
agentId: sourceRun.agentId,
status: "cancelled",
errorCode: null,
errorSummary: null,
},
});
const rows = allRows(notice.metadata);
expect(rows).toContainEqual({
type: "run_link",
label: "Source run",
runId: sourceRun.id,
agentId: sourceRun.agentId,
title: "cancelled",
});
expect(rows.some((row) => row.label === "Failure code")).toBe(false);
expect(rows.some((row) => row.label === "Failure summary")).toBe(false);
});
it("is matched by the metadata-based escalation dedupe matcher", () => {
const notice = buildStrandedRecoveryEscalationNotice({
seed: buildImmediateExecutionPathRecoveryNoticeSeed({ status: "todo" }),
recoveryActionId: actionId,
recoveryOwner: owner,
sourceRun,
});
// The escalation dedupe in recovery/service.ts no longer relies on the
// `Recovery action: \`id\`` body marker for new comments; the metadata
// matcher must recognize every notice this builder produces.
expect(notice.body).not.toContain(actionId);
expect(noticeMetadataReferencesRecoveryAction(notice.metadata, actionId)).toBe(true);
expect(
noticeMetadataReferencesRecoveryAction(notice.metadata, "1f2e3d4c-5b6a-4798-8899-444444444444"),
).toBe(false);
});
});
@@ -0,0 +1,158 @@
import type { IssueCommentMetadata, IssueCommentPresentation } from "@paperclipai/shared";
import {
agentLinkRow,
keyValueRow,
runLinkRow,
systemNoticePresentation,
type NoticeMetadataRow,
type NoticeMetadataSection,
} from "./notice-format.js";
// Short human-readable body plus the presentation header for one recovery
// family. The escalation path merges in the metadata rows only it knows
// (recovery action, owner, source run) via buildStrandedRecoveryEscalationNotice.
export type StrandedRecoveryNoticeSeed = {
body: string;
title: string;
tone: IssueCommentPresentation["tone"];
};
export type StrandedRecoveryEscalationNotice = {
body: string;
presentation: IssueCommentPresentation;
metadata: IssueCommentMetadata;
};
export const DEFAULT_STRANDED_RECOVERY_NOTICE_BODY =
"Paperclip could not restore a live execution path for this issue automatically. " +
"Moving it to `blocked` so it is visible for intervention.";
const DEFAULT_STRANDED_RECOVERY_NOTICE_TITLE = "Automatic recovery blocked";
const STRANDED_RECOVERY_NOTICE_TITLES_BY_CAUSE: Record<string, string> = {
workspace_validation_failed: "Workspace validation failed",
configuration_incomplete: "Configuration incomplete",
execution_review_participant_recovery: "Review recovery stalled",
};
export function buildImmediateExecutionPathRecoveryNoticeSeed(input: {
status: "todo" | "in_progress";
}): StrandedRecoveryNoticeSeed {
const retryDescription = input.status === "todo"
? "Paperclip automatically retried dispatch for this assigned `todo` issue during terminal run recovery"
: "Paperclip automatically retried continuation for this assigned `in_progress` issue during terminal run recovery";
return {
body:
`${retryDescription}, but it still has no live execution path. ` +
"Moving it to `blocked` so it is visible for intervention.",
title: "No live execution path",
tone: "danger",
};
}
export function buildWorkspaceValidationRecoveryNoticeSeed(): StrandedRecoveryNoticeSeed {
return {
body:
"Paperclip stopped before launching the local adapter because the issue workspace failed validation. " +
"Moving it to `blocked` so the workspace link, cwd, or git checkout can be repaired before resuming.",
title: "Workspace validation failed",
tone: "danger",
};
}
export function buildConfigurationIncompleteRecoveryNoticeSeed(): StrandedRecoveryNoticeSeed {
return {
body:
"Paperclip stopped before dispatching the adapter because required secret/env bindings are missing. " +
"Moving it to `blocked` so an operator can bind the missing secret(s) before resuming.",
title: "Configuration incomplete",
tone: "danger",
};
}
export function buildExecutionReviewParticipantRecoveryNoticeSeed(): StrandedRecoveryNoticeSeed {
return {
body:
"Paperclip retried the pending execution-review participant once, but the review stage still has no " +
"completed decision or live reviewer run. Moving it to `blocked` so the recovery owner can repair the " +
"reviewer runtime, restore the review stage, or record an intentional manual resolution.",
title: "Review recovery stalled",
tone: "danger",
};
}
export function buildExecutionReviewParticipantUnavailableNoticeSeed(): StrandedRecoveryNoticeSeed {
return {
body:
"Paperclip cannot continue the pending execution-review participant because the participant is not " +
"invokable and the review stage has no completed decision or live reviewer run. Moving it to `blocked` " +
"so the recovery owner can repair the reviewer runtime, restore the review stage, or record an " +
"intentional manual resolution.",
title: "Review recovery stalled",
tone: "danger",
};
}
// Escalation dedupe matches the `Recovery action` key_value row via
// noticeMetadataReferencesRecoveryAction, so this builder must always emit
// that row with the raw action id.
export function buildStrandedRecoveryEscalationNotice(input: {
seed?: StrandedRecoveryNoticeSeed | null;
fallbackBody?: string | null;
recoveryCause?: string | null;
recoveryActionId: string;
recoveryOwner: { id: string; name: string | null } | null | undefined;
sourceRun: {
id: string;
agentId?: string | null;
status: string;
errorCode?: string | null;
errorSummary?: string | null;
} | null | undefined;
}): StrandedRecoveryEscalationNotice {
const fallbackBody = input.fallbackBody?.trim();
const body = input.seed?.body ?? (fallbackBody || DEFAULT_STRANDED_RECOVERY_NOTICE_BODY);
const title = input.seed?.title ??
STRANDED_RECOVERY_NOTICE_TITLES_BY_CAUSE[input.recoveryCause ?? ""] ??
DEFAULT_STRANDED_RECOVERY_NOTICE_TITLE;
const recoveryRows: NoticeMetadataRow[] = [
keyValueRow("Recovery action", input.recoveryActionId),
input.recoveryOwner
? agentLinkRow("Recovery owner", input.recoveryOwner)
: keyValueRow(
"Recovery owner",
"Board escalation - Paperclip could not find an invokable manager, creator, or executive owner with budget available",
),
keyValueRow(
"Next action",
input.recoveryOwner
? "The recovery owner should either restore a live execution path or record the manual resolution on the source issue"
: "A board operator should assign an invokable recovery owner, fix the agent/runtime state, or record an intentional manual resolution",
),
];
const runRows: NoticeMetadataRow[] = [];
if (input.sourceRun) {
runRows.push(runLinkRow("Source run", input.sourceRun));
const failureCode = input.sourceRun.errorCode?.trim();
if (failureCode) runRows.push(keyValueRow("Failure code", failureCode));
const failureSummary = input.sourceRun.errorSummary?.trim();
if (failureSummary) runRows.push(keyValueRow("Failure summary", failureSummary));
}
const sections: NoticeMetadataSection[] = [
{ title: "Recovery", rows: recoveryRows },
...(runRows.length > 0 ? [{ title: "Run evidence", rows: runRows }] : []),
];
return {
body,
presentation: systemNoticePresentation({ tone: input.seed?.tone ?? "danger", title }),
metadata: {
version: 1,
sourceRunId: input.sourceRun?.id ?? null,
sections,
},
};
}
@@ -389,6 +389,7 @@ describe("successful run handoff decision", () => {
run: {
id: "22222222-2222-4222-8222-222222222222",
status: "succeeded",
agentId: "33333333-3333-4333-8333-333333333333",
} as any,
agent: {
id: "33333333-3333-4333-8333-333333333333",
@@ -417,7 +418,11 @@ describe("successful run handoff decision", () => {
expect.objectContaining({
title: "Run evidence",
rows: expect.arrayContaining([
expect.objectContaining({ type: "run_link", runId: "22222222-2222-4222-8222-222222222222" }),
expect.objectContaining({
type: "run_link",
runId: "22222222-2222-4222-8222-222222222222",
agentId: "33333333-3333-4333-8333-333333333333",
}),
expect.objectContaining({ type: "key_value", label: "Normalized cause", value: SUCCESSFUL_RUN_MISSING_STATE_REASON }),
expect.objectContaining({ type: "key_value", label: "Detected progress" }),
]),
@@ -433,8 +438,16 @@ describe("successful run handoff decision", () => {
title: "Finish backend handoff",
status: "in_progress",
} as any,
sourceRun: { id: "22222222-2222-4222-8222-222222222222", status: "succeeded" } as any,
correctiveRun: { id: "44444444-4444-4444-8444-444444444444", status: "failed" } as any,
sourceRun: {
id: "22222222-2222-4222-8222-222222222222",
status: "succeeded",
agentId: "33333333-3333-4333-8333-333333333333",
} as any,
correctiveRun: {
id: "44444444-4444-4444-8444-444444444444",
status: "failed",
agentId: "66666666-6666-4666-8666-666666666666",
} as any,
sourceAssignee: { id: "33333333-3333-4333-8333-333333333333", name: "CodexCoder" } as any,
recoveryIssue: {
id: "55555555-5555-4555-8555-555555555555",
@@ -467,7 +480,16 @@ describe("successful run handoff decision", () => {
expect.objectContaining({
title: "Run evidence",
rows: expect.arrayContaining([
expect.objectContaining({ type: "run_link", label: "Source run" }),
expect.objectContaining({
type: "run_link",
label: "Source run",
agentId: "33333333-3333-4333-8333-333333333333",
}),
expect.objectContaining({
type: "run_link",
label: "Corrective handoff run",
agentId: "66666666-6666-4666-8666-666666666666",
}),
expect.objectContaining({ type: "run_link", label: "Corrective handoff run" }),
expect.objectContaining({ type: "key_value", label: "Missing disposition", value: "clear_next_step" }),
]),
@@ -3,6 +3,13 @@ import type { Db } from "@paperclipai/db";
import { agentWakeupRequests, agents, heartbeatRuns, issues } from "@paperclipai/db";
import type { IssueCommentMetadata, IssueCommentPresentation, RunLivenessState } from "@paperclipai/shared";
import { withRecoveryModelProfileHint } from "./model-profile-hint.js";
import {
agentLinkRow,
issueLinkRow,
keyValueRow,
runLinkRow,
systemNoticePresentation,
} from "./notice-format.js";
export const FINISH_SUCCESSFUL_RUN_HANDOFF_REASON = "finish_successful_run_handoff";
export const SUCCESSFUL_RUN_MISSING_STATE_REASON = "successful_run_missing_state";
@@ -57,7 +64,7 @@ type IssueRow = Pick<
>;
type AgentRow = Pick<typeof agents.$inferSelect, "id" | "companyId" | "status">;
type NoticeIssue = Pick<typeof issues.$inferSelect, "id" | "identifier" | "title" | "status">;
type NoticeRun = Pick<typeof heartbeatRuns.$inferSelect, "id" | "status">;
type NoticeRun = Pick<typeof heartbeatRuns.$inferSelect, "id" | "status" | "agentId">;
type NoticeAgent = Pick<typeof agents.$inferSelect, "id" | "name">;
type NullableNoticeAgent = NoticeAgent | null | undefined;
type NullableNoticeIssue = NoticeIssue | null | undefined;
@@ -115,58 +122,6 @@ export function isSuccessfulRunHandoffValidPathSkip(
return decision.kind === "skip" && SUCCESSFUL_RUN_HANDOFF_VALID_PATH_SKIP_REASONS.has(decision.reason);
}
function metadataText(value: unknown, fallback = "unknown") {
const text = typeof value === "string" ? value.trim() : value == null ? "" : String(value).trim();
const resolved = text.length > 0 ? text : fallback;
return resolved.length > 2000 ? `${resolved.slice(0, 1997)}...` : resolved;
}
function keyValueRow(label: string, value: unknown): IssueCommentMetadata["sections"][number]["rows"][number] {
return { type: "key_value", label, value: metadataText(value) };
}
function issueLinkRow(
label: string,
issue: NullableNoticeIssue,
): IssueCommentMetadata["sections"][number]["rows"][number] {
if (!issue) return keyValueRow(label, "unknown");
return {
type: "issue_link",
label,
issueId: issue.id,
identifier: issue.identifier,
title: issue.title,
};
}
function runLinkRow(
label: string,
run: NullableNoticeRun,
): IssueCommentMetadata["sections"][number]["rows"][number] {
if (!run) return keyValueRow(label, "unknown");
return { type: "run_link", label, runId: run.id, title: run.status };
}
function agentLinkRow(
label: string,
agent: NullableNoticeAgent,
): IssueCommentMetadata["sections"][number]["rows"][number] {
if (!agent) return keyValueRow(label, "unknown");
return { type: "agent_link", label, agentId: agent.id, name: agent.name };
}
function systemNoticePresentation(input: {
tone: IssueCommentPresentation["tone"];
title: string;
}): IssueCommentPresentation {
return {
kind: "system_notice",
tone: input.tone,
title: input.title,
detailsDefaultOpen: false,
};
}
export function isSuccessfulRunHandoffRequiredNoticeBody(body: string) {
const trimmed = body.trim();
return trimmed === SUCCESSFUL_RUN_HANDOFF_REQUIRED_NOTICE_BODY ||