perf(recovery): stop re-asserting a failed legacy run whose decision is recorded (#21621)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Stranded-work recovery checks tasks that are assigned to an agent
but have no live run. It runs as a scoped-deadline queue that a change
notification or a stored deadline wakes.
> - For a failed legacy run, the check calls
`terminalizeLegacyExecution` with the run's current status. When the
run's recovery decision is already recorded (for example "continue in a
fresh run from saved work"), that call changes nothing. But it still
updated the run row and notified about 18 delivery queues, including the
run's own recovery graph.
> - The graph notification re-checked the same task, which re-asserted
the same run again. The loop ran about once a second per waiting task,
forever, with zero runs active. On a hosted instance this shows up as
constant idle transactions.
> - This pull request makes a re-assertion that changes nothing return
before it writes or notifies.
> - The benefit is that an idle instance does no stranded-recovery work
for tasks that wait on a recorded decision. A real change (another run
fails, a comment, a status change) still notifies and recovers at once.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

With zero runs active, the server runs many transactions per second. A
CPU profile groups most samples under `reconcileStrandedAssignedIssues`
(`server/src/services/recovery/service.ts`). A local fixture on real
PostgreSQL reproduces it: a task in `in_progress`, assigned to an agent,
whose latest legacy run failed and whose recovery decision is recorded.

Each check reaches the branch that terminalizes a failed legacy run that
needs reconciliation (`service.ts`, the
`legacyExecutionNeedsReconciliationWithEvidence` branch, and the same
call in `enqueueStrandedIssueRecovery`). `terminalizeLegacyExecution`
then:
- updates the run with the same status (only `updated_at` changes);
- calls `notifyExecutionRunWork`, `notifyInstructionCollection` and
`notifyUnsafeWorkspaceExportRecovery`, which hint the run graph of the
task and about 18 queues;
- finds the recorded decision and returns.

The graph hint re-checks the same task after the queue's 1-second pass
interval, so the loop never ends.

**Expected behavior**

A check that finds nothing to change stores nothing and wakes nothing.
Only a real change re-checks the task.

**Steps to reproduce**

1. Create a task in `in_progress` assigned to an agent, with a failed
legacy run (`adapter_failed`).
2. Start the issue-state recovery work. The boot pass records a
fresh-continuation decision for the run.
3. Leave the instance idle. On master: 591,045 statements in 30 minutes
for 10 such tasks (fake clock, real PostgreSQL), and one no-op `update
"heartbeat_runs"` per task per second.

**Paperclip version or commit**

master at 7db92ea319.

## What Changed

- `server/src/services/legacy-execution-recovery.ts`:
- `terminalizeLegacyExecution` returns the unchanged run, without a
write or a notification, when all of these are true: the call re-asserts
the current status with no patch; the run has no workspace restore
failure or required workspace recovery; the task's execution and
checkout locks do not point at the run; and either the run needs no
reconciliation (`reconcileIfNeeded`), no assignee or reviewer owns a
decision, or the decision is already recorded.
- Why the short-circuit is safe: stranded-work and quota checks
re-assert the run's unchanged terminal status each time they revisit its
task. Once the decision is recorded, that re-assertion changes nothing.
If it wrote or notified anyway, the notifications would re-check the
same task, which would re-assert the run again, about once a second for
as long as the task waits.
- `findRecordedLegacyDecision` returns the resolved decision already
recorded for the run (a reconciliation for this run, an explicit user
continuation with a real successor, or a no-replay hold).
`decisionOwner` says whether the task's assignee or current reviewer
owns a decision about the run.
- The recorded-decision query moves into `findRecordedLegacyDecision`,
and the owner condition into `decisionOwner`. The full path uses both
helpers, so its behavior does not change.
- `server/src/__tests__/issue-recovery-recorded-decision-idle.test.ts`:
two new tests on real PostgreSQL. They use the real coordinator, the
real issue-state recovery work and the production 5-minute backstop on a
fake clock.

## Verification

- `stays quiet for half an hour once each failed run has its decision`:
10 waiting tasks, zero runs. The test first checks that each failed run
has a recorded decision, then counts statements over 30 idle minutes.
  - master: 591,045 statements (red).
- this branch: 0 statements (green). No run is re-written or started,
and no task moves.
- `still recovers promptly when another run of a waiting task fails`:
after 10 idle minutes, a new legacy run of one task fails and notifies
like a finished run.
- this branch: within 5 seconds, recovery creates the run's
`legacy-execution:<run>` hold. Within the next 6 minutes, the
fresh-continuation fallback resolves that hold with
`executionReconciliation.runId` set to the new run. Over the next 30
minutes there is no `update "heartbeat_runs"` and fewer than 5,000
statements (the assigned-work invariant keeps its own spaced re-checks).
- master: 18,000 no-op `update "heartbeat_runs"` statements in the same
window (red).
- Neighbouring suites pass locally: `legacy-workspace-restore-recovery`,
`legacy-workspace-restore-lock-contention`,
`legacy-continuation-authority`, `heartbeat-process-recovery`,
`issue-recovery-assignments`, `recovery-deadline-ordinary-bootstrap`,
`issue-recovery-deadlines` (600 of 601 in one parallel run; `resolves a
run's task graph through index lookups` failed once under machine load
on a planner row count, and passes alone; it does not use the changed
code).
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low risk. The short-circuit applies only when the full path would also
write nothing except `updated_at`. Every other case (status change,
patch, workspace restore evidence, an owned lock, no recorded decision)
takes the unchanged full path.
- A re-assertion no longer re-sends notifications. Queues that relied on
that repeated hint as a side effect lose it. The first terminalization
of a run still sends all notifications, and run, comment and status
writes still send their own.
- No schema or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21552, #21616, #21617.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 10:30:03 -06:00
1 parent 68d5c9c120
commit 45531b719e
2 files changed
+154 -47

No files matched your search

@@ -0,0 +1,88 @@
import { setImmediate as immediate } from "node:timers/promises";
import { and, eq, sql } from "drizzle-orm";
import { expect, it, vi } from "vitest";
import { agents, heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db";
import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js";
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
describeIssueRecovery("stranded recovery of runs with a recorded decision", () => {
const state = issueRecoveryFixture();
const TASKS = 10;
async function seed() {
const f = await state.fixture();
const db = state.db;
await db.update(agents).set({ status: "error" }).where(eq(agents.id, f.agentId));
const tasks = await db.insert(issues).values(Array.from({ length: TASKS }, (_, index) => ({
companyId: f.companyId, title: `Waiting task ${index}`, status: "in_progress", assigneeAgentId: f.agentId,
}))).returning({ id: issues.id });
await db.insert(heartbeatRuns).values(tasks.map(task => ({ companyId: f.companyId, agentId: f.agentId,
status: "failed", errorCode: "adapter_failed", finishedAt: new Date(Date.now() - 60 * 60_000),
contextSnapshot: { issueId: task.id } })));
return { ...f, tasks };
}
async function advance(ms: number) {
const until = Date.now() + ms;
while (Date.now() < until) {
await vi.advanceTimersByTimeAsync(Math.min(500, until - Date.now()));
await Promise.allSettled([...state.passes]); await immediate();
}
}
async function countStatements(work: () => Promise<unknown>) {
const statements: string[] = [], prior = state.db.$client.options.debug;
state.db.$client.options.debug = (_connection, query) => { statements.push(query); };
try { await work(); } finally { state.db.$client.options.debug = prior; }
return statements;
}
async function startAndRecordDecisions(f: Awaited<ReturnType<typeof seed>>) {
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
await state.start({});
await advance(6 * 60_000);
const decided = await state.db.select({ issueId: issueRecoveryActions.sourceIssueId }).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, f.companyId), eq(issueRecoveryActions.status, "resolved"),
sql`${issueRecoveryActions.evidence} ? 'executionReconciliation'`));
expect(new Set(decided.map(row => row.issueId))).toEqual(new Set(f.tasks.map(task => task.id)));
}
it("stays quiet for half an hour once each failed run has its decision", async () => {
const f = await seed();
await startAndRecordDecisions(f);
const runsBefore = await state.db.select({ id: heartbeatRuns.id, status: heartbeatRuns.status, updatedAt: heartbeatRuns.updatedAt })
.from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
const idle = await countStatements(() => advance(30 * 60_000));
expect(idle).toHaveLength(0);
const runsAfter = await state.db.select({ id: heartbeatRuns.id, status: heartbeatRuns.status, updatedAt: heartbeatRuns.updatedAt })
.from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
const byId = (a: { id: string }, b: { id: string }) => a.id.localeCompare(b.id);
expect(runsAfter.sort(byId)).toEqual(runsBefore.sort(byId));
const statuses = await state.db.select({ status: issues.status }).from(issues).where(eq(issues.companyId, f.companyId));
expect(statuses.every(row => row.status === "in_progress")).toBe(true);
}, 300_000);
it("still recovers promptly when another run of a waiting task fails", async () => {
const f = await seed();
await startAndRecordDecisions(f);
await advance(10 * 60_000);
const target = f.tasks[0].id;
const [failed] = await state.db.transaction(async tx => {
const rows = await tx.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "failed",
errorCode: "adapter_failed", finishedAt: new Date(), contextSnapshot: { issueId: target } }).returning();
await notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: rows[0].id });
return rows;
});
const actionFor = async () => (await state.db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, f.companyId), eq(issueRecoveryActions.sourceIssueId, target),
eq(issueRecoveryActions.fingerprint, `legacy-execution:${failed.id}`))))[0];
await advance(5_000);
expect(await actionFor()).toMatchObject({ kind: "active_run_watchdog", evidence: expect.objectContaining({ runId: failed.id }) });
await advance(6 * 60_000);
expect(await actionFor()).toMatchObject({ status: "resolved",
evidence: expect.objectContaining({ executionReconciliation: expect.objectContaining({ runId: failed.id }) }) });
const after = await countStatements(() => advance(30 * 60_000));
expect(after.filter(query => query.startsWith("update \"heartbeat_runs\""))).toEqual([]);
expect(after.length).toBeLessThan(TASKS * 500);
}, 300_000);
});
@@ -111,6 +111,58 @@ export async function legacyExecutionNeedsReconciliationWithEvidence(db: Db, run
&& !(await isPreDispatchExecutionWaitVerified(db, run)));
}
async function findRecordedLegacyDecision(db: Db, run: Run, taskId: string, resultJson: Run["resultJson"]) {
const [reconciled] = await db.select({ id: issueRecoveryActions.id })
.from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, run.companyId),
eq(issueRecoveryActions.sourceIssueId, taskId),
eq(issueRecoveryActions.status, "resolved"),
...(hasRequiredWorkspaceRecovery(resultJson) ? [
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
] : []),
or(
sql`${issueRecoveryActions.evidence}->'executionReconciliation'->>'runId' = ${run.id}`,
and(
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
sql`exists (
select 1 from heartbeat_runs successor
where successor.id = ${canonicalExecutionUuid(sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'runId'`)}
and successor.company_id = ${run.companyId} and successor.agent_id = ${run.agentId}
and coalesce(successor.native_issue_id::text, successor.issue_id::text,
successor.context_snapshot->>'issueId') = ${taskId}
and successor.context_snapshot->>'previousRunId' = ${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'previousRunId'
and successor.context_snapshot->'explicitUserContinuation'->>'previousRunId' = ${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'previousRunId'
and successor.context_snapshot->'explicitUserContinuation'->>'commentId' is not distinct from
${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'commentId'
)`,
),
and(
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
or(sql`${issueRecoveryActions.evidence}->>'workspaceRestoreFailure' = 'restore_unsafe_archive'`,
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`),
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
),
and(
eq(issueRecoveryActions.cause, LEGACY_RECOVERY_CAUSE),
eq(issueRecoveryActions.fingerprint, `legacy-execution:${run.id}`),
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
),
),
)).limit(1);
return reconciled ?? null;
}
function decisionOwner(task: typeof issues.$inferSelect, run: Run, retainedLeaseCount: number) {
const review = task.status === "in_review" ? parseIssueExecutionState(task.executionState) : null;
const isCurrentReviewer = review?.status === "pending" &&
review.currentParticipant?.type === "agent" && review.currentParticipant.agentId === run.agentId;
const owns = !isSupersededConversationRun(task, run) &&
(task.assigneeAgentId === run.agentId || isCurrentReviewer) &&
(!["done", "cancelled"].includes(task.status) || retainedLeaseCount > 0);
return { owns, isCurrentReviewer };
}
/** Persist the failed legacy run, owned lock release and operator decision together. */
export async function terminalizeLegacyExecution(input: {
db: Db;
@@ -152,6 +204,16 @@ export async function terminalizeLegacyExecution(input: {
)).for("update");
if (!current || (current.runtimeMode !== "legacy" && !(input.recordRestoreFailureOnly && current.runtimeMode === "native")) ||
(!input.recordRestoreFailureOnly && !(input.fromStatuses ?? [run.status]).includes(current.status))) return null;
if (!input.recordRestoreFailureOnly && current.status === status && Object.keys(patch ?? {}).length === 0
&& !hasWorkspaceRestoreFailure(current.resultJson) && !hasRequiredWorkspaceRecovery(current.resultJson)
&& task?.executionRunId !== run.id && task?.checkoutRunId !== run.id) {
const [unchanged] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId)));
if (input.reconcileIfNeeded && !(await legacyExecutionNeedsReconciliationWithEvidence(tx as unknown as Db, unchanged)))
return unchanged;
if (!task || !decisionOwner(task, unchanged, 0).owns) return unchanged;
if (await findRecordedLegacyDecision(tx as unknown as Db, run, task.id, unchanged.resultJson)) return unchanged;
}
if (patch?.resultJson !== undefined) patch = { ...patch,
resultJson: preserveWorkspaceRestoreRecoveryMetadata(current.resultJson,
input.recordRestoreFailureOnly ? { ...current.resultJson, ...patch.resultJson } : patch.resultJson),
@@ -255,58 +317,15 @@ export async function terminalizeLegacyExecution(input: {
fingerprint, attemptCount: 1 });
return updated;
}
const review = task?.status === "in_review" ? parseIssueExecutionState(task.executionState) : null;
const isCurrentReviewer = review?.status === "pending" &&
review.currentParticipant?.type === "agent" && review.currentParticipant.agentId === run.agentId;
if (
task &&
!isSupersededConversationRun(task, updated) &&
(task.assigneeAgentId === run.agentId || isCurrentReviewer) &&
(!["done", "cancelled"].includes(task.status) || retainedLeaseIds.length > 0)
) {
const owner = task ? decisionOwner(task, updated, retainedLeaseIds.length) : null;
if (task && owner?.owns) {
const { isCurrentReviewer } = owner;
// Periodic stranded-work checks may revisit this terminal run before its
// reconciled or explicitly requested continuation is dispatched. Preserve
// its recorded decision instead of creating another hold. The explicit
// receipt must identify an actual successor under the same task/agent;
// retained user intent remains authoritative after that successor finishes.
const [reconciled] = await tx.select({ id: issueRecoveryActions.id })
.from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, run.companyId),
eq(issueRecoveryActions.sourceIssueId, task.id),
eq(issueRecoveryActions.status, "resolved"),
...(hasRequiredWorkspaceRecovery(updated.resultJson) ? [
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
] : []),
or(
sql`${issueRecoveryActions.evidence}->'executionReconciliation'->>'runId' = ${run.id}`,
and(
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
sql`exists (
select 1 from heartbeat_runs successor
where successor.id = ${canonicalExecutionUuid(sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'runId'`)}
and successor.company_id = ${run.companyId} and successor.agent_id = ${run.agentId}
and coalesce(successor.native_issue_id::text, successor.issue_id::text,
successor.context_snapshot->>'issueId') = ${task.id}
and successor.context_snapshot->>'previousRunId' = ${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'previousRunId'
and successor.context_snapshot->'explicitUserContinuation'->>'previousRunId' = ${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'previousRunId'
and successor.context_snapshot->'explicitUserContinuation'->>'commentId' is not distinct from
${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'commentId'
)`,
),
and(
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
or(sql`${issueRecoveryActions.evidence}->>'workspaceRestoreFailure' = 'restore_unsafe_archive'`,
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`),
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
),
and(
eq(issueRecoveryActions.cause, LEGACY_RECOVERY_CAUSE),
eq(issueRecoveryActions.fingerprint, `legacy-execution:${run.id}`),
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
),
),
)).limit(1);
const reconciled = await findRecordedLegacyDecision(tx as unknown as Db, run, task.id, updated.resultJson);
if (reconciled) return updated;
await issueRecoveryActionService(tx as unknown as Db).upsertSourceScoped({
companyId: run.companyId,