From 45531b719eab6b2af25c18b35a882b4cc1aeebd6 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 10:30:03 -0600 Subject: [PATCH] perf(recovery): stop re-asserting a failed legacy run whose decision is recorded (#21621) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Stranded-work recovery checks tasks that are assigned to an agent but have no live run. It runs as a scoped-deadline queue that a change notification or a stored deadline wakes. > - For a failed legacy run, the check calls `terminalizeLegacyExecution` with the run's current status. When the run's recovery decision is already recorded (for example "continue in a fresh run from saved work"), that call changes nothing. But it still updated the run row and notified about 18 delivery queues, including the run's own recovery graph. > - The graph notification re-checked the same task, which re-asserted the same run again. The loop ran about once a second per waiting task, forever, with zero runs active. On a hosted instance this shows up as constant idle transactions. > - This pull request makes a re-assertion that changes nothing return before it writes or notifies. > - The benefit is that an idle instance does no stranded-recovery work for tasks that wait on a recorded decision. A real change (another run fails, a comment, a status change) still notifies and recovers at once. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** With zero runs active, the server runs many transactions per second. A CPU profile groups most samples under `reconcileStrandedAssignedIssues` (`server/src/services/recovery/service.ts`). A local fixture on real PostgreSQL reproduces it: a task in `in_progress`, assigned to an agent, whose latest legacy run failed and whose recovery decision is recorded. Each check reaches the branch that terminalizes a failed legacy run that needs reconciliation (`service.ts`, the `legacyExecutionNeedsReconciliationWithEvidence` branch, and the same call in `enqueueStrandedIssueRecovery`). `terminalizeLegacyExecution` then: - updates the run with the same status (only `updated_at` changes); - calls `notifyExecutionRunWork`, `notifyInstructionCollection` and `notifyUnsafeWorkspaceExportRecovery`, which hint the run graph of the task and about 18 queues; - finds the recorded decision and returns. The graph hint re-checks the same task after the queue's 1-second pass interval, so the loop never ends. **Expected behavior** A check that finds nothing to change stores nothing and wakes nothing. Only a real change re-checks the task. **Steps to reproduce** 1. Create a task in `in_progress` assigned to an agent, with a failed legacy run (`adapter_failed`). 2. Start the issue-state recovery work. The boot pass records a fresh-continuation decision for the run. 3. Leave the instance idle. On master: 591,045 statements in 30 minutes for 10 such tasks (fake clock, real PostgreSQL), and one no-op `update "heartbeat_runs"` per task per second. **Paperclip version or commit** master at 7db92ea319. ## What Changed - `server/src/services/legacy-execution-recovery.ts`: - `terminalizeLegacyExecution` returns the unchanged run, without a write or a notification, when all of these are true: the call re-asserts the current status with no patch; the run has no workspace restore failure or required workspace recovery; the task's execution and checkout locks do not point at the run; and either the run needs no reconciliation (`reconcileIfNeeded`), no assignee or reviewer owns a decision, or the decision is already recorded. - Why the short-circuit is safe: stranded-work and quota checks re-assert the run's unchanged terminal status each time they revisit its task. Once the decision is recorded, that re-assertion changes nothing. If it wrote or notified anyway, the notifications would re-check the same task, which would re-assert the run again, about once a second for as long as the task waits. - `findRecordedLegacyDecision` returns the resolved decision already recorded for the run (a reconciliation for this run, an explicit user continuation with a real successor, or a no-replay hold). `decisionOwner` says whether the task's assignee or current reviewer owns a decision about the run. - The recorded-decision query moves into `findRecordedLegacyDecision`, and the owner condition into `decisionOwner`. The full path uses both helpers, so its behavior does not change. - `server/src/__tests__/issue-recovery-recorded-decision-idle.test.ts`: two new tests on real PostgreSQL. They use the real coordinator, the real issue-state recovery work and the production 5-minute backstop on a fake clock. ## Verification - `stays quiet for half an hour once each failed run has its decision`: 10 waiting tasks, zero runs. The test first checks that each failed run has a recorded decision, then counts statements over 30 idle minutes. - master: 591,045 statements (red). - this branch: 0 statements (green). No run is re-written or started, and no task moves. - `still recovers promptly when another run of a waiting task fails`: after 10 idle minutes, a new legacy run of one task fails and notifies like a finished run. - this branch: within 5 seconds, recovery creates the run's `legacy-execution:` hold. Within the next 6 minutes, the fresh-continuation fallback resolves that hold with `executionReconciliation.runId` set to the new run. Over the next 30 minutes there is no `update "heartbeat_runs"` and fewer than 5,000 statements (the assigned-work invariant keeps its own spaced re-checks). - master: 18,000 no-op `update "heartbeat_runs"` statements in the same window (red). - Neighbouring suites pass locally: `legacy-workspace-restore-recovery`, `legacy-workspace-restore-lock-contention`, `legacy-continuation-authority`, `heartbeat-process-recovery`, `issue-recovery-assignments`, `recovery-deadline-ordinary-bootstrap`, `issue-recovery-deadlines` (600 of 601 in one parallel run; `resolves a run's task graph through index lookups` failed once under machine load on a planner row count, and passes alone; it does not use the changed code). - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. The short-circuit applies only when the full path would also write nothing except `updated_at`. Every other case (status change, patch, workspace restore evidence, an owned lock, no recorded decision) takes the unchanged full path. - A re-assertion no longer re-sends notifications. Queues that relied on that repeated hint as a side effect lose it. The first terminalization of a run still sends all notifications, and run, comment and status writes still send their own. - No schema or API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local PostgreSQL test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #21552, #21616, #21617. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ...ue-recovery-recorded-decision-idle.test.ts | 88 ++++++++++++++ .../src/services/legacy-execution-recovery.ts | 113 ++++++++++-------- 2 files changed, 154 insertions(+), 47 deletions(-) create mode 100644 server/src/__tests__/issue-recovery-recorded-decision-idle.test.ts diff --git a/server/src/__tests__/issue-recovery-recorded-decision-idle.test.ts b/server/src/__tests__/issue-recovery-recorded-decision-idle.test.ts new file mode 100644 index 00000000000..3cbbd7d2504 --- /dev/null +++ b/server/src/__tests__/issue-recovery-recorded-decision-idle.test.ts @@ -0,0 +1,88 @@ +import { setImmediate as immediate } from "node:timers/promises"; +import { and, eq, sql } from "drizzle-orm"; +import { expect, it, vi } from "vitest"; +import { agents, heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db"; +import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js"; +import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js"; + +describeIssueRecovery("stranded recovery of runs with a recorded decision", () => { + const state = issueRecoveryFixture(); + const TASKS = 10; + + async function seed() { + const f = await state.fixture(); + const db = state.db; + await db.update(agents).set({ status: "error" }).where(eq(agents.id, f.agentId)); + const tasks = await db.insert(issues).values(Array.from({ length: TASKS }, (_, index) => ({ + companyId: f.companyId, title: `Waiting task ${index}`, status: "in_progress", assigneeAgentId: f.agentId, + }))).returning({ id: issues.id }); + await db.insert(heartbeatRuns).values(tasks.map(task => ({ companyId: f.companyId, agentId: f.agentId, + status: "failed", errorCode: "adapter_failed", finishedAt: new Date(Date.now() - 60 * 60_000), + contextSnapshot: { issueId: task.id } }))); + return { ...f, tasks }; + } + + async function advance(ms: number) { + const until = Date.now() + ms; + while (Date.now() < until) { + await vi.advanceTimersByTimeAsync(Math.min(500, until - Date.now())); + await Promise.allSettled([...state.passes]); await immediate(); + } + } + + async function countStatements(work: () => Promise) { + const statements: string[] = [], prior = state.db.$client.options.debug; + state.db.$client.options.debug = (_connection, query) => { statements.push(query); }; + try { await work(); } finally { state.db.$client.options.debug = prior; } + return statements; + } + + async function startAndRecordDecisions(f: Awaited>) { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); + await state.start({}); + await advance(6 * 60_000); + const decided = await state.db.select({ issueId: issueRecoveryActions.sourceIssueId }).from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, f.companyId), eq(issueRecoveryActions.status, "resolved"), + sql`${issueRecoveryActions.evidence} ? 'executionReconciliation'`)); + expect(new Set(decided.map(row => row.issueId))).toEqual(new Set(f.tasks.map(task => task.id))); + } + + it("stays quiet for half an hour once each failed run has its decision", async () => { + const f = await seed(); + await startAndRecordDecisions(f); + const runsBefore = await state.db.select({ id: heartbeatRuns.id, status: heartbeatRuns.status, updatedAt: heartbeatRuns.updatedAt }) + .from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)); + const idle = await countStatements(() => advance(30 * 60_000)); + expect(idle).toHaveLength(0); + const runsAfter = await state.db.select({ id: heartbeatRuns.id, status: heartbeatRuns.status, updatedAt: heartbeatRuns.updatedAt }) + .from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)); + const byId = (a: { id: string }, b: { id: string }) => a.id.localeCompare(b.id); + expect(runsAfter.sort(byId)).toEqual(runsBefore.sort(byId)); + const statuses = await state.db.select({ status: issues.status }).from(issues).where(eq(issues.companyId, f.companyId)); + expect(statuses.every(row => row.status === "in_progress")).toBe(true); + }, 300_000); + + it("still recovers promptly when another run of a waiting task fails", async () => { + const f = await seed(); + await startAndRecordDecisions(f); + await advance(10 * 60_000); + const target = f.tasks[0].id; + const [failed] = await state.db.transaction(async tx => { + const rows = await tx.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "failed", + errorCode: "adapter_failed", finishedAt: new Date(), contextSnapshot: { issueId: target } }).returning(); + await notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: rows[0].id }); + return rows; + }); + const actionFor = async () => (await state.db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, f.companyId), eq(issueRecoveryActions.sourceIssueId, target), + eq(issueRecoveryActions.fingerprint, `legacy-execution:${failed.id}`))))[0]; + await advance(5_000); + expect(await actionFor()).toMatchObject({ kind: "active_run_watchdog", evidence: expect.objectContaining({ runId: failed.id }) }); + await advance(6 * 60_000); + expect(await actionFor()).toMatchObject({ status: "resolved", + evidence: expect.objectContaining({ executionReconciliation: expect.objectContaining({ runId: failed.id }) }) }); + const after = await countStatements(() => advance(30 * 60_000)); + expect(after.filter(query => query.startsWith("update \"heartbeat_runs\""))).toEqual([]); + expect(after.length).toBeLessThan(TASKS * 500); + }, 300_000); +}); diff --git a/server/src/services/legacy-execution-recovery.ts b/server/src/services/legacy-execution-recovery.ts index 6d48112de2d..0f9fe34c63d 100644 --- a/server/src/services/legacy-execution-recovery.ts +++ b/server/src/services/legacy-execution-recovery.ts @@ -111,6 +111,58 @@ export async function legacyExecutionNeedsReconciliationWithEvidence(db: Db, run && !(await isPreDispatchExecutionWaitVerified(db, run))); } +async function findRecordedLegacyDecision(db: Db, run: Run, taskId: string, resultJson: Run["resultJson"]) { + const [reconciled] = await db.select({ id: issueRecoveryActions.id }) + .from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, run.companyId), + eq(issueRecoveryActions.sourceIssueId, taskId), + eq(issueRecoveryActions.status, "resolved"), + ...(hasRequiredWorkspaceRecovery(resultJson) ? [ + sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`, + ] : []), + or( + sql`${issueRecoveryActions.evidence}->'executionReconciliation'->>'runId' = ${run.id}`, + and( + sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, + sql`exists ( + select 1 from heartbeat_runs successor + where successor.id = ${canonicalExecutionUuid(sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'runId'`)} + and successor.company_id = ${run.companyId} and successor.agent_id = ${run.agentId} + and coalesce(successor.native_issue_id::text, successor.issue_id::text, + successor.context_snapshot->>'issueId') = ${taskId} + and successor.context_snapshot->>'previousRunId' = ${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'previousRunId' + and successor.context_snapshot->'explicitUserContinuation'->>'previousRunId' = ${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'previousRunId' + and successor.context_snapshot->'explicitUserContinuation'->>'commentId' is not distinct from + ${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'commentId' + )`, + ), + and( + sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, + or(sql`${issueRecoveryActions.evidence}->>'workspaceRestoreFailure' = 'restore_unsafe_archive'`, + sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`), + sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`, + ), + and( + eq(issueRecoveryActions.cause, LEGACY_RECOVERY_CAUSE), + eq(issueRecoveryActions.fingerprint, `legacy-execution:${run.id}`), + sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, + sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`, + ), + ), + )).limit(1); + return reconciled ?? null; +} + +function decisionOwner(task: typeof issues.$inferSelect, run: Run, retainedLeaseCount: number) { + const review = task.status === "in_review" ? parseIssueExecutionState(task.executionState) : null; + const isCurrentReviewer = review?.status === "pending" && + review.currentParticipant?.type === "agent" && review.currentParticipant.agentId === run.agentId; + const owns = !isSupersededConversationRun(task, run) && + (task.assigneeAgentId === run.agentId || isCurrentReviewer) && + (!["done", "cancelled"].includes(task.status) || retainedLeaseCount > 0); + return { owns, isCurrentReviewer }; +} + /** Persist the failed legacy run, owned lock release and operator decision together. */ export async function terminalizeLegacyExecution(input: { db: Db; @@ -152,6 +204,16 @@ export async function terminalizeLegacyExecution(input: { )).for("update"); if (!current || (current.runtimeMode !== "legacy" && !(input.recordRestoreFailureOnly && current.runtimeMode === "native")) || (!input.recordRestoreFailureOnly && !(input.fromStatuses ?? [run.status]).includes(current.status))) return null; + if (!input.recordRestoreFailureOnly && current.status === status && Object.keys(patch ?? {}).length === 0 + && !hasWorkspaceRestoreFailure(current.resultJson) && !hasRequiredWorkspaceRecovery(current.resultJson) + && task?.executionRunId !== run.id && task?.checkoutRunId !== run.id) { + const [unchanged] = await tx.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId))); + if (input.reconcileIfNeeded && !(await legacyExecutionNeedsReconciliationWithEvidence(tx as unknown as Db, unchanged))) + return unchanged; + if (!task || !decisionOwner(task, unchanged, 0).owns) return unchanged; + if (await findRecordedLegacyDecision(tx as unknown as Db, run, task.id, unchanged.resultJson)) return unchanged; + } if (patch?.resultJson !== undefined) patch = { ...patch, resultJson: preserveWorkspaceRestoreRecoveryMetadata(current.resultJson, input.recordRestoreFailureOnly ? { ...current.resultJson, ...patch.resultJson } : patch.resultJson), @@ -255,58 +317,15 @@ export async function terminalizeLegacyExecution(input: { fingerprint, attemptCount: 1 }); return updated; } - const review = task?.status === "in_review" ? parseIssueExecutionState(task.executionState) : null; - const isCurrentReviewer = review?.status === "pending" && - review.currentParticipant?.type === "agent" && review.currentParticipant.agentId === run.agentId; - if ( - task && - !isSupersededConversationRun(task, updated) && - (task.assigneeAgentId === run.agentId || isCurrentReviewer) && - (!["done", "cancelled"].includes(task.status) || retainedLeaseIds.length > 0) - ) { + const owner = task ? decisionOwner(task, updated, retainedLeaseIds.length) : null; + if (task && owner?.owns) { + const { isCurrentReviewer } = owner; // Periodic stranded-work checks may revisit this terminal run before its // reconciled or explicitly requested continuation is dispatched. Preserve // its recorded decision instead of creating another hold. The explicit // receipt must identify an actual successor under the same task/agent; // retained user intent remains authoritative after that successor finishes. - const [reconciled] = await tx.select({ id: issueRecoveryActions.id }) - .from(issueRecoveryActions).where(and( - eq(issueRecoveryActions.companyId, run.companyId), - eq(issueRecoveryActions.sourceIssueId, task.id), - eq(issueRecoveryActions.status, "resolved"), - ...(hasRequiredWorkspaceRecovery(updated.resultJson) ? [ - sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`, - ] : []), - or( - sql`${issueRecoveryActions.evidence}->'executionReconciliation'->>'runId' = ${run.id}`, - and( - sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, - sql`exists ( - select 1 from heartbeat_runs successor - where successor.id = ${canonicalExecutionUuid(sql`${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'runId'`)} - and successor.company_id = ${run.companyId} and successor.agent_id = ${run.agentId} - and coalesce(successor.native_issue_id::text, successor.issue_id::text, - successor.context_snapshot->>'issueId') = ${task.id} - and successor.context_snapshot->>'previousRunId' = ${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'previousRunId' - and successor.context_snapshot->'explicitUserContinuation'->>'previousRunId' = ${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'previousRunId' - and successor.context_snapshot->'explicitUserContinuation'->>'commentId' is not distinct from - ${issueRecoveryActions.evidence}->'explicitUserContinuation'->>'commentId' - )`, - ), - and( - sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, - or(sql`${issueRecoveryActions.evidence}->>'workspaceRestoreFailure' = 'restore_unsafe_archive'`, - sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`), - sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`, - ), - and( - eq(issueRecoveryActions.cause, LEGACY_RECOVERY_CAUSE), - eq(issueRecoveryActions.fingerprint, `legacy-execution:${run.id}`), - sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, - sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`, - ), - ), - )).limit(1); + const reconciled = await findRecordedLegacyDecision(tx as unknown as Db, run, task.id, updated.resultJson); if (reconciled) return updated; await issueRecoveryActionService(tx as unknown as Db).upsertSourceScoped({ companyId: run.companyId,