From dcba9628c79a7dec6f079be3e16046316547d04f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 18:06:38 -0600 Subject: [PATCH] perf(recovery): keep a finished task's settled attention settled and halve graph hint fan-out (#21676) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Issue recovery records "recovery actions" on tasks. A sweep resolves the active action of a task that is already done or cancelled. > - On an idle hosted instance, a debug-log run (from #21640) named a ping-pong. A native worker re-surfaces a board attention for 13 finished tasks. Recovery resolves it as `source_terminal`. The unique "one active action" index then lets the worker insert it again, about every 26 seconds per task. That built 45,008 rows since 2026-10-21. Each insert and each resolve also signals run dispatch and the task's recovery graph, which wakes both sides again. > - Separately, every task-graph recovery hint is published twice per task queue: once as the graph hint and once as a `{companyId, backstop: true}` twin. That is half of all idle signals, for a backstop that is time-gated anyway. > - This pull request breaks the ping-pong from the recovery side and removes the backstop twin. > - The benefit is that a finished task's settled attention stays settled and wakes nothing. Each graph hint costs half the signals. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** With zero runs active, recovery and native workers kept re-creating and re-resolving the same board attention on 13 done or cancelled tasks. The rows are `active_run_watchdog`, owner `board`, cause `native_restart_recovery_blocked`. The loop added about one row per task every ~26 s and about 2.6 work-signal commits per second. **Expected behavior** Once recovery settles a finished task's attention, the same attention identity is not surfaced again, and settling it wakes nothing that a finished task cannot use. **Steps to reproduce** 1. Create a cancelled task. 2. Surface the native "controller process space unverified" attention for it (same cause and fingerprint each time), then run `reconcileActiveRecoveryActions`. 3. Repeat step 2. On master each cycle inserts a new active row and resolves it, and each resolve signals the task queues and run dispatch. **Paperclip version or commit** master at 36101ebcb9. ## What Changed - `server/src/services/issue-recovery-actions.ts`: - **No-match skip.** `resolveActiveForIssue` first reads whether any action matches. When none does, it returns `null` with no transaction and no signal. Before, `writeExecutionControl` always published run-dispatch, graph-recovery and execution hints first. - **Settled-attention guard.** `upsertSourceScoped` returns the existing resolved action instead of inserting when ALL of these hold: - the task is done or cancelled; - it has no active action; - an action with the same cause and fingerprint was already resolved with `source_terminal`. The guard writes no row and sends no signal, and the caller gets that resolved action. It covers only the exact identity that recovery itself settled because the task finished. A new fingerprint (renewed controller authority), a different cause, or a reopened task still creates an action. - **Quiet terminal resolve.** A new `sourceTerminal` resolve mode. It updates the row only while the task is still done or cancelled. It still notifies native recovery notices, unsafe-workspace export recovery, and a preserved native finalization waiter. It does NOT signal run dispatch, the task's recovery graph, or the execution disposition/continuation queues: a finished task has no run to dispatch and no assignment to recover. - `server/src/services/recovery/service.ts`: the `source_terminal` branch of `reconcileActiveRecoveryActions` uses `sourceTerminal: true`. - `server/src/services/issue-recovery-notifications.ts`: `notifyIssueGraphRecovery` publishes one graph hint per task queue and no longer adds a `{companyId, backstop: true}` hint per queue. A new `issueRecoveryBackstopHint(companyId)` builds that hint string. - `server/src/services/recovery/work.ts`: when a task queue loads a graph scope, it arms its own backstop in process (`worker.invalidate(issueRecoveryBackstopHint(companyId))`). The #21552 guarantee stays: any graph activity arms the company backstop, which runs at most once per backstop interval after the last full pass. It no longer costs one signal per queue per hint. - `server/src/__tests__/issue-recovery-actions.test.ts`: the fake-DB unit tests accept the guard's extra read. - `server/src/__tests__/issue-recovery-signal-fanout.test.ts` (new, real PostgreSQL), six tests: - one graph hint per task queue and no backstop hint; - a resolve with no matching action returns null and signals nothing; - an escalated action on a finished task is still resolved and still signals; - the ping-pong cycle: surface then reconcile, five times, leaves one row and no stranded or active-recovery signals; - settling a finished task's attention sends no task-queue, run-dispatch or active-recovery signals; - a renewed authority and a reopened task still get a new attention, which stays active on a reopened task. ## Verification - New test file: - master: 4 of 6 fail (red). - 8 task-queue hints per graph hint instead of 4. - A no-match resolve publishes 18 settled signals. - Cycle 2 inserts a new active row; master builds a new row every cycle. - Settling a finished task's attention signals the task queues and run dispatch. - This branch: 6 of 6 pass (green). - Regression on real PostgreSQL: - heartbeat-native-restart-suspension, native-recovery-notice-work, issue-recovery-deadlines (including the #21552 backstop tests), assigned-work-invariant, execution-control-work, issue-recovery-actions, issue-recovery-work-lifecycle, issue-recovery-assignments, execution-recovery-resolution, automatic-run-brake-recovery-passes: all pass (381 tests). - Four unit tests in issue-recovery-actions use a hand-written fake DB. They needed `innerJoin` on the fake query and an expected count of two selects, because the guard adds one read. - recovery-deadline-acceptance: passes. - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - The guard returns a resolved action to a caller that asked to surface one. I checked the callers: the one that loops (`surfaceNativeControllerNamespaceAttention`) ignores the return value. The others only reach the guard for a finished task whose identical attention recovery already settled, where re-surfacing it would be resolved again within one sweep. - The terminal resolve no longer wakes run dispatch or the task's graph. Completing or cancelling a task already sends those signals itself. - This does not stop the native worker from trying to surface attention for finished tasks. That fix belongs in the native owner-release code, which another change owns. - The ~45k historical resolved rows on the hosted instance stay as they are. A cleanup would need a migration or an operator script that deletes resolved `native_restart_recovery_blocked` rows whose source is done or cancelled, keeping the newest per (task, fingerprint) so the guard still matches. That is not part of this change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local PostgreSQL test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #21552, #21621, #21640. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../__tests__/issue-recovery-actions.test.ts | 7 +- .../issue-recovery-signal-fanout.test.ts | 133 ++++++++++++++++++ server/src/services/issue-recovery-actions.ts | 54 ++++++- .../services/issue-recovery-notifications.ts | 9 +- server/src/services/recovery/service.ts | 5 +- server/src/services/recovery/work.ts | 3 +- 6 files changed, 195 insertions(+), 16 deletions(-) create mode 100644 server/src/__tests__/issue-recovery-signal-fanout.test.ts diff --git a/server/src/__tests__/issue-recovery-actions.test.ts b/server/src/__tests__/issue-recovery-actions.test.ts index 372cd77ba5e..36c2d056587 100644 --- a/server/src/__tests__/issue-recovery-actions.test.ts +++ b/server/src/__tests__/issue-recovery-actions.test.ts @@ -95,7 +95,7 @@ describe("issueRecoveryActionService", () => { const error: Error & { cause?: unknown } = new Error("Wrapped database failure"); error.cause = kind === "unrelated unique constraint" ? { code: "23505", constraint_name: "other_unique_constraint" } : kind === "other SQLSTATE" ? { code: "23503", constraint_name: constraint } : error; - const query = { from() { return this; }, where() { return this; }, orderBy() { return this; }, + const query = { from() { return this; }, innerJoin() { return this; }, where() { return this; }, orderBy() { return this; }, limit() { return Promise.resolve([]); } }; const fakeDb = { async transaction(fn: (tx: unknown) => Promise): Promise { return fn(fakeDb); }, @@ -106,7 +106,7 @@ describe("issueRecoveryActionService", () => { await expect(issueRecoveryActionService(fakeDb as never).upsertSourceScoped({ companyId: "company", sourceIssueId: "issue", kind: "missing_disposition", cause: "saved-debt", fingerprint: "same-source", nextAction: "Restore existing work." })) .rejects.toBe(error); - expect(fakeDb.select).toHaveBeenCalledOnce(); + expect(fakeDb.select).toHaveBeenCalledTimes(2); expect(fakeDb.insert).toHaveBeenCalledOnce(); }); it("does not reactivate an action resolved between the active read and update", async () => { @@ -118,6 +118,9 @@ describe("issueRecoveryActionService", () => { from() { return this; }, + innerJoin() { + return this; + }, where() { return this; }, diff --git a/server/src/__tests__/issue-recovery-signal-fanout.test.ts b/server/src/__tests__/issue-recovery-signal-fanout.test.ts new file mode 100644 index 00000000000..585f63efcc3 --- /dev/null +++ b/server/src/__tests__/issue-recovery-signal-fanout.test.ts @@ -0,0 +1,133 @@ +import { and, eq } from "drizzle-orm"; +import { expect, it } from "vitest"; +import { issueRecoveryActions, issues } from "@paperclipai/db"; +import { subscribeDatabaseWork } from "@paperclipai/db/work-signals"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; +import { issueRecoveryActionService } from "../services/issue-recovery-actions.js"; +import { notifyIssueGraphRecovery } from "../services/issue-recovery-notifications.js"; +import { recoveryService } from "../services/recovery/service.js"; +import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js"; + +const TASK_QUEUES = [DELIVERY_QUEUES.strandedAssignments, DELIVERY_QUEUES.assignedWorkInvariant, + DELIVERY_QUEUES.resolvedDependencies, DELIVERY_QUEUES.staleIssueLocks]; +const WATCHED = [...TASK_QUEUES, DELIVERY_QUEUES.orphanBlockers, DELIVERY_QUEUES.activeRecoveryActions, + DELIVERY_QUEUES.taskWatchdogs, DELIVERY_QUEUES.queuedRunDispatch, DELIVERY_QUEUES.executionDisposition]; + +describeIssueRecovery("issue recovery signal fan-out", () => { + const state = issueRecoveryFixture(); + + function record(owner: object) { + const settled: Array<[string, string[]]> = []; + const stops = WATCHED.map(queue => subscribeDatabaseWork(owner, queue, (event, hints) => { + if (event === "settled") settled.push([queue, [...hints ?? []]]); + })); + return { settled, stop: () => { for (const stop of stops) stop(); } }; + } + + it("publishes one graph hint per task queue and no per-queue backstop twin", async () => { + const f = await state.fixture(); + const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Task", status: "todo" }).returning(); + const recorder = record(state.db); + try { + await state.db.transaction(tx => notifyIssueGraphRecovery(tx, { companyId: f.companyId, issueId: task.id })); + } finally { recorder.stop(); } + const taskHints = recorder.settled.filter(([queue]) => TASK_QUEUES.includes(queue as never)).flatMap(([, hints]) => hints); + expect(taskHints).toHaveLength(TASK_QUEUES.length); + expect(taskHints.every(hint => JSON.parse(hint).graphIssueId === task.id)).toBe(true); + expect(recorder.settled.flatMap(([, hints]) => hints).some(hint => hint.includes("\"backstop\""))).toBe(false); + }); + + it("resolves nothing and signals nothing when the issue has no matching active recovery action", async () => { + const f = await state.fixture(); + const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Done", status: "done" }).returning(); + const [resolved] = await state.db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: task.id, + kind: "stranded_assigned_issue", status: "resolved", ownerType: "agent", ownerAgentId: f.agentId, + cause: "stranded_assigned_issue", fingerprint: "already-resolved", nextAction: "Done", outcome: "restored", + resolvedAt: new Date(), evidence: {} }).returning(); + const recorder = record(state.db); + const actions = issueRecoveryActionService(state.db); + try { + expect(await actions.resolveActiveForIssue({ companyId: f.companyId, sourceIssueId: task.id, actionId: resolved.id, + status: "resolved", outcome: "restored", resolutionNote: "source_terminal" })).toBeNull(); + expect(await actions.resolveActiveForIssue({ companyId: f.companyId, sourceIssueId: task.id, + status: "cancelled", outcome: "cancelled", resolutionNote: "source_state_changed" })).toBeNull(); + } finally { recorder.stop(); } + expect(recorder.settled).toEqual([]); + }); + + it("still resolves an active action and wakes its task graph", async () => { + const f = await state.fixture(); + const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Done", status: "done" }).returning(); + const [active] = await state.db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: task.id, + kind: "stranded_assigned_issue", status: "escalated", ownerType: "agent", ownerAgentId: f.agentId, + cause: "stranded_assigned_issue", fingerprint: "escalated", nextAction: "Restore", evidence: {} }).returning(); + const recorder = record(state.db); + let result; + try { + result = await issueRecoveryActionService(state.db).resolveActiveForIssue({ companyId: f.companyId, sourceIssueId: task.id, + actionId: active.id, status: "resolved", outcome: "restored", resolutionNote: "source_terminal" }); + } finally { recorder.stop(); } + expect(result).toMatchObject({ id: active.id, status: "resolved", outcome: "restored" }); + expect(recorder.settled.filter(([queue]) => queue === DELIVERY_QUEUES.strandedAssignments).flatMap(([, hints]) => hints) + .map(hint => JSON.parse(hint))).toContainEqual({ companyId: f.companyId, graphIssueId: task.id }); + }); + + async function blockedAttention(companyId: string, sourceIssueId: string, agentId: string, fingerprint = "native-controller-space:run:digest") { + return issueRecoveryActionService(state.db).upsertSourceScoped({ companyId, sourceIssueId, kind: "active_run_watchdog", + ownerType: "board", returnOwnerAgentId: agentId, cause: "native_restart_recovery_blocked", fingerprint, + evidence: { runId: "run", reason: "controller_process_space_unverified", executorStopEvidence: "unverified" }, + nextAction: "Wait for the previous controller to stop" }); + } + + it("does not re-open attention that recovery settled because its task finished", async () => { + const f = await state.fixture(); + const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Cancelled", status: "cancelled" }).returning(); + const recovery = recoveryService(state.db, { enqueueWakeup: async () => null }); + await blockedAttention(f.companyId, task.id, f.agentId); + await recovery.reconcileActiveRecoveryActions({ companyId: f.companyId }); + const recorder = record(state.db); + try { + for (let cycle = 0; cycle < 5; cycle++) { + const surfaced = await blockedAttention(f.companyId, task.id, f.agentId); + expect(surfaced.status).toBe("resolved"); + await recovery.reconcileActiveRecoveryActions({ companyId: f.companyId }); + } + } finally { recorder.stop(); } + const rows = await state.db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, f.companyId), eq(issueRecoveryActions.sourceIssueId, task.id))); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ status: "resolved", outcome: "restored", resolutionNote: "source_terminal" }); + expect(recorder.settled.filter(([queue]) => queue === DELIVERY_QUEUES.strandedAssignments + || queue === DELIVERY_QUEUES.activeRecoveryActions)).toEqual([]); + }); + + it("settles a finished task's attention without waking run dispatch or its task graph", async () => { + const f = await state.fixture(); + const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Done", status: "done" }).returning(); + await blockedAttention(f.companyId, task.id, f.agentId); + const recorder = record(state.db); + try { + expect(await recoveryService(state.db, { enqueueWakeup: async () => null }).reconcileActiveRecoveryActions({ companyId: f.companyId })) + .toMatchObject({ resolved: 1 }); + } finally { recorder.stop(); } + expect(recorder.settled.filter(([queue]) => queue !== DELIVERY_QUEUES.executionDisposition) + .filter(([queue]) => [...TASK_QUEUES, DELIVERY_QUEUES.queuedRunDispatch, DELIVERY_QUEUES.activeRecoveryActions].includes(queue as never))) + .toEqual([]); + }); + + it("surfaces attention again once the task is reopened, and for a new controller authority", async () => { + const f = await state.fixture(); + const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Cancelled", status: "cancelled" }).returning(); + const recovery = recoveryService(state.db, { enqueueWakeup: async () => null }); + await blockedAttention(f.companyId, task.id, f.agentId); + await recovery.reconcileActiveRecoveryActions({ companyId: f.companyId }); + expect((await blockedAttention(f.companyId, task.id, f.agentId, "native-controller-space:run:renewed")).status).toBe("active"); + await recovery.reconcileActiveRecoveryActions({ companyId: f.companyId }); + await state.db.update(issues).set({ status: "todo" }).where(eq(issues.id, task.id)); + const reopened = await blockedAttention(f.companyId, task.id, f.agentId); + expect(reopened.status).toBe("active"); + await recovery.reconcileActiveRecoveryActions({ companyId: f.companyId }); + expect((await state.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, reopened.id)))[0].status) + .toBe("active"); + }); +}); diff --git a/server/src/services/issue-recovery-actions.ts b/server/src/services/issue-recovery-actions.ts index f9509f33210..9fb786b91fc 100644 --- a/server/src/services/issue-recovery-actions.ts +++ b/server/src/services/issue-recovery-actions.ts @@ -6,7 +6,7 @@ import { writeExecutionControl } from "./execution-control-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { and, desc, eq, inArray, isNull, isNotNull, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; -import { heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, workspaceOperations } from "@paperclipai/db"; +import { heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, workspaceOperations } from "@paperclipai/db"; import { isUuidLike } from "@paperclipai/shared"; import type { IssueRecoveryAction, @@ -86,8 +86,11 @@ export type ResolveIssueRecoveryActionInput = { status: Extract; outcome: IssueRecoveryActionOutcome; resolutionNote?: string | null; + sourceTerminal?: boolean; }; +export const SOURCE_TERMINAL_RESOLUTION = "source_terminal"; + /** A spent saved-work recovery is described from its current state. Text saved * by an older release could ask a person to inspect the run's environment. */ function readModelNextAction(row: IssueRecoveryActionRow): string { @@ -390,6 +393,21 @@ export function issueRecoveryActionService(db: DbOrTransaction) { const existing = await getActiveForIssue(input.companyId, input.sourceIssueId); const now = new Date(); if (existing && input.createOnly) return existing; + if (!existing) { + const [source] = await db.select({ status: issues.status }).from(issues) + .where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.sourceIssueId))).limit(1); + if (source && (source.status === "done" || source.status === "cancelled")) { + const [settled] = await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, input.companyId), + eq(issueRecoveryActions.sourceIssueId, input.sourceIssueId), + eq(issueRecoveryActions.status, "resolved"), + eq(issueRecoveryActions.cause, input.cause), + eq(issueRecoveryActions.fingerprint, input.fingerprint), + eq(issueRecoveryActions.resolutionNote, SOURCE_TERMINAL_RESOLUTION), + )).limit(1); + if (settled) return toReadModel(settled); + } + } const ownerType = input.ownerType ?? (input.ownerAgentId ? "agent" : "board"); if (existing) { // A distinct failure identity must not overwrite the active action of a @@ -619,12 +637,11 @@ export function issueRecoveryActionService(db: DbOrTransaction) { predicates.push(eq(issueRecoveryActions.fingerprint, input.fingerprint)); } - const [updated] = await writeExecutionControl(dbOrTx, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => { - const pending = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions) - .where(and(...predicates)).limit(1); - if (!pending.length) return []; - // Resolving another attention releases this issue's preserved Native - // waiter to one normal inspection. No recurring monitor or new turn. + const [candidate] = await dbOrTx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions) + .where(and(...predicates)).limit(1); + if (!candidate) return null; + + const releaseWaiter = async (tx: DbOrTransaction) => { const [waiter] = await tx.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations) .innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)).where(and( eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.sourceIssueId), @@ -632,6 +649,29 @@ export function issueRecoveryActionService(db: DbOrTransaction) { isNotNull(nativeRunFinalizations.leaseOwner), inArray(heartbeatRuns.status, ["failed", "timed_out", "interrupted"]), )).limit(1); if (waiter) await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization); + }; + + if (input.sourceTerminal) { + const [settled] = await dbOrTx.transaction(async tx => { + await releaseWaiter(tx as unknown as DbOrTransaction); + await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); + await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); + return tx.update(issueRecoveryActions) + .set({ status: input.status, outcome: input.outcome, resolutionNote: input.resolutionNote ?? null, resolvedAt: now, updatedAt: now }) + .where(and(...predicates, sql`exists (select 1 from ${issues} where ${issues.id} = ${issueRecoveryActions.sourceIssueId} + and ${issues.companyId} = ${issueRecoveryActions.companyId} and ${issues.status} in ('done', 'cancelled'))`)) + .returning(); + }); + return settled ? toReadModel(settled) : null; + } + + const [updated] = await writeExecutionControl(dbOrTx, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => { + const pending = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions) + .where(and(...predicates)).limit(1); + if (!pending.length) return []; + // Resolving another attention releases this issue's preserved Native + // waiter to one normal inspection. No recurring monitor or new turn. + await releaseWaiter(tx as unknown as DbOrTransaction); await notifyIssueGraphRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); diff --git a/server/src/services/issue-recovery-notifications.ts b/server/src/services/issue-recovery-notifications.ts index ec3770b7c4d..08751b36220 100644 --- a/server/src/services/issue-recovery-notifications.ts +++ b/server/src/services/issue-recovery-notifications.ts @@ -26,6 +26,10 @@ export async function notifyIssueRecovery(transaction: object, scope?: IssueReco } } +export function issueRecoveryBackstopHint(companyId: string) { + return JSON.stringify({ companyId, backstop: true }); +} + export function decodeIssueRecoveryHint(hint: string): IssueRecoveryScope | undefined { return hint === "*" ? undefined : JSON.parse(hint) as IssueRecoveryScope; } @@ -44,10 +48,7 @@ export async function notifyIssueGraphRecovery(transaction: object, const companyHint = JSON.stringify({ companyId: scope.companyId }); for (const queue of COMPANY_QUEUES) await notifyDeliveryWork(transaction, queue, companyHint); const graphHint = JSON.stringify({ companyId: scope.companyId, graphRunId: scope.runId ?? undefined, graphIssueId: scope.issueId ?? undefined }); - for (const queue of TASK_QUEUES) { - await notifyDeliveryWork(transaction, queue, graphHint); - await notifyDeliveryWork(transaction, queue, JSON.stringify({ companyId: scope.companyId, backstop: true })); - } + for (const queue of TASK_QUEUES) await notifyDeliveryWork(transaction, queue, graphHint); } export async function resolveIssueGraph(db: Db, scope: { companyId: string; runId?: string; issueId?: string }): Promise { diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index cd12fdcd29c..7a855a721d0 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -127,7 +127,7 @@ import { emitAgentTaskRun } from "../agent-task-run-telemetry.js"; import { budgetService } from "../budgets.js"; import { isBoardOwnedBudgetBlock } from "../budget-block-ownership.js"; import { unadmittedChatWakeupCondition } from "../durable-chat-wakeup.js"; -import { issueRecoveryActionService } from "../issue-recovery-actions.js"; +import { issueRecoveryActionService, SOURCE_TERMINAL_RESOLUTION } from "../issue-recovery-actions.js"; import { PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS } from "../system-recovery-state.js"; import { parseProviderQuotaClockReset } from "../provider-quota-reset.js"; import { hasUnverifiedNativeWorkspaceFinalizationOwner } from "../native-runtime/native-workspace-finalization-recovery-state.js"; @@ -3670,7 +3670,8 @@ export function recoveryService( actionId: action.id, status: "resolved", outcome: "restored", - resolutionNote: "source_terminal", + resolutionNote: SOURCE_TERMINAL_RESOLUTION, + sourceTerminal: true, }); if (resolved) { result.resolved += 1; diff --git a/server/src/services/recovery/work.ts b/server/src/services/recovery/work.ts index 7c9e02c2271..6fae89d60e0 100644 --- a/server/src/services/recovery/work.ts +++ b/server/src/services/recovery/work.ts @@ -3,7 +3,7 @@ import { and, asc, eq, gt, inArray, isNotNull, isNull, or, sql, type SQL } from import { issueRecoveryActions, issueRelations, issues, type Db } from "@paperclipai/db"; import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; import { DELIVERY_QUEUES, type DeliveryQueue } from "../delivery-work-notifications.js"; -import { decodeIssueRecoveryHint, ISSUE_RECOVERY_BACKSTOP_MS, issueRecoveryScopeMatches, resolveIssueGraph, type IssueRecoveryScope } +import { decodeIssueRecoveryHint, ISSUE_RECOVERY_BACKSTOP_MS, issueRecoveryBackstopHint, issueRecoveryScopeMatches, resolveIssueGraph, type IssueRecoveryScope } from "../issue-recovery-notifications.js"; import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; import { workSignalDebug } from "@paperclipai/db/work-signal-debug"; @@ -71,6 +71,7 @@ export function registerIssueRecoveryWork( scope.wholeCompany = ids === null; } async function toCompany(scope: IssueRecoveryScope | undefined, worker: { invalidate(hint?: string): void }) { + if (scope && isGraph(scope)) worker.invalidate(issueRecoveryBackstopHint(scope.companyId)); await resolveGraph(scope); if (!scope?.wholeCompany) return false; worker.invalidate(JSON.stringify({ companyId: scope.companyId }));