mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
perf(recovery): keep a finished task's settled attention settled and halve graph hint fan-out (#21676)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Issue recovery records "recovery actions" on tasks. A sweep resolves
the active action of a task that is already done or cancelled.
> - On an idle hosted instance, a debug-log run (from #21640) named a
ping-pong. A native worker re-surfaces a board attention for 13 finished
tasks. Recovery resolves it as `source_terminal`. The unique "one active
action" index then lets the worker insert it again, about every 26
seconds per task. That built 45,008 rows since 2026-10-21. Each insert
and each resolve also signals run dispatch and the task's recovery
graph, which wakes both sides again.
> - Separately, every task-graph recovery hint is published twice per
task queue: once as the graph hint and once as a `{companyId, backstop:
true}` twin. That is half of all idle signals, for a backstop that is
time-gated anyway.
> - This pull request breaks the ping-pong from the recovery side and
removes the backstop twin.
> - The benefit is that a finished task's settled attention stays
settled and wakes nothing. Each graph hint costs half the signals.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
With zero runs active, recovery and native workers kept re-creating and
re-resolving the same board attention on 13 done or cancelled tasks. The
rows are `active_run_watchdog`, owner `board`, cause
`native_restart_recovery_blocked`. The loop added about one row per task
every ~26 s and about 2.6 work-signal commits per second.
**Expected behavior**
Once recovery settles a finished task's attention, the same attention
identity is not surfaced again, and settling it wakes nothing that a
finished task cannot use.
**Steps to reproduce**
1. Create a cancelled task.
2. Surface the native "controller process space unverified" attention
for it (same cause and fingerprint each time), then run
`reconcileActiveRecoveryActions`.
3. Repeat step 2. On master each cycle inserts a new active row and
resolves it, and each resolve signals the task queues and run dispatch.
**Paperclip version or commit**
master at 36101ebcb9.
## What Changed
- `server/src/services/issue-recovery-actions.ts`:
- **No-match skip.** `resolveActiveForIssue` first reads whether any
action matches. When none does, it returns `null` with no transaction
and no signal. Before, `writeExecutionControl` always published
run-dispatch, graph-recovery and execution hints first.
- **Settled-attention guard.** `upsertSourceScoped` returns the existing
resolved action instead of inserting when ALL of these hold:
- the task is done or cancelled;
- it has no active action;
- an action with the same cause and fingerprint was already resolved
with `source_terminal`.
The guard writes no row and sends no signal, and the caller gets that
resolved action. It covers only the exact identity that recovery itself
settled because the task finished. A new fingerprint (renewed controller
authority), a different cause, or a reopened task still creates an
action.
- **Quiet terminal resolve.** A new `sourceTerminal` resolve mode. It
updates the row only while the task is still done or cancelled. It still
notifies native recovery notices, unsafe-workspace export recovery, and
a preserved native finalization waiter. It does NOT signal run dispatch,
the task's recovery graph, or the execution disposition/continuation
queues: a finished task has no run to dispatch and no assignment to
recover.
- `server/src/services/recovery/service.ts`: the `source_terminal`
branch of `reconcileActiveRecoveryActions` uses `sourceTerminal: true`.
- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery` publishes one graph hint per task queue and
no longer adds a `{companyId, backstop: true}` hint per queue. A new
`issueRecoveryBackstopHint(companyId)` builds that hint string.
- `server/src/services/recovery/work.ts`: when a task queue loads a
graph scope, it arms its own backstop in process
(`worker.invalidate(issueRecoveryBackstopHint(companyId))`). The #21552
guarantee stays: any graph activity arms the company backstop, which
runs at most once per backstop interval after the last full pass. It no
longer costs one signal per queue per hint.
- `server/src/__tests__/issue-recovery-actions.test.ts`: the fake-DB
unit tests accept the guard's extra read.
- `server/src/__tests__/issue-recovery-signal-fanout.test.ts` (new, real
PostgreSQL), six tests:
- one graph hint per task queue and no backstop hint;
- a resolve with no matching action returns null and signals nothing;
- an escalated action on a finished task is still resolved and still
signals;
- the ping-pong cycle: surface then reconcile, five times, leaves one
row and no stranded or active-recovery signals;
- settling a finished task's attention sends no task-queue, run-dispatch
or active-recovery signals;
- a renewed authority and a reopened task still get a new attention,
which stays active on a reopened task.
## Verification
- New test file:
- master: 4 of 6 fail (red).
- 8 task-queue hints per graph hint instead of 4.
- A no-match resolve publishes 18 settled signals.
- Cycle 2 inserts a new active row; master builds a new row every cycle.
- Settling a finished task's attention signals the task queues and run
dispatch.
- This branch: 6 of 6 pass (green).
- Regression on real PostgreSQL:
- heartbeat-native-restart-suspension, native-recovery-notice-work,
issue-recovery-deadlines (including the #21552 backstop tests),
assigned-work-invariant, execution-control-work, issue-recovery-actions,
issue-recovery-work-lifecycle, issue-recovery-assignments,
execution-recovery-resolution, automatic-run-brake-recovery-passes: all
pass (381 tests).
- Four unit tests in issue-recovery-actions use a hand-written fake DB.
They needed `innerJoin` on the fake query and an expected count of two
selects, because the guard adds one read.
- recovery-deadline-acceptance: passes.
- `tsc --noEmit -p server/tsconfig.json` passes.
## Risks
- The guard returns a resolved action to a caller that asked to surface
one. I checked the callers: the one that loops
(`surfaceNativeControllerNamespaceAttention`) ignores the return value.
The others only reach the guard for a finished task whose identical
attention recovery already settled, where re-surfacing it would be
resolved again within one sweep.
- The terminal resolve no longer wakes run dispatch or the task's graph.
Completing or cancelling a task already sends those signals itself.
- This does not stop the native worker from trying to surface attention
for finished tasks. That fix belongs in the native owner-release code,
which another change owns.
- The ~45k historical resolved rows on the hosted instance stay as they
are. A cleanup would need a migration or an operator script that deletes
resolved `native_restart_recovery_blocked` rows whose source is done or
cancelled, keeping the newest per (task, fingerprint) so the guard still
matches. That is not part of this change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #21552, #21621, #21640.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
97a167093d
commit
dcba9628c7
6 files changed
+195
-16
No files matched your search
@@ -95,7 +95,7 @@ describe("issueRecoveryActionService", () => {
|
||||
const error: Error & { cause?: unknown } = new Error("Wrapped database failure");
|
||||
error.cause = kind === "unrelated unique constraint" ? { code: "23505", constraint_name: "other_unique_constraint" }
|
||||
: kind === "other SQLSTATE" ? { code: "23503", constraint_name: constraint } : error;
|
||||
const query = { from() { return this; }, where() { return this; }, orderBy() { return this; },
|
||||
const query = { from() { return this; }, innerJoin() { return this; }, where() { return this; }, orderBy() { return this; },
|
||||
limit() { return Promise.resolve([]); } };
|
||||
const fakeDb = {
|
||||
async transaction<T>(fn: (tx: unknown) => Promise<T>): Promise<T> { return fn(fakeDb); },
|
||||
@@ -106,7 +106,7 @@ describe("issueRecoveryActionService", () => {
|
||||
await expect(issueRecoveryActionService(fakeDb as never).upsertSourceScoped({ companyId: "company", sourceIssueId: "issue",
|
||||
kind: "missing_disposition", cause: "saved-debt", fingerprint: "same-source", nextAction: "Restore existing work." }))
|
||||
.rejects.toBe(error);
|
||||
expect(fakeDb.select).toHaveBeenCalledOnce();
|
||||
expect(fakeDb.select).toHaveBeenCalledTimes(2);
|
||||
expect(fakeDb.insert).toHaveBeenCalledOnce();
|
||||
});
|
||||
it("does not reactivate an action resolved between the active read and update", async () => {
|
||||
@@ -118,6 +118,9 @@ describe("issueRecoveryActionService", () => {
|
||||
from() {
|
||||
return this;
|
||||
},
|
||||
innerJoin() {
|
||||
return this;
|
||||
},
|
||||
where() {
|
||||
return this;
|
||||
},
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { expect, it } from "vitest";
|
||||
import { issueRecoveryActions, issues } from "@paperclipai/db";
|
||||
import { subscribeDatabaseWork } from "@paperclipai/db/work-signals";
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { issueRecoveryActionService } from "../services/issue-recovery-actions.js";
|
||||
import { notifyIssueGraphRecovery } from "../services/issue-recovery-notifications.js";
|
||||
import { recoveryService } from "../services/recovery/service.js";
|
||||
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
|
||||
|
||||
const TASK_QUEUES = [DELIVERY_QUEUES.strandedAssignments, DELIVERY_QUEUES.assignedWorkInvariant,
|
||||
DELIVERY_QUEUES.resolvedDependencies, DELIVERY_QUEUES.staleIssueLocks];
|
||||
const WATCHED = [...TASK_QUEUES, DELIVERY_QUEUES.orphanBlockers, DELIVERY_QUEUES.activeRecoveryActions,
|
||||
DELIVERY_QUEUES.taskWatchdogs, DELIVERY_QUEUES.queuedRunDispatch, DELIVERY_QUEUES.executionDisposition];
|
||||
|
||||
describeIssueRecovery("issue recovery signal fan-out", () => {
|
||||
const state = issueRecoveryFixture();
|
||||
|
||||
function record(owner: object) {
|
||||
const settled: Array<[string, string[]]> = [];
|
||||
const stops = WATCHED.map(queue => subscribeDatabaseWork(owner, queue, (event, hints) => {
|
||||
if (event === "settled") settled.push([queue, [...hints ?? []]]);
|
||||
}));
|
||||
return { settled, stop: () => { for (const stop of stops) stop(); } };
|
||||
}
|
||||
|
||||
it("publishes one graph hint per task queue and no per-queue backstop twin", async () => {
|
||||
const f = await state.fixture();
|
||||
const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Task", status: "todo" }).returning();
|
||||
const recorder = record(state.db);
|
||||
try {
|
||||
await state.db.transaction(tx => notifyIssueGraphRecovery(tx, { companyId: f.companyId, issueId: task.id }));
|
||||
} finally { recorder.stop(); }
|
||||
const taskHints = recorder.settled.filter(([queue]) => TASK_QUEUES.includes(queue as never)).flatMap(([, hints]) => hints);
|
||||
expect(taskHints).toHaveLength(TASK_QUEUES.length);
|
||||
expect(taskHints.every(hint => JSON.parse(hint).graphIssueId === task.id)).toBe(true);
|
||||
expect(recorder.settled.flatMap(([, hints]) => hints).some(hint => hint.includes("\"backstop\""))).toBe(false);
|
||||
});
|
||||
|
||||
it("resolves nothing and signals nothing when the issue has no matching active recovery action", async () => {
|
||||
const f = await state.fixture();
|
||||
const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Done", status: "done" }).returning();
|
||||
const [resolved] = await state.db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: task.id,
|
||||
kind: "stranded_assigned_issue", status: "resolved", ownerType: "agent", ownerAgentId: f.agentId,
|
||||
cause: "stranded_assigned_issue", fingerprint: "already-resolved", nextAction: "Done", outcome: "restored",
|
||||
resolvedAt: new Date(), evidence: {} }).returning();
|
||||
const recorder = record(state.db);
|
||||
const actions = issueRecoveryActionService(state.db);
|
||||
try {
|
||||
expect(await actions.resolveActiveForIssue({ companyId: f.companyId, sourceIssueId: task.id, actionId: resolved.id,
|
||||
status: "resolved", outcome: "restored", resolutionNote: "source_terminal" })).toBeNull();
|
||||
expect(await actions.resolveActiveForIssue({ companyId: f.companyId, sourceIssueId: task.id,
|
||||
status: "cancelled", outcome: "cancelled", resolutionNote: "source_state_changed" })).toBeNull();
|
||||
} finally { recorder.stop(); }
|
||||
expect(recorder.settled).toEqual([]);
|
||||
});
|
||||
|
||||
it("still resolves an active action and wakes its task graph", async () => {
|
||||
const f = await state.fixture();
|
||||
const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Done", status: "done" }).returning();
|
||||
const [active] = await state.db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: task.id,
|
||||
kind: "stranded_assigned_issue", status: "escalated", ownerType: "agent", ownerAgentId: f.agentId,
|
||||
cause: "stranded_assigned_issue", fingerprint: "escalated", nextAction: "Restore", evidence: {} }).returning();
|
||||
const recorder = record(state.db);
|
||||
let result;
|
||||
try {
|
||||
result = await issueRecoveryActionService(state.db).resolveActiveForIssue({ companyId: f.companyId, sourceIssueId: task.id,
|
||||
actionId: active.id, status: "resolved", outcome: "restored", resolutionNote: "source_terminal" });
|
||||
} finally { recorder.stop(); }
|
||||
expect(result).toMatchObject({ id: active.id, status: "resolved", outcome: "restored" });
|
||||
expect(recorder.settled.filter(([queue]) => queue === DELIVERY_QUEUES.strandedAssignments).flatMap(([, hints]) => hints)
|
||||
.map(hint => JSON.parse(hint))).toContainEqual({ companyId: f.companyId, graphIssueId: task.id });
|
||||
});
|
||||
|
||||
async function blockedAttention(companyId: string, sourceIssueId: string, agentId: string, fingerprint = "native-controller-space:run:digest") {
|
||||
return issueRecoveryActionService(state.db).upsertSourceScoped({ companyId, sourceIssueId, kind: "active_run_watchdog",
|
||||
ownerType: "board", returnOwnerAgentId: agentId, cause: "native_restart_recovery_blocked", fingerprint,
|
||||
evidence: { runId: "run", reason: "controller_process_space_unverified", executorStopEvidence: "unverified" },
|
||||
nextAction: "Wait for the previous controller to stop" });
|
||||
}
|
||||
|
||||
it("does not re-open attention that recovery settled because its task finished", async () => {
|
||||
const f = await state.fixture();
|
||||
const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Cancelled", status: "cancelled" }).returning();
|
||||
const recovery = recoveryService(state.db, { enqueueWakeup: async () => null });
|
||||
await blockedAttention(f.companyId, task.id, f.agentId);
|
||||
await recovery.reconcileActiveRecoveryActions({ companyId: f.companyId });
|
||||
const recorder = record(state.db);
|
||||
try {
|
||||
for (let cycle = 0; cycle < 5; cycle++) {
|
||||
const surfaced = await blockedAttention(f.companyId, task.id, f.agentId);
|
||||
expect(surfaced.status).toBe("resolved");
|
||||
await recovery.reconcileActiveRecoveryActions({ companyId: f.companyId });
|
||||
}
|
||||
} finally { recorder.stop(); }
|
||||
const rows = await state.db.select().from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, f.companyId), eq(issueRecoveryActions.sourceIssueId, task.id)));
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]).toMatchObject({ status: "resolved", outcome: "restored", resolutionNote: "source_terminal" });
|
||||
expect(recorder.settled.filter(([queue]) => queue === DELIVERY_QUEUES.strandedAssignments
|
||||
|| queue === DELIVERY_QUEUES.activeRecoveryActions)).toEqual([]);
|
||||
});
|
||||
|
||||
it("settles a finished task's attention without waking run dispatch or its task graph", async () => {
|
||||
const f = await state.fixture();
|
||||
const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Done", status: "done" }).returning();
|
||||
await blockedAttention(f.companyId, task.id, f.agentId);
|
||||
const recorder = record(state.db);
|
||||
try {
|
||||
expect(await recoveryService(state.db, { enqueueWakeup: async () => null }).reconcileActiveRecoveryActions({ companyId: f.companyId }))
|
||||
.toMatchObject({ resolved: 1 });
|
||||
} finally { recorder.stop(); }
|
||||
expect(recorder.settled.filter(([queue]) => queue !== DELIVERY_QUEUES.executionDisposition)
|
||||
.filter(([queue]) => [...TASK_QUEUES, DELIVERY_QUEUES.queuedRunDispatch, DELIVERY_QUEUES.activeRecoveryActions].includes(queue as never)))
|
||||
.toEqual([]);
|
||||
});
|
||||
|
||||
it("surfaces attention again once the task is reopened, and for a new controller authority", async () => {
|
||||
const f = await state.fixture();
|
||||
const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Cancelled", status: "cancelled" }).returning();
|
||||
const recovery = recoveryService(state.db, { enqueueWakeup: async () => null });
|
||||
await blockedAttention(f.companyId, task.id, f.agentId);
|
||||
await recovery.reconcileActiveRecoveryActions({ companyId: f.companyId });
|
||||
expect((await blockedAttention(f.companyId, task.id, f.agentId, "native-controller-space:run:renewed")).status).toBe("active");
|
||||
await recovery.reconcileActiveRecoveryActions({ companyId: f.companyId });
|
||||
await state.db.update(issues).set({ status: "todo" }).where(eq(issues.id, task.id));
|
||||
const reopened = await blockedAttention(f.companyId, task.id, f.agentId);
|
||||
expect(reopened.status).toBe("active");
|
||||
await recovery.reconcileActiveRecoveryActions({ companyId: f.companyId });
|
||||
expect((await state.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, reopened.id)))[0].status)
|
||||
.toBe("active");
|
||||
});
|
||||
});
|
||||
@@ -6,7 +6,7 @@ import { writeExecutionControl } from "./execution-control-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import { and, desc, eq, inArray, isNull, isNotNull, or, sql } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, workspaceOperations } from "@paperclipai/db";
|
||||
import { heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, workspaceOperations } from "@paperclipai/db";
|
||||
import { isUuidLike } from "@paperclipai/shared";
|
||||
import type {
|
||||
IssueRecoveryAction,
|
||||
@@ -86,8 +86,11 @@ export type ResolveIssueRecoveryActionInput = {
|
||||
status: Extract<IssueRecoveryActionStatus, "resolved" | "cancelled">;
|
||||
outcome: IssueRecoveryActionOutcome;
|
||||
resolutionNote?: string | null;
|
||||
sourceTerminal?: boolean;
|
||||
};
|
||||
|
||||
export const SOURCE_TERMINAL_RESOLUTION = "source_terminal";
|
||||
|
||||
/** A spent saved-work recovery is described from its current state. Text saved
|
||||
* by an older release could ask a person to inspect the run's environment. */
|
||||
function readModelNextAction(row: IssueRecoveryActionRow): string {
|
||||
@@ -390,6 +393,21 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
const existing = await getActiveForIssue(input.companyId, input.sourceIssueId);
|
||||
const now = new Date();
|
||||
if (existing && input.createOnly) return existing;
|
||||
if (!existing) {
|
||||
const [source] = await db.select({ status: issues.status }).from(issues)
|
||||
.where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.sourceIssueId))).limit(1);
|
||||
if (source && (source.status === "done" || source.status === "cancelled")) {
|
||||
const [settled] = await db.select().from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, input.companyId),
|
||||
eq(issueRecoveryActions.sourceIssueId, input.sourceIssueId),
|
||||
eq(issueRecoveryActions.status, "resolved"),
|
||||
eq(issueRecoveryActions.cause, input.cause),
|
||||
eq(issueRecoveryActions.fingerprint, input.fingerprint),
|
||||
eq(issueRecoveryActions.resolutionNote, SOURCE_TERMINAL_RESOLUTION),
|
||||
)).limit(1);
|
||||
if (settled) return toReadModel(settled);
|
||||
}
|
||||
}
|
||||
const ownerType = input.ownerType ?? (input.ownerAgentId ? "agent" : "board");
|
||||
if (existing) {
|
||||
// A distinct failure identity must not overwrite the active action of a
|
||||
@@ -619,12 +637,11 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
predicates.push(eq(issueRecoveryActions.fingerprint, input.fingerprint));
|
||||
}
|
||||
|
||||
const [updated] = await writeExecutionControl(dbOrTx, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => {
|
||||
const pending = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions)
|
||||
.where(and(...predicates)).limit(1);
|
||||
if (!pending.length) return [];
|
||||
// Resolving another attention releases this issue's preserved Native
|
||||
// waiter to one normal inspection. No recurring monitor or new turn.
|
||||
const [candidate] = await dbOrTx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions)
|
||||
.where(and(...predicates)).limit(1);
|
||||
if (!candidate) return null;
|
||||
|
||||
const releaseWaiter = async (tx: DbOrTransaction) => {
|
||||
const [waiter] = await tx.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
|
||||
.innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)).where(and(
|
||||
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.sourceIssueId),
|
||||
@@ -632,6 +649,29 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
isNotNull(nativeRunFinalizations.leaseOwner), inArray(heartbeatRuns.status, ["failed", "timed_out", "interrupted"]),
|
||||
)).limit(1);
|
||||
if (waiter) await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization);
|
||||
};
|
||||
|
||||
if (input.sourceTerminal) {
|
||||
const [settled] = await dbOrTx.transaction(async tx => {
|
||||
await releaseWaiter(tx as unknown as DbOrTransaction);
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
return tx.update(issueRecoveryActions)
|
||||
.set({ status: input.status, outcome: input.outcome, resolutionNote: input.resolutionNote ?? null, resolvedAt: now, updatedAt: now })
|
||||
.where(and(...predicates, sql`exists (select 1 from ${issues} where ${issues.id} = ${issueRecoveryActions.sourceIssueId}
|
||||
and ${issues.companyId} = ${issueRecoveryActions.companyId} and ${issues.status} in ('done', 'cancelled'))`))
|
||||
.returning();
|
||||
});
|
||||
return settled ? toReadModel(settled) : null;
|
||||
}
|
||||
|
||||
const [updated] = await writeExecutionControl(dbOrTx, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => {
|
||||
const pending = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions)
|
||||
.where(and(...predicates)).limit(1);
|
||||
if (!pending.length) return [];
|
||||
// Resolving another attention releases this issue's preserved Native
|
||||
// waiter to one normal inspection. No recurring monitor or new turn.
|
||||
await releaseWaiter(tx as unknown as DbOrTransaction);
|
||||
await notifyIssueGraphRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
|
||||
@@ -26,6 +26,10 @@ export async function notifyIssueRecovery(transaction: object, scope?: IssueReco
|
||||
}
|
||||
}
|
||||
|
||||
export function issueRecoveryBackstopHint(companyId: string) {
|
||||
return JSON.stringify({ companyId, backstop: true });
|
||||
}
|
||||
|
||||
export function decodeIssueRecoveryHint(hint: string): IssueRecoveryScope | undefined {
|
||||
return hint === "*" ? undefined : JSON.parse(hint) as IssueRecoveryScope;
|
||||
}
|
||||
@@ -44,10 +48,7 @@ export async function notifyIssueGraphRecovery(transaction: object,
|
||||
const companyHint = JSON.stringify({ companyId: scope.companyId });
|
||||
for (const queue of COMPANY_QUEUES) await notifyDeliveryWork(transaction, queue, companyHint);
|
||||
const graphHint = JSON.stringify({ companyId: scope.companyId, graphRunId: scope.runId ?? undefined, graphIssueId: scope.issueId ?? undefined });
|
||||
for (const queue of TASK_QUEUES) {
|
||||
await notifyDeliveryWork(transaction, queue, graphHint);
|
||||
await notifyDeliveryWork(transaction, queue, JSON.stringify({ companyId: scope.companyId, backstop: true }));
|
||||
}
|
||||
for (const queue of TASK_QUEUES) await notifyDeliveryWork(transaction, queue, graphHint);
|
||||
}
|
||||
|
||||
export async function resolveIssueGraph(db: Db, scope: { companyId: string; runId?: string; issueId?: string }): Promise<string[] | null> {
|
||||
|
||||
@@ -127,7 +127,7 @@ import { emitAgentTaskRun } from "../agent-task-run-telemetry.js";
|
||||
import { budgetService } from "../budgets.js";
|
||||
import { isBoardOwnedBudgetBlock } from "../budget-block-ownership.js";
|
||||
import { unadmittedChatWakeupCondition } from "../durable-chat-wakeup.js";
|
||||
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
|
||||
import { issueRecoveryActionService, SOURCE_TERMINAL_RESOLUTION } from "../issue-recovery-actions.js";
|
||||
import { PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS } from "../system-recovery-state.js";
|
||||
import { parseProviderQuotaClockReset } from "../provider-quota-reset.js";
|
||||
import { hasUnverifiedNativeWorkspaceFinalizationOwner } from "../native-runtime/native-workspace-finalization-recovery-state.js";
|
||||
@@ -3670,7 +3670,8 @@ export function recoveryService(
|
||||
actionId: action.id,
|
||||
status: "resolved",
|
||||
outcome: "restored",
|
||||
resolutionNote: "source_terminal",
|
||||
resolutionNote: SOURCE_TERMINAL_RESOLUTION,
|
||||
sourceTerminal: true,
|
||||
});
|
||||
if (resolved) {
|
||||
result.resolved += 1;
|
||||
|
||||
@@ -3,7 +3,7 @@ import { and, asc, eq, gt, inArray, isNotNull, isNull, or, sql, type SQL } from
|
||||
import { issueRecoveryActions, issueRelations, issues, type Db } from "@paperclipai/db";
|
||||
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES, type DeliveryQueue } from "../delivery-work-notifications.js";
|
||||
import { decodeIssueRecoveryHint, ISSUE_RECOVERY_BACKSTOP_MS, issueRecoveryScopeMatches, resolveIssueGraph, type IssueRecoveryScope }
|
||||
import { decodeIssueRecoveryHint, ISSUE_RECOVERY_BACKSTOP_MS, issueRecoveryBackstopHint, issueRecoveryScopeMatches, resolveIssueGraph, type IssueRecoveryScope }
|
||||
from "../issue-recovery-notifications.js";
|
||||
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
|
||||
import { workSignalDebug } from "@paperclipai/db/work-signal-debug";
|
||||
@@ -71,6 +71,7 @@ export function registerIssueRecoveryWork(
|
||||
scope.wholeCompany = ids === null;
|
||||
}
|
||||
async function toCompany(scope: IssueRecoveryScope | undefined, worker: { invalidate(hint?: string): void }) {
|
||||
if (scope && isGraph(scope)) worker.invalidate(issueRecoveryBackstopHint(scope.companyId));
|
||||
await resolveGraph(scope);
|
||||
if (!scope?.wholeCompany) return false;
|
||||
worker.invalidate(JSON.stringify({ companyId: scope.companyId }));
|
||||
|
||||
Reference in new issue
Block a user