fix(server): schedule automatic completion review retirement (#17680)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native finalization preserves accepted work and resolves task
status.
> - A historical cleanup removes automatic completion-review cards that
current policy no longer creates.
> - The periodic finalization pass repeats that lookup even after all
matching cards are gone.
> - This PR gives this cleanup its own committed hints and pending-card
retries.
> - The existing owner retains withdrawal receipts and protects genuine
reviews.
> - Idle history stops causing recurring cleanup reads.

## Linked Issues or Issue Description

Refs #17679. This is a separate stacked slice for automatic
completion-review retirement. Obsolete version-change reviews belong to
the parent PR.

**What needs to be improved?**

`reconcileNativeFinalizations` repeatedly discovers old automatic
completion reviews. The discovery helper also converts SQL errors to an
empty result, which would let a new event-driven worker forget failed
reconstruction.

**Why is this improvement important?**

Idle tenants should not query for absent cleanup work. Failed discovery
must preserve the retry, and removing a historical card must preserve
the durable trigger for later evidence reassessment.

**Proposed solution**

Use the existing card, decision, effect, assessment and contract rows.
Reconstruct candidates at startup, signal the existing decision commit,
and retry exact pending cards. Keep the direct cleanup call inside
active finalization. ROADMAP.md and the related native review work were
inspected; this change owns the scheduling migration only.

## What Changed

- Add automatic completion-review registration to the shared scoped
deadline worker. Reconstruct 100-row keyset pages and dispatch fair
batches of twenty.
- Signal company/issue IDs before the native decision transaction
writes. Keep outer-transaction settlement with the existing work-signal
boundary.
- Split raw discovery from the public helper's compatibility error
handling. Worker lookup failures propagate so the shared scheduler
retains dirty scope and retry.
- Reuse the complete historical predicate and existing withdrawal owner.
Recheck the selected card identity, payload and system author fields in
the conditional update.
- Preserve the withdrawal reason, chat publication, activity record,
direct finalizer call, startup/manual recovery, and subsequent evidence
reassessment.
- Skip only this duty in periodic reaping. Retain admission,
drain/standby, retry and joined shutdown.
- Add producer/lifecycle documentation. Add no table, schema migration,
provider branch or independent timer.

## Verification

- Current source `3f4d35ac0154aedc76cc39486d3758c73b1e6ff4`: all 462
cases across seven complete files pass with no skips in 49.31 seconds.
This covers both review workers through the real status-authority
corpus, startup/timing, recovery, finalization, external chat and
governed milestones. Nested corpus fixtures are not double-counted.
- Five separate fresh PostgreSQL fixtures migrated through parent 0409
before setup. Database build, server typecheck/build, module boundaries
and target migration order pass. The whole database source tree matches
the parent; no migration or table is added.
- Normal integration preserves substantive owned edits and unrelated
upstream blobs. Actual landed-parent ancestry and fresh current-head
Greptile 5/5 and clean CI remain required. This is worker and
integration acceptance, not deployed sleep qualification.

- The inherited P2 fix changes only two fixture prefix values. All
production code, other fixtures, assertions and timing limits are
unchanged. The full 72-case corpus also passes again on the same
database in 12.95 seconds. Retained PSC/OUT rows and an unrelated
sentinel remain unchanged after both runs. The repeat is not
double-counted among the 462 distinct cases. The parent audit retains
the original collision and rejected cleanup attempt.

- CI correction: the 40f head fails the passive native-presentation
initial idle observation in chat shard 1. An unchanged local pair
passes. A controlled committed hint plus a 600 ms held run attempt makes
the original one-second observation fail; the identical probe passes
with the established five-second observation bound. This demonstrates
the coalescing boundary, not the unrecorded exact CI token timeline. No
unchanged-head CI retry was used.
- The probe is removed. The only permanent code change is that initial
poll bound. It still requires zero active work and no deadline, followed
by two minutes with no SQL, transaction or provider writes. Production
code, worker/business deadlines, rollback and committed-settlement
assertions are unchanged. All 274 cases in the named GitHub agent-review
workflow pass in 232.06 seconds. Other cases in the large file are
outside that named selection. The earlier 462-case selection retains
identical source. Both owning audits preserve the failure and scope
limits. Fresh updated-head CI and reviews remain required.

- Master integration: independent master
`49fe8b87a49aa5a8985a08dd3d5ce57bdd24756c` adds one company-skill
cleanup assertion. A normal merge preserves both complete binary
patches. All production code and the accepted GitHub integration file
are unchanged. The full six-case cleanup-removal suite passes in 8.16
seconds on fresh PostgreSQL. Database history is unchanged.

## Risks

- A normal native decision can cause one scoped lookup even when it
creates no historical card. It retains no periodic scan when settled.
- Startup reconstruction cost scales with historical data and joins. No
constant-cost or fleet-savings claim is made.
- Process-local signals cover instrumented writers. Independent SQL
writers and concurrent older controllers need explicit integration.
- Retirement and status reassessment are distinct obligations. The
existing durable withdrawal receipt and remaining finalization owner are
preserved.
- Failed withdrawals retain retries while exact pending cards remain. A
database lookup failure must not be interpreted as completion.
- Other finalization and recovery duties remain. Cloud wake coverage and
host-sleep eligibility are unchanged.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL
tests. The runtime does not expose a more precise model ID or
context-window size for this turn.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Remote acceptance: head `be668fb975f119efeb4830405d4bb1ab65791f87` has
53 successful gates, verified with `gh pr checks` (no non-passing
checks). Greptile reviewed that head at 5/5. There are no unresolved
review threads. The PR remains draft and unmerged.

Current published head `181bf800a79449961e09c995f2785dd7f8060bb3` has
passing CI, fresh Greptile 5/5 and Ponytail 5/5 with no actionable
findings. Earlier failures remain recorded in the audits.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-28 23:40:45 -05:00
1 parent 49fe8b87a4
commit 73d24aa5ac
15 files changed
+452 -39

No files matched your search

@@ -4114,7 +4114,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
try {
await githubResponseCommentService(db, f.providerFetch).registerTerminalResponseWork(coordinator, () => true).ready;
await expect.poll(() => ({ next: coordinator.nextWakeAt(), active: idleWorkSnapshot().active }))
await expect.poll(() => ({ next: coordinator.nextWakeAt(), active: idleWorkSnapshot().active }), { timeout: 5000 })
.toEqual({ next: null, active: 0 });
expect(f.responseWrites).toHaveLength(baseline);
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
@@ -1,6 +1,6 @@
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
import { dismissAutomaticCompletionReviews } from "../services/native-runtime/automatic-completion-reviews.js";
import { dismissAutomaticCompletionReviews, registerAutomaticCompletionReviewWork } from "../services/native-runtime/automatic-completion-reviews.js";
import * as childCompletionDelivery from "../services/native-runtime/native-child-completion-delivery.js";
import { nativeCompletionFeedback } from "../services/native-runtime/native-completion-feedback.js";
import { randomUUID } from "node:crypto";
@@ -2050,7 +2050,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
}
describe("obsolete policy review scheduling", () => {
describe.each(["obsolete policy", "automatic completion"] as const)("%s review scheduling", kind => {
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
let admitted = true;
const errors: unknown[] = [];
@@ -2061,13 +2061,29 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
afterEach(async () => {
await coordinator?.stop(); coordinator = undefined;
vi.useRealTimers(); vi.restoreAllMocks();
await dismissObsoleteNativePolicyReviews(db);
if (kind === "obsolete policy") await dismissObsoleteNativePolicyReviews(db);
else await dismissAutomaticCompletionReviews(db);
expect(errors).toEqual([]);
});
async function seedReview(options: Parameters<typeof seedPolicyReview>[0] = {}) {
if (kind === "obsolete policy" || options.genuine) return seedPolicyReview(options);
return seedPolicyReview({ ...options, commit: async input => {
const [assessment] = await db.select().from(workAssessments).where(eq(workAssessments.id, input.assessmentId));
await db.update(completionContracts).set({ risk: "low", completionAuthority: "agent_claim_policy" })
.where(eq(completionContracts.id, assessment!.contractId));
await db.update(workAssessments).set({ assessmentJson: { ...assessment!.assessmentJson, attentionRequests: [] } })
.where(eq(workAssessments.id, input.assessmentId));
return (options.commit ?? commitNativeStatusDecision)({ ...input, decision: { ...input.decision,
reasonCode: "completion_claim_incomplete", effects: [{ kind: "bind_reviewer", ownerUserId: null,
prompt: "Review the persisted native-run evidence and confirm whether this issue may be completed." }],
} });
} });
}
async function start() {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => admitted,
onError: error => errors.push(error) });
await registerObsoletePolicyReviewWork(db, coordinator, () => admitted).ready;
const register = kind === "obsolete policy" ? registerObsoletePolicyReviewWork : registerAutomaticCompletionReviewWork;
await register(db, coordinator, () => admitted).ready;
}
async function flush() {
await vi.waitFor(async () => {
@@ -2089,9 +2105,10 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
}
it("leaves genuine pending reviews and completed history quiet", async () => {
const genuine = await seedPolicyReview({ genuine: true });
const settled = await seedPolicyReview();
await dismissObsoleteNativePolicyReviews(db, [settled.runId]);
const genuine = await seedReview({ genuine: true });
const settled = await seedReview();
if (kind === "obsolete policy") await dismissObsoleteNativePolicyReviews(db, [settled.runId]);
else await dismissAutomaticCompletionReviews(db, settled.issueId);
await start(); await flush();
expect((await card(genuine.interaction.id)).status).toBe("pending");
expect((await card(settled.interaction.id)).status).toBe("cancelled");
@@ -2100,10 +2117,10 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
it("wakes from the real decision commit after becoming idle and preserves decision history", async () => {
await start(); await flush(); await quiet();
const created = await seedPolicyReview();
const created = await seedReview();
await flush();
expect((await card(created.interaction.id)).status).toBe("cancelled");
expect((await issueService(db).getById(created.issueId))!.status).toBe("in_progress");
expect((await issueService(db).getById(created.issueId))!.status).toBe(kind === "obsolete policy" ? "in_progress" : "in_review");
expect((await db.select().from(statusDecisions).where(eq(statusDecisions.id, created.decision.id)))[0]).toEqual(created.decision);
await quiet();
});
@@ -2112,7 +2129,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
await start(); await flush();
const entered = gate(), release = gate();
let issueId = "";
const creating = seedPolicyReview({ commit: async input => db.transaction(async tx => {
const creating = seedReview({ commit: async input => db.transaction(async tx => {
issueId = input.issueId;
const result = await commitNativeStatusDecision({ ...input, db: tx as unknown as typeof db });
entered.release(); await release.promise;
@@ -2137,7 +2154,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
it.each(["restart", "drain", "standby"] as const)("recovers committed cards after %s", async mode => {
if (mode !== "restart") { await start(); await flush(); admitted = false; }
const created = await seedPolicyReview();
const created = await seedReview();
if (mode === "restart") await start();
else {
await vi.advanceTimersByTimeAsync(1000);
@@ -2149,14 +2166,14 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
});
it("retains startup discovery while admission is closed", async () => {
const created = await seedPolicyReview(); admitted = false;
const created = await seedReview(); admitted = false;
await start(); await quiet(); expect((await card(created.interaction.id)).status).toBe("pending");
admitted = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
expect((await card(created.interaction.id)).status).toBe("cancelled"); await quiet();
});
it("retries only a failed cleanup while another candidate settles", async () => {
const first = await seedPolicyReview(), second = await seedPolicyReview();
const first = await seedReview(), second = await seedReview();
const write = vi.spyOn(db, "transaction").mockRejectedValueOnce(new Error("injected obsolete cleanup failure"));
await start();
await vi.waitFor(async () => {
@@ -2173,7 +2190,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
});
it("does not withdraw a review edited after discovery", async () => {
const created = await seedPolicyReview();
const created = await seedReview();
const original = db.transaction.bind(db);
const write = vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => {
await db.update(issueThreadInteractions).set({ payload: { ...created.interaction.payload, prompt: "Approve this deployment." } })
@@ -2186,7 +2203,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
});
it("joins admitted cleanup on shutdown and reconstructs quiet state on restart", async () => {
const created = await seedPolicyReview(), entered = gate(), release = gate();
const created = await seedReview(), entered = gate(), release = gate();
const original = db.transaction.bind(db);
const write = vi.spyOn(db, "transaction").mockImplementationOnce((callback, config) => original(async tx => {
entered.release(); await release.promise; return callback(tx);
@@ -2199,9 +2216,34 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
await start(); await flush(); await quiet();
});
it("retains discovery after a database error and retries on resume", async () => {
const created = await seedReview();
const read = vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("injected review discovery failure"); });
await start();
await vi.waitFor(() => expect(errors.length).toBeGreaterThan(0));
read.mockRestore(); expect(coordinator!.nextWakeAt()).not.toBeNull();
expect((await card(created.interaction.id)).status).toBe("pending");
errors.length = 0; admitted = false; await quiet(); admitted = true;
await vi.advanceTimersByTimeAsync(30_000); await flush();
expect((await card(created.interaction.id)).status).toBe("cancelled"); await quiet();
});
it("preserves a newly answered card between discovery and withdrawal", async () => {
const created = await seedReview();
const original = db.transaction.bind(db);
const write = vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => {
await db.update(issueThreadInteractions).set({ status: "accepted" }).where(eq(issueThreadInteractions.id, created.interaction.id));
return original(callback, config);
});
await start(); await flush(); write.mockRestore();
expect((await card(created.interaction.id)).status).toBe("accepted");
expect((await issueService(db).getById(created.issueId))!.status).toBe("in_review"); await quiet();
});
it("drains more than one reconstruction page without replaying settled reviews", async () => {
const created = [];
for (let index = 0; index < 103; index++) created.push(await seedPolicyReview());
for (let index = 0; index < 103; index++) created.push(await seedReview());
await start(); await flush();
for (const value of created) expect((await card(value.interaction.id)).status).toBe("cancelled");
await quiet();
@@ -116,6 +116,7 @@ const {
reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })),
registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerAutomaticCompletionReviewWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerObsoletePolicyReviewWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerCancelledWorkspacePreservationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerInstructionDirectoryCleanupWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
@@ -555,6 +556,13 @@ describe("startServer feedback export wiring", () => {
process.env.BETTER_AUTH_SECRET = "test-secret";
});
it.each([false, true])("registers automatic completion review cleanup only when maintenance is enabled (%s)", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
await startServer(); await settleStartupHousekeeping();
expect(heartbeatServiceMock.registerAutomaticCompletionReviewWork).toHaveBeenCalledTimes(enabled ? 1 : 0);
if (enabled) expect(heartbeatServiceMock.registerAutomaticCompletionReviewWork.mock.calls[0][1]()).toBe(true);
});
it.each([false, true])("registers obsolete policy review cleanup only when maintenance is enabled (%s)", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
await startServer(); await settleStartupHousekeeping();
@@ -981,7 +989,7 @@ describe("startServer feedback export wiring", () => {
intervalCallback?.();
await new Promise<void>((resolve) => setImmediate(resolve));
expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2);
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true });
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true });
expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1);
expect(retiredDetector).not.toHaveBeenCalled();
} finally {
+3 -1
View File
@@ -1824,6 +1824,8 @@ async function startServerWithDatabaseTeardown(
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerAutomaticCompletionReviewWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerObsoletePolicyReviewWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerCancelledWorkspacePreservationWork(deliveryWork,
@@ -1958,7 +1960,7 @@ async function startServerWithDatabaseTeardown(
// Periodically reap orphaned runs (5-min staleness threshold) and make sure
// persisted queued work is still being driven forward.
runPeriodicRecovery(() => heartbeat
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true })
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true })
.then(() => heartbeat.promoteDueScheduledRetries())
.then(async (promotion) => {
await heartbeat.resumeQueuedRuns();
@@ -0,0 +1,8 @@
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export type AutomaticCompletionReviewScope = { companyId: string; issueId: string };
export async function notifyAutomaticCompletionReviews(transaction: object, scope: AutomaticCompletionReviewScope) {
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.automaticCompletionReviews,
JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }));
}
@@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
/** Topics for existing durable queues; register intent before their writes. */
export const DELIVERY_QUEUES = {
automaticCompletionReviews: "automatic-completion-reviews",
obsoletePolicyReviews: "obsolete-policy-reviews",
cancelledWorkspacePreservation: "cancelled-workspace-preservation",
instructionCollection: "instruction-collection",
+3
View File
@@ -1,3 +1,4 @@
import { registerAutomaticCompletionReviewWork } from "./native-runtime/automatic-completion-reviews.js";
import { registerObsoletePolicyReviewWork } from "./native-runtime/obsolete-policy-reviews.js";
import { registerCancelledWorkspacePreservationWork } from "./native-runtime/native-failed-workspace-recovery.js";
import { readUpgradeReadiness } from "./heartbeat/upgrade-readiness.js";
@@ -4086,6 +4087,8 @@ export function heartbeatService(
triggerIssueMonitor,
reportRunActivity: clearDetachedRunWarning,
registerAutomaticCompletionReviewWork: (coordinator: Parameters<typeof registerAutomaticCompletionReviewWork>[1], canRun: () => boolean) =>
registerAutomaticCompletionReviewWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
registerObsoletePolicyReviewWork: (coordinator: Parameters<typeof registerObsoletePolicyReviewWork>[1], canRun: () => boolean) =>
registerObsoletePolicyReviewWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
registerCancelledWorkspacePreservationWork: (
+2 -1
View File
@@ -1947,7 +1947,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
let settledProcessesDiscovered = false;
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean }) {
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean }) {
if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db);
const staleThresholdMs = opts?.staleThresholdMs ?? 0;
const now = new Date();
@@ -1966,6 +1966,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
// mode, never the current feature flag.
await reconcileNativeFinalizations(db, undefined, {
skipObsoletePolicyReviews: opts?.skipObsoletePolicyReviews,
skipAutomaticCompletionReviews: opts?.skipAutomaticCompletionReviews,
environmentRuntime,
pluginWorkerManager: dependencies.pluginWorkerManager,
onWorkspaceSettled: settleRecoveredNativeWorkspace,
@@ -1,4 +1,8 @@
import { and, eq, inArray, isNull, sql } from "drizzle-orm";
import type { AutomaticCompletionReviewScope } from "../automatic-completion-review-notifications.js";
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
import { and, asc, eq, gt, inArray, isNull, sql } from "drizzle-orm";
import {
completionContracts,
issueThreadInteractions,
@@ -24,7 +28,7 @@ const automaticPrompt =
"Review the persisted native-run evidence and confirm whether this issue may be completed.";
/** Identify only proven system fallback cards; this lookup never changes state. */
export async function findAutomaticCompletionReviews(db: Db, issueId?: string) {
function findAutomaticReviewCandidates(db: Db, options: { issueId?: string; companyId?: string; id?: string; after?: string } = {}) {
return db
.select({ interaction: issueThreadInteractions, decision: statusDecisions })
.from(issueThreadInteractions)
@@ -83,17 +87,20 @@ export async function findAutomaticCompletionReviews(db: Db, issueId?: string) {
sql`${issueThreadInteractions.payload}->'target'->>'key' = 'native_completion_review'`,
sql`${issueThreadInteractions.payload}->'target'->>'revisionId' = ${statusDecisions.id}::text`,
sql`split_part(${issueThreadInteractions.payload}->>'prompt', E'\n', 1) = ${automaticPrompt}`,
...(issueId ? [eq(issueThreadInteractions.issueId, issueId)] : []),
options.issueId ? eq(issueThreadInteractions.issueId, options.issueId) : undefined,
options.companyId ? eq(issueThreadInteractions.companyId, options.companyId) : undefined,
options.id ? eq(issueThreadInteractions.id, options.id) : undefined,
options.after ? gt(issueThreadInteractions.id, options.after) : undefined,
),
)
.limit(100)
.catch((err) => {
logger.warn(
{ err },
"Automatic completion review lookup failed; will retry",
);
return [];
});
.orderBy(asc(issueThreadInteractions.id)).limit(100);
}
export async function findAutomaticCompletionReviews(db: Db, issueId?: string) {
return findAutomaticReviewCandidates(db, { issueId }).catch((err) => {
logger.warn({ err }, "Automatic completion review lookup failed; will retry");
return [];
});
}
/** Narrow, replay-safe retirement. Explicit requests and answered cards are immutable here. */
@@ -101,7 +108,10 @@ export async function dismissAutomaticCompletionReviews(
db: Db,
issueId?: string,
) {
const candidates = await findAutomaticCompletionReviews(db, issueId);
await dismissCandidates(db, await findAutomaticCompletionReviews(db, issueId));
}
async function dismissCandidates(db: Db, candidates: Awaited<ReturnType<typeof findAutomaticReviewCandidates>>) {
for (const { interaction, decision } of candidates) {
const publications: ActivityPublication[] = [];
try {
@@ -147,6 +157,10 @@ export async function dismissAutomaticCompletionReviews(
eq(issueThreadInteractions.companyId, issue.companyId),
eq(issueThreadInteractions.status, "pending"),
eq(issueThreadInteractions.payload, interaction.payload),
eq(issueThreadInteractions.kind, interaction.kind),
eq(issueThreadInteractions.sourceRunId, decision.runId),
eq(issueThreadInteractions.idempotencyKey, interaction.idempotencyKey!),
isNull(issueThreadInteractions.createdByAgentId), isNull(issueThreadInteractions.createdByUserId),
),
)
.returning();
@@ -213,3 +227,33 @@ export async function decisionHasRetiredAutomaticReview(
.limit(1);
return rows.length > 0;
}
export function registerAutomaticCompletionReviewWork(db: Db, coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
type Entry = { id: string; companyId: string; issueId: string; dueAt: number };
async function load(scope?: AutomaticCompletionReviewScope, signal?: AbortSignal): Promise<Entry[]> {
const entries = new Map<string, Entry>();
let after: string | undefined;
while (canRun() && !signal?.aborted) {
const rows = await findAutomaticReviewCandidates(db, { ...scope, after });
for (const { interaction } of rows) entries.set(interaction.id, {
id: interaction.id, companyId: interaction.companyId, issueId: interaction.issueId, dueAt: 0,
});
if (rows.length < 100) return [...entries.values()];
after = rows[rows.length - 1].interaction.id;
}
throw new Error("Automatic completion review discovery stopped");
}
return registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.automaticCompletionReviews, batchSize: 20,
decodeScope: (hint: string): AutomaticCompletionReviewScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (entry: Entry, scope?: AutomaticCompletionReviewScope) => !scope || entry.companyId === scope.companyId && entry.issueId === scope.issueId,
load,
check: async (entry: Entry, signal?: AbortSignal) => {
const candidates = await findAutomaticReviewCandidates(db, entry);
if (!canRun() || signal?.aborted) throw new Error("Automatic completion review cleanup stopped");
await dismissCandidates(db, candidates);
const pending = await findAutomaticReviewCandidates(db, entry);
return pending.length ? { ...entry, dueAt: Date.now() + 30_000 } : null;
},
}, canRun);
}
@@ -32,6 +32,14 @@ describe("native finalization reconciliation timing", () => {
timing.phase.mockClear(); timing.finish.mockClear();
});
it("omits automatic review discovery when its worker owns that duty", async () => {
timing.phase.mockClear();
const db = { select: vi.fn(() => { throw new Error("candidate scan reached"); }) };
await expect(reconcileNativeFinalizations(db as never, ["run"], { skipAutomaticCompletionReviews: true })).rejects.toThrow("candidate scan reached");
expect(timing.phase.mock.calls.map(([phase]) => phase)).toEqual(["legacy_workspace_exports", "obsolete_policy_reviews", "candidate_scan"]);
timing.phase.mockClear(); timing.finish.mockClear();
});
it("records the phases and a failed outcome when a pass fails, and rethrows", async () => {
await expect(reconcileNativeFinalizations({} as never)).rejects.toThrow("review lookup failed");
@@ -638,6 +638,7 @@ export async function claimNativeSessionResumptions(input: {
type NativeReconciliationOptions = {
skipObsoletePolicyReviews?: boolean;
skipAutomaticCompletionReviews?: boolean;
pluginWorkerManager?: WorkspaceWorkers;
environmentRuntime?: EnvironmentRuntimeService;
onWorkspaceSettled?: (input: {
@@ -730,13 +731,15 @@ async function reconcileNativeFinalizationsPass(
logger.warn({ err }, "Obsolete native policy review lookup failed; continuing native reconciliation");
});
}
timing.phase("automatic_completion_reviews");
if (runIds?.length) {
const scopes = await db.select({ issueId: nativeRunFinalizations.issueId }).from(nativeRunFinalizations)
.where(inArray(nativeRunFinalizations.runId, runIds));
for (const scope of scopes) await dismissAutomaticCompletionReviews(db, scope.issueId);
} else {
await dismissAutomaticCompletionReviews(db);
if (!options.skipAutomaticCompletionReviews) {
timing.phase("automatic_completion_reviews");
if (runIds?.length) {
const scopes = await db.select({ issueId: nativeRunFinalizations.issueId }).from(nativeRunFinalizations)
.where(inArray(nativeRunFinalizations.runId, runIds));
for (const scope of scopes) await dismissAutomaticCompletionReviews(db, scope.issueId);
} else {
await dismissAutomaticCompletionReviews(db);
}
}
timing.phase("candidate_scan");
const now = new Date();
@@ -1,3 +1,4 @@
import { notifyAutomaticCompletionReviews } from "../automatic-completion-review-notifications.js";
import { notifyObsoletePolicyReviews } from "../obsolete-policy-review-notifications.js";
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
@@ -1983,6 +1984,7 @@ export async function commitNativeStatusDecision(input: {
}
await notifyChatRunMilestones(tx, input);
await notifyObsoletePolicyReviews(tx, input);
await notifyAutomaticCompletionReviews(tx, input);
if (!decisionRow) {
const latestDecisionVersion = await tx
.select({