mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 13:20:54 +01:00
fix(server): schedule automatic completion review retirement (#17680)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native finalization preserves accepted work and resolves task status. > - A historical cleanup removes automatic completion-review cards that current policy no longer creates. > - The periodic finalization pass repeats that lookup even after all matching cards are gone. > - This PR gives this cleanup its own committed hints and pending-card retries. > - The existing owner retains withdrawal receipts and protects genuine reviews. > - Idle history stops causing recurring cleanup reads. ## Linked Issues or Issue Description Refs #17679. This is a separate stacked slice for automatic completion-review retirement. Obsolete version-change reviews belong to the parent PR. **What needs to be improved?** `reconcileNativeFinalizations` repeatedly discovers old automatic completion reviews. The discovery helper also converts SQL errors to an empty result, which would let a new event-driven worker forget failed reconstruction. **Why is this improvement important?** Idle tenants should not query for absent cleanup work. Failed discovery must preserve the retry, and removing a historical card must preserve the durable trigger for later evidence reassessment. **Proposed solution** Use the existing card, decision, effect, assessment and contract rows. Reconstruct candidates at startup, signal the existing decision commit, and retry exact pending cards. Keep the direct cleanup call inside active finalization. ROADMAP.md and the related native review work were inspected; this change owns the scheduling migration only. ## What Changed - Add automatic completion-review registration to the shared scoped deadline worker. Reconstruct 100-row keyset pages and dispatch fair batches of twenty. - Signal company/issue IDs before the native decision transaction writes. Keep outer-transaction settlement with the existing work-signal boundary. - Split raw discovery from the public helper's compatibility error handling. Worker lookup failures propagate so the shared scheduler retains dirty scope and retry. - Reuse the complete historical predicate and existing withdrawal owner. Recheck the selected card identity, payload and system author fields in the conditional update. - Preserve the withdrawal reason, chat publication, activity record, direct finalizer call, startup/manual recovery, and subsequent evidence reassessment. - Skip only this duty in periodic reaping. Retain admission, drain/standby, retry and joined shutdown. - Add producer/lifecycle documentation. Add no table, schema migration, provider branch or independent timer. ## Verification - Current source `3f4d35ac0154aedc76cc39486d3758c73b1e6ff4`: all 462 cases across seven complete files pass with no skips in 49.31 seconds. This covers both review workers through the real status-authority corpus, startup/timing, recovery, finalization, external chat and governed milestones. Nested corpus fixtures are not double-counted. - Five separate fresh PostgreSQL fixtures migrated through parent 0409 before setup. Database build, server typecheck/build, module boundaries and target migration order pass. The whole database source tree matches the parent; no migration or table is added. - Normal integration preserves substantive owned edits and unrelated upstream blobs. Actual landed-parent ancestry and fresh current-head Greptile 5/5 and clean CI remain required. This is worker and integration acceptance, not deployed sleep qualification. - The inherited P2 fix changes only two fixture prefix values. All production code, other fixtures, assertions and timing limits are unchanged. The full 72-case corpus also passes again on the same database in 12.95 seconds. Retained PSC/OUT rows and an unrelated sentinel remain unchanged after both runs. The repeat is not double-counted among the 462 distinct cases. The parent audit retains the original collision and rejected cleanup attempt. - CI correction: the 40f head fails the passive native-presentation initial idle observation in chat shard 1. An unchanged local pair passes. A controlled committed hint plus a 600 ms held run attempt makes the original one-second observation fail; the identical probe passes with the established five-second observation bound. This demonstrates the coalescing boundary, not the unrecorded exact CI token timeline. No unchanged-head CI retry was used. - The probe is removed. The only permanent code change is that initial poll bound. It still requires zero active work and no deadline, followed by two minutes with no SQL, transaction or provider writes. Production code, worker/business deadlines, rollback and committed-settlement assertions are unchanged. All 274 cases in the named GitHub agent-review workflow pass in 232.06 seconds. Other cases in the large file are outside that named selection. The earlier 462-case selection retains identical source. Both owning audits preserve the failure and scope limits. Fresh updated-head CI and reviews remain required. - Master integration: independent master `49fe8b87a49aa5a8985a08dd3d5ce57bdd24756c` adds one company-skill cleanup assertion. A normal merge preserves both complete binary patches. All production code and the accepted GitHub integration file are unchanged. The full six-case cleanup-removal suite passes in 8.16 seconds on fresh PostgreSQL. Database history is unchanged. ## Risks - A normal native decision can cause one scoped lookup even when it creates no historical card. It retains no periodic scan when settled. - Startup reconstruction cost scales with historical data and joins. No constant-cost or fleet-savings claim is made. - Process-local signals cover instrumented writers. Independent SQL writers and concurrent older controllers need explicit integration. - Retirement and status reassessment are distinct obligations. The existing durable withdrawal receipt and remaining finalization owner are preserved. - Failed withdrawals retain retries while exact pending cards remain. A database lookup failure must not be interpreted as completion. - Other finalization and recovery duties remain. Cloud wake coverage and host-sleep eligibility are unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Remote acceptance: head `be668fb975f119efeb4830405d4bb1ab65791f87` has 53 successful gates, verified with `gh pr checks` (no non-passing checks). Greptile reviewed that head at 5/5. There are no unresolved review threads. The PR remains draft and unmerged. Current published head `181bf800a79449961e09c995f2785dd7f8060bb3` has passing CI, fresh Greptile 5/5 and Ponytail 5/5 with no actionable findings. Earlier failures remain recorded in the audits. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
49fe8b87a4
commit
73d24aa5ac
15 files changed
+452
-39
No files matched your search
@@ -4114,7 +4114,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
try {
|
||||
await githubResponseCommentService(db, f.providerFetch).registerTerminalResponseWork(coordinator, () => true).ready;
|
||||
await expect.poll(() => ({ next: coordinator.nextWakeAt(), active: idleWorkSnapshot().active }))
|
||||
await expect.poll(() => ({ next: coordinator.nextWakeAt(), active: idleWorkSnapshot().active }), { timeout: 5000 })
|
||||
.toEqual({ next: null, active: 0 });
|
||||
expect(f.responseWrites).toHaveLength(baseline);
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { dismissAutomaticCompletionReviews } from "../services/native-runtime/automatic-completion-reviews.js";
|
||||
import { dismissAutomaticCompletionReviews, registerAutomaticCompletionReviewWork } from "../services/native-runtime/automatic-completion-reviews.js";
|
||||
import * as childCompletionDelivery from "../services/native-runtime/native-child-completion-delivery.js";
|
||||
import { nativeCompletionFeedback } from "../services/native-runtime/native-completion-feedback.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
@@ -2050,7 +2050,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
}
|
||||
|
||||
|
||||
describe("obsolete policy review scheduling", () => {
|
||||
describe.each(["obsolete policy", "automatic completion"] as const)("%s review scheduling", kind => {
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
let admitted = true;
|
||||
const errors: unknown[] = [];
|
||||
@@ -2061,13 +2061,29 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
afterEach(async () => {
|
||||
await coordinator?.stop(); coordinator = undefined;
|
||||
vi.useRealTimers(); vi.restoreAllMocks();
|
||||
await dismissObsoleteNativePolicyReviews(db);
|
||||
if (kind === "obsolete policy") await dismissObsoleteNativePolicyReviews(db);
|
||||
else await dismissAutomaticCompletionReviews(db);
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
async function seedReview(options: Parameters<typeof seedPolicyReview>[0] = {}) {
|
||||
if (kind === "obsolete policy" || options.genuine) return seedPolicyReview(options);
|
||||
return seedPolicyReview({ ...options, commit: async input => {
|
||||
const [assessment] = await db.select().from(workAssessments).where(eq(workAssessments.id, input.assessmentId));
|
||||
await db.update(completionContracts).set({ risk: "low", completionAuthority: "agent_claim_policy" })
|
||||
.where(eq(completionContracts.id, assessment!.contractId));
|
||||
await db.update(workAssessments).set({ assessmentJson: { ...assessment!.assessmentJson, attentionRequests: [] } })
|
||||
.where(eq(workAssessments.id, input.assessmentId));
|
||||
return (options.commit ?? commitNativeStatusDecision)({ ...input, decision: { ...input.decision,
|
||||
reasonCode: "completion_claim_incomplete", effects: [{ kind: "bind_reviewer", ownerUserId: null,
|
||||
prompt: "Review the persisted native-run evidence and confirm whether this issue may be completed." }],
|
||||
} });
|
||||
} });
|
||||
}
|
||||
async function start() {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => admitted,
|
||||
onError: error => errors.push(error) });
|
||||
await registerObsoletePolicyReviewWork(db, coordinator, () => admitted).ready;
|
||||
const register = kind === "obsolete policy" ? registerObsoletePolicyReviewWork : registerAutomaticCompletionReviewWork;
|
||||
await register(db, coordinator, () => admitted).ready;
|
||||
}
|
||||
async function flush() {
|
||||
await vi.waitFor(async () => {
|
||||
@@ -2089,9 +2105,10 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
}
|
||||
|
||||
it("leaves genuine pending reviews and completed history quiet", async () => {
|
||||
const genuine = await seedPolicyReview({ genuine: true });
|
||||
const settled = await seedPolicyReview();
|
||||
await dismissObsoleteNativePolicyReviews(db, [settled.runId]);
|
||||
const genuine = await seedReview({ genuine: true });
|
||||
const settled = await seedReview();
|
||||
if (kind === "obsolete policy") await dismissObsoleteNativePolicyReviews(db, [settled.runId]);
|
||||
else await dismissAutomaticCompletionReviews(db, settled.issueId);
|
||||
await start(); await flush();
|
||||
expect((await card(genuine.interaction.id)).status).toBe("pending");
|
||||
expect((await card(settled.interaction.id)).status).toBe("cancelled");
|
||||
@@ -2100,10 +2117,10 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
|
||||
it("wakes from the real decision commit after becoming idle and preserves decision history", async () => {
|
||||
await start(); await flush(); await quiet();
|
||||
const created = await seedPolicyReview();
|
||||
const created = await seedReview();
|
||||
await flush();
|
||||
expect((await card(created.interaction.id)).status).toBe("cancelled");
|
||||
expect((await issueService(db).getById(created.issueId))!.status).toBe("in_progress");
|
||||
expect((await issueService(db).getById(created.issueId))!.status).toBe(kind === "obsolete policy" ? "in_progress" : "in_review");
|
||||
expect((await db.select().from(statusDecisions).where(eq(statusDecisions.id, created.decision.id)))[0]).toEqual(created.decision);
|
||||
await quiet();
|
||||
});
|
||||
@@ -2112,7 +2129,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
await start(); await flush();
|
||||
const entered = gate(), release = gate();
|
||||
let issueId = "";
|
||||
const creating = seedPolicyReview({ commit: async input => db.transaction(async tx => {
|
||||
const creating = seedReview({ commit: async input => db.transaction(async tx => {
|
||||
issueId = input.issueId;
|
||||
const result = await commitNativeStatusDecision({ ...input, db: tx as unknown as typeof db });
|
||||
entered.release(); await release.promise;
|
||||
@@ -2137,7 +2154,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
|
||||
it.each(["restart", "drain", "standby"] as const)("recovers committed cards after %s", async mode => {
|
||||
if (mode !== "restart") { await start(); await flush(); admitted = false; }
|
||||
const created = await seedPolicyReview();
|
||||
const created = await seedReview();
|
||||
if (mode === "restart") await start();
|
||||
else {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
@@ -2149,14 +2166,14 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
});
|
||||
|
||||
it("retains startup discovery while admission is closed", async () => {
|
||||
const created = await seedPolicyReview(); admitted = false;
|
||||
const created = await seedReview(); admitted = false;
|
||||
await start(); await quiet(); expect((await card(created.interaction.id)).status).toBe("pending");
|
||||
admitted = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
|
||||
expect((await card(created.interaction.id)).status).toBe("cancelled"); await quiet();
|
||||
});
|
||||
|
||||
it("retries only a failed cleanup while another candidate settles", async () => {
|
||||
const first = await seedPolicyReview(), second = await seedPolicyReview();
|
||||
const first = await seedReview(), second = await seedReview();
|
||||
const write = vi.spyOn(db, "transaction").mockRejectedValueOnce(new Error("injected obsolete cleanup failure"));
|
||||
await start();
|
||||
await vi.waitFor(async () => {
|
||||
@@ -2173,7 +2190,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
});
|
||||
|
||||
it("does not withdraw a review edited after discovery", async () => {
|
||||
const created = await seedPolicyReview();
|
||||
const created = await seedReview();
|
||||
const original = db.transaction.bind(db);
|
||||
const write = vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => {
|
||||
await db.update(issueThreadInteractions).set({ payload: { ...created.interaction.payload, prompt: "Approve this deployment." } })
|
||||
@@ -2186,7 +2203,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
});
|
||||
|
||||
it("joins admitted cleanup on shutdown and reconstructs quiet state on restart", async () => {
|
||||
const created = await seedPolicyReview(), entered = gate(), release = gate();
|
||||
const created = await seedReview(), entered = gate(), release = gate();
|
||||
const original = db.transaction.bind(db);
|
||||
const write = vi.spyOn(db, "transaction").mockImplementationOnce((callback, config) => original(async tx => {
|
||||
entered.release(); await release.promise; return callback(tx);
|
||||
@@ -2199,9 +2216,34 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
await start(); await flush(); await quiet();
|
||||
});
|
||||
|
||||
|
||||
it("retains discovery after a database error and retries on resume", async () => {
|
||||
const created = await seedReview();
|
||||
const read = vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("injected review discovery failure"); });
|
||||
await start();
|
||||
await vi.waitFor(() => expect(errors.length).toBeGreaterThan(0));
|
||||
read.mockRestore(); expect(coordinator!.nextWakeAt()).not.toBeNull();
|
||||
expect((await card(created.interaction.id)).status).toBe("pending");
|
||||
errors.length = 0; admitted = false; await quiet(); admitted = true;
|
||||
await vi.advanceTimersByTimeAsync(30_000); await flush();
|
||||
expect((await card(created.interaction.id)).status).toBe("cancelled"); await quiet();
|
||||
});
|
||||
|
||||
it("preserves a newly answered card between discovery and withdrawal", async () => {
|
||||
const created = await seedReview();
|
||||
const original = db.transaction.bind(db);
|
||||
const write = vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => {
|
||||
await db.update(issueThreadInteractions).set({ status: "accepted" }).where(eq(issueThreadInteractions.id, created.interaction.id));
|
||||
return original(callback, config);
|
||||
});
|
||||
await start(); await flush(); write.mockRestore();
|
||||
expect((await card(created.interaction.id)).status).toBe("accepted");
|
||||
expect((await issueService(db).getById(created.issueId))!.status).toBe("in_review"); await quiet();
|
||||
});
|
||||
|
||||
it("drains more than one reconstruction page without replaying settled reviews", async () => {
|
||||
const created = [];
|
||||
for (let index = 0; index < 103; index++) created.push(await seedPolicyReview());
|
||||
for (let index = 0; index < 103; index++) created.push(await seedReview());
|
||||
await start(); await flush();
|
||||
for (const value of created) expect((await card(value.interaction.id)).status).toBe("cancelled");
|
||||
await quiet();
|
||||
|
||||
@@ -116,6 +116,7 @@ const {
|
||||
reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })),
|
||||
registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerAutomaticCompletionReviewWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerObsoletePolicyReviewWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerCancelledWorkspacePreservationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerInstructionDirectoryCleanupWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
@@ -555,6 +556,13 @@ describe("startServer feedback export wiring", () => {
|
||||
process.env.BETTER_AUTH_SECRET = "test-secret";
|
||||
});
|
||||
|
||||
it.each([false, true])("registers automatic completion review cleanup only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
expect(heartbeatServiceMock.registerAutomaticCompletionReviewWork).toHaveBeenCalledTimes(enabled ? 1 : 0);
|
||||
if (enabled) expect(heartbeatServiceMock.registerAutomaticCompletionReviewWork.mock.calls[0][1]()).toBe(true);
|
||||
});
|
||||
|
||||
it.each([false, true])("registers obsolete policy review cleanup only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
@@ -981,7 +989,7 @@ describe("startServer feedback export wiring", () => {
|
||||
intervalCallback?.();
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2);
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true });
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true });
|
||||
expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1);
|
||||
expect(retiredDetector).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
|
||||
+3
-1
@@ -1824,6 +1824,8 @@ async function startServerWithDatabaseTeardown(
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerAutomaticCompletionReviewWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerObsoletePolicyReviewWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerCancelledWorkspacePreservationWork(deliveryWork,
|
||||
@@ -1958,7 +1960,7 @@ async function startServerWithDatabaseTeardown(
|
||||
// Periodically reap orphaned runs (5-min staleness threshold) and make sure
|
||||
// persisted queued work is still being driven forward.
|
||||
runPeriodicRecovery(() => heartbeat
|
||||
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true })
|
||||
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true })
|
||||
.then(() => heartbeat.promoteDueScheduledRetries())
|
||||
.then(async (promotion) => {
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type AutomaticCompletionReviewScope = { companyId: string; issueId: string };
|
||||
|
||||
export async function notifyAutomaticCompletionReviews(transaction: object, scope: AutomaticCompletionReviewScope) {
|
||||
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.automaticCompletionReviews,
|
||||
JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }));
|
||||
}
|
||||
@@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
|
||||
/** Topics for existing durable queues; register intent before their writes. */
|
||||
export const DELIVERY_QUEUES = {
|
||||
automaticCompletionReviews: "automatic-completion-reviews",
|
||||
obsoletePolicyReviews: "obsolete-policy-reviews",
|
||||
cancelledWorkspacePreservation: "cancelled-workspace-preservation",
|
||||
instructionCollection: "instruction-collection",
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { registerAutomaticCompletionReviewWork } from "./native-runtime/automatic-completion-reviews.js";
|
||||
import { registerObsoletePolicyReviewWork } from "./native-runtime/obsolete-policy-reviews.js";
|
||||
import { registerCancelledWorkspacePreservationWork } from "./native-runtime/native-failed-workspace-recovery.js";
|
||||
import { readUpgradeReadiness } from "./heartbeat/upgrade-readiness.js";
|
||||
@@ -4086,6 +4087,8 @@ export function heartbeatService(
|
||||
triggerIssueMonitor,
|
||||
|
||||
reportRunActivity: clearDetachedRunWarning,
|
||||
registerAutomaticCompletionReviewWork: (coordinator: Parameters<typeof registerAutomaticCompletionReviewWork>[1], canRun: () => boolean) =>
|
||||
registerAutomaticCompletionReviewWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
|
||||
registerObsoletePolicyReviewWork: (coordinator: Parameters<typeof registerObsoletePolicyReviewWork>[1], canRun: () => boolean) =>
|
||||
registerObsoletePolicyReviewWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
|
||||
registerCancelledWorkspacePreservationWork: (
|
||||
|
||||
@@ -1947,7 +1947,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
|
||||
let settledProcessesDiscovered = false;
|
||||
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean }) {
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean }) {
|
||||
if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db);
|
||||
const staleThresholdMs = opts?.staleThresholdMs ?? 0;
|
||||
const now = new Date();
|
||||
@@ -1966,6 +1966,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
// mode, never the current feature flag.
|
||||
await reconcileNativeFinalizations(db, undefined, {
|
||||
skipObsoletePolicyReviews: opts?.skipObsoletePolicyReviews,
|
||||
skipAutomaticCompletionReviews: opts?.skipAutomaticCompletionReviews,
|
||||
environmentRuntime,
|
||||
pluginWorkerManager: dependencies.pluginWorkerManager,
|
||||
onWorkspaceSettled: settleRecoveredNativeWorkspace,
|
||||
|
||||
@@ -1,4 +1,8 @@
|
||||
import { and, eq, inArray, isNull, sql } from "drizzle-orm";
|
||||
import type { AutomaticCompletionReviewScope } from "../automatic-completion-review-notifications.js";
|
||||
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
||||
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
|
||||
import { and, asc, eq, gt, inArray, isNull, sql } from "drizzle-orm";
|
||||
import {
|
||||
completionContracts,
|
||||
issueThreadInteractions,
|
||||
@@ -24,7 +28,7 @@ const automaticPrompt =
|
||||
"Review the persisted native-run evidence and confirm whether this issue may be completed.";
|
||||
|
||||
/** Identify only proven system fallback cards; this lookup never changes state. */
|
||||
export async function findAutomaticCompletionReviews(db: Db, issueId?: string) {
|
||||
function findAutomaticReviewCandidates(db: Db, options: { issueId?: string; companyId?: string; id?: string; after?: string } = {}) {
|
||||
return db
|
||||
.select({ interaction: issueThreadInteractions, decision: statusDecisions })
|
||||
.from(issueThreadInteractions)
|
||||
@@ -83,17 +87,20 @@ export async function findAutomaticCompletionReviews(db: Db, issueId?: string) {
|
||||
sql`${issueThreadInteractions.payload}->'target'->>'key' = 'native_completion_review'`,
|
||||
sql`${issueThreadInteractions.payload}->'target'->>'revisionId' = ${statusDecisions.id}::text`,
|
||||
sql`split_part(${issueThreadInteractions.payload}->>'prompt', E'\n', 1) = ${automaticPrompt}`,
|
||||
...(issueId ? [eq(issueThreadInteractions.issueId, issueId)] : []),
|
||||
options.issueId ? eq(issueThreadInteractions.issueId, options.issueId) : undefined,
|
||||
options.companyId ? eq(issueThreadInteractions.companyId, options.companyId) : undefined,
|
||||
options.id ? eq(issueThreadInteractions.id, options.id) : undefined,
|
||||
options.after ? gt(issueThreadInteractions.id, options.after) : undefined,
|
||||
),
|
||||
)
|
||||
.limit(100)
|
||||
.catch((err) => {
|
||||
logger.warn(
|
||||
{ err },
|
||||
"Automatic completion review lookup failed; will retry",
|
||||
);
|
||||
return [];
|
||||
});
|
||||
.orderBy(asc(issueThreadInteractions.id)).limit(100);
|
||||
}
|
||||
|
||||
export async function findAutomaticCompletionReviews(db: Db, issueId?: string) {
|
||||
return findAutomaticReviewCandidates(db, { issueId }).catch((err) => {
|
||||
logger.warn({ err }, "Automatic completion review lookup failed; will retry");
|
||||
return [];
|
||||
});
|
||||
}
|
||||
|
||||
/** Narrow, replay-safe retirement. Explicit requests and answered cards are immutable here. */
|
||||
@@ -101,7 +108,10 @@ export async function dismissAutomaticCompletionReviews(
|
||||
db: Db,
|
||||
issueId?: string,
|
||||
) {
|
||||
const candidates = await findAutomaticCompletionReviews(db, issueId);
|
||||
await dismissCandidates(db, await findAutomaticCompletionReviews(db, issueId));
|
||||
}
|
||||
|
||||
async function dismissCandidates(db: Db, candidates: Awaited<ReturnType<typeof findAutomaticReviewCandidates>>) {
|
||||
for (const { interaction, decision } of candidates) {
|
||||
const publications: ActivityPublication[] = [];
|
||||
try {
|
||||
@@ -147,6 +157,10 @@ export async function dismissAutomaticCompletionReviews(
|
||||
eq(issueThreadInteractions.companyId, issue.companyId),
|
||||
eq(issueThreadInteractions.status, "pending"),
|
||||
eq(issueThreadInteractions.payload, interaction.payload),
|
||||
eq(issueThreadInteractions.kind, interaction.kind),
|
||||
eq(issueThreadInteractions.sourceRunId, decision.runId),
|
||||
eq(issueThreadInteractions.idempotencyKey, interaction.idempotencyKey!),
|
||||
isNull(issueThreadInteractions.createdByAgentId), isNull(issueThreadInteractions.createdByUserId),
|
||||
),
|
||||
)
|
||||
.returning();
|
||||
@@ -213,3 +227,33 @@ export async function decisionHasRetiredAutomaticReview(
|
||||
.limit(1);
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
export function registerAutomaticCompletionReviewWork(db: Db, coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
|
||||
type Entry = { id: string; companyId: string; issueId: string; dueAt: number };
|
||||
async function load(scope?: AutomaticCompletionReviewScope, signal?: AbortSignal): Promise<Entry[]> {
|
||||
const entries = new Map<string, Entry>();
|
||||
let after: string | undefined;
|
||||
while (canRun() && !signal?.aborted) {
|
||||
const rows = await findAutomaticReviewCandidates(db, { ...scope, after });
|
||||
for (const { interaction } of rows) entries.set(interaction.id, {
|
||||
id: interaction.id, companyId: interaction.companyId, issueId: interaction.issueId, dueAt: 0,
|
||||
});
|
||||
if (rows.length < 100) return [...entries.values()];
|
||||
after = rows[rows.length - 1].interaction.id;
|
||||
}
|
||||
throw new Error("Automatic completion review discovery stopped");
|
||||
}
|
||||
return registerScopedDeadlineWork(coordinator, {
|
||||
queue: DELIVERY_QUEUES.automaticCompletionReviews, batchSize: 20,
|
||||
decodeScope: (hint: string): AutomaticCompletionReviewScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
|
||||
matches: (entry: Entry, scope?: AutomaticCompletionReviewScope) => !scope || entry.companyId === scope.companyId && entry.issueId === scope.issueId,
|
||||
load,
|
||||
check: async (entry: Entry, signal?: AbortSignal) => {
|
||||
const candidates = await findAutomaticReviewCandidates(db, entry);
|
||||
if (!canRun() || signal?.aborted) throw new Error("Automatic completion review cleanup stopped");
|
||||
await dismissCandidates(db, candidates);
|
||||
const pending = await findAutomaticReviewCandidates(db, entry);
|
||||
return pending.length ? { ...entry, dueAt: Date.now() + 30_000 } : null;
|
||||
},
|
||||
}, canRun);
|
||||
}
|
||||
@@ -32,6 +32,14 @@ describe("native finalization reconciliation timing", () => {
|
||||
timing.phase.mockClear(); timing.finish.mockClear();
|
||||
});
|
||||
|
||||
it("omits automatic review discovery when its worker owns that duty", async () => {
|
||||
timing.phase.mockClear();
|
||||
const db = { select: vi.fn(() => { throw new Error("candidate scan reached"); }) };
|
||||
await expect(reconcileNativeFinalizations(db as never, ["run"], { skipAutomaticCompletionReviews: true })).rejects.toThrow("candidate scan reached");
|
||||
expect(timing.phase.mock.calls.map(([phase]) => phase)).toEqual(["legacy_workspace_exports", "obsolete_policy_reviews", "candidate_scan"]);
|
||||
timing.phase.mockClear(); timing.finish.mockClear();
|
||||
});
|
||||
|
||||
it("records the phases and a failed outcome when a pass fails, and rethrows", async () => {
|
||||
await expect(reconcileNativeFinalizations({} as never)).rejects.toThrow("review lookup failed");
|
||||
|
||||
|
||||
@@ -638,6 +638,7 @@ export async function claimNativeSessionResumptions(input: {
|
||||
|
||||
type NativeReconciliationOptions = {
|
||||
skipObsoletePolicyReviews?: boolean;
|
||||
skipAutomaticCompletionReviews?: boolean;
|
||||
pluginWorkerManager?: WorkspaceWorkers;
|
||||
environmentRuntime?: EnvironmentRuntimeService;
|
||||
onWorkspaceSettled?: (input: {
|
||||
@@ -730,13 +731,15 @@ async function reconcileNativeFinalizationsPass(
|
||||
logger.warn({ err }, "Obsolete native policy review lookup failed; continuing native reconciliation");
|
||||
});
|
||||
}
|
||||
timing.phase("automatic_completion_reviews");
|
||||
if (runIds?.length) {
|
||||
const scopes = await db.select({ issueId: nativeRunFinalizations.issueId }).from(nativeRunFinalizations)
|
||||
.where(inArray(nativeRunFinalizations.runId, runIds));
|
||||
for (const scope of scopes) await dismissAutomaticCompletionReviews(db, scope.issueId);
|
||||
} else {
|
||||
await dismissAutomaticCompletionReviews(db);
|
||||
if (!options.skipAutomaticCompletionReviews) {
|
||||
timing.phase("automatic_completion_reviews");
|
||||
if (runIds?.length) {
|
||||
const scopes = await db.select({ issueId: nativeRunFinalizations.issueId }).from(nativeRunFinalizations)
|
||||
.where(inArray(nativeRunFinalizations.runId, runIds));
|
||||
for (const scope of scopes) await dismissAutomaticCompletionReviews(db, scope.issueId);
|
||||
} else {
|
||||
await dismissAutomaticCompletionReviews(db);
|
||||
}
|
||||
}
|
||||
timing.phase("candidate_scan");
|
||||
const now = new Date();
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyAutomaticCompletionReviews } from "../automatic-completion-review-notifications.js";
|
||||
import { notifyObsoletePolicyReviews } from "../obsolete-policy-review-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
||||
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
||||
@@ -1983,6 +1984,7 @@ export async function commitNativeStatusDecision(input: {
|
||||
}
|
||||
await notifyChatRunMilestones(tx, input);
|
||||
await notifyObsoletePolicyReviews(tx, input);
|
||||
await notifyAutomaticCompletionReviews(tx, input);
|
||||
if (!decisionRow) {
|
||||
const latestDecisionVersion = await tx
|
||||
.select({
|
||||
|
||||
Reference in new issue
Block a user