fix(server): schedule governed review response recovery (#17702)

## Thinking Path

> - Paperclip manages agents, tasks and their external chat responses.
> - A completed chat turn can owe a response while an earlier review
gate remains pending.
> - Periodic recovery revisits that response history even when no work
can advance.
> - This change uses the existing delivery coordinator and saved retry
deadlines.
> - Approval, interaction and authority changes wake the affected issue
or run after commit.
> - The existing presentation owner verifies the durable result and
current authority.
> - Settled and dormant history becomes quiet without losing future
eligible responses.

## Linked Issues or Issue Description

Refs #17696 (merged) and #15625. This is slice 49, now based on master.
It owns governed-review response presentation. Accepted responses and
pending decision effects keep their existing owners.

**What happened?**

Native finalization recovery repeatedly inspects committed review
responses. Removing the scan without covering approval, interaction,
task-policy and authority changes would strand responses that become
eligible later.

**Expected behavior**

Recover an owed response once from existing durable facts. Enforce
authority at publication, wake after relevant outer transactions commit,
retain real retry deadlines and restart/drain recovery, and stop
recurring SQL when no response can advance.

**Steps to reproduce**

Complete an external chat turn while an earlier completion review
remains pending. Block its response behind an approval or interaction,
then resolve that blocker. The response must appear once without a
periodic scan or resolving the original review gate.

## What Changed

- Add a scoped governed-review worker using the existing coordinator,
finalizations, decisions and comment receipts. No queue table or
transaction wrapper is added.
- Keep recovery and publication policy in the native finalizer. Preserve
binding, source/session authority, ownership, deadline and
deleted-comment fences; match coordinator-before-issue lock ordering.
- Notify through actual approval, interaction, OAuth, task-policy and
lifecycle producers before their owned writes. Serialize approval
resolution with task linking so a concurrent link retains its wake.
Nested transactions still settle at outer commit.
- Scope ordinary milestones, finalization and coalesced input to their
issue/run. Use the explicit company authority boundary for genuine
global restoration, including membership changes. Both response workers
share this boundary.
- Register under maintenance and drain gates, disable the two periodic
repair entrances, and retain fresh finalization and explicit recovery.
Add the concurrent 0413 comment-receipt index and migration
registration; retain five snapshots.
- Preserve the producer inventory, source qualifications and earlier
failures in the owner audit. Retain a real Photon regression proving
routine ownership assertions stay quiet and concurrent authority
restoration wakes each worker once.

## Verification

- Scoped repair source `2341714720b5c273f0ce011b48345af0ef39227f` passes
**2053 distinct cases**: 1703 in 15 complete producer/authority/recovery
files, including all 273 governed-response cases and 95 wake-queue
adapter cases, plus 350 selected chat cases. Another 1463 chat cases are
outside that selection. Thirteen fresh PostgreSQL 17 fixtures migrate
through unchanged 0413; the one temporary queue-fixture import restores
exactly. All sixteen captured hashes match.
- Two actual repeated coarse-hint SQL regressions fail before repair and
pass afterward. Actual membership restoration passes before and after.
The same assertions and time limits prove dormant work stays quiet and
real authority restoration still publishes once. Real coalesced input
covers both queues after commit.
- Combined source `586f5674cd9798b33cd22437255e0480b5726d20`, including
independent master #18129, passes **1297 distinct cases**: 1093 in five
complete server files, 185 in five runner files, 14 Node protocol cases
and five UI registry cases. This includes the complete 22-case Photon
file for both response workers. The external Photon transport is mocked;
Paperclip callbacks, transaction fences and observed SQL are real. The
earlier and combined case counts are separate, with no cumulative
double-counting.
- A normal merge retains both complete owned and independent master
binary patches. All twelve new-context hashes and sixteen earlier repair
hashes match. Three dedicated fresh 0413 fixtures plus the live-event
helper's own fresh database use PostgreSQL 17 without source
adaptations. Database build, server types/build, boundaries, full-SHA
migration order and generated TS/Rust profiles pass. The new inherited
Runner vendor inputs rebuild successfully.
- Database bytes remain identical to the previously qualified 0413
source. The recorded populated 0412-to0413 upgrade retains all 411 prior
receipts and 38 companies, appends one receipt and none on replay; the
index is valid and ready. The earlier 2828-case acceptance, CLI journey,
setup failures and their repairs retain their recorded source scope in
`doc/plans/2026-10-26-governed-review-response-sleep-audit.md`.
- Parent #17696 merged with 58 clean checks, Greptile 5/5 and Ponytail
5/5. Its Photon warning was disproved by real callback/SQL evidence and
the security check became successful. This PR still requires its own
fresh exact-head CI and review gates. No deployed sleep, external alarm
or fleet savings claim is made.

## Risks

- Notifications use the existing process-local writer contract.
Independent SQL writers and uncoordinated replicas are outside its
coverage.
- Genuine startup and company authority reconstruction remain
proportional to the owed backlog. Queries page 100 rows, but discovery
accumulates the candidates; this is not a constant-cost global
operation.
- Current publication authorization, durable binding, ownership and lock
ordering remain mandatory. The worker does not resolve the original
review gate.
- The concurrent index uses existing migration replay handling. Other
recovery duties and Cloud sleep eligibility remain separate.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution and PostgreSQL tests. The runtime does not expose a more
specific serving snapshot or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
authored and GitHub committed 2026-10-29 02:54:46 -05:00
1 parent 80a2b9284b
commit 79ce86630a
40 files changed
+1953 -145

No files matched your search

@@ -1,6 +1,6 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { approvalService } from "../services/approvals.ts";
import { companies } from "@paperclipai/db";
import { approvals, companies, issueApprovals } from "@paperclipai/db";
vi.mock("../services/budgets.js", async importOriginal => ({
...await importOriginal<typeof import("../services/budgets.js")>(),
@@ -50,10 +50,14 @@ function createDbStub(selectResults: ApprovalRecord[][], updateResults: Approval
const pendingSelectResults = [...selectResults];
let inTransaction = false;
const selectWhere = vi.fn(async () => (inTransaction ? pendingSelectResults.shift() : pendingSelectResults[0]) ?? []);
const from = vi.fn((table) => table === companies
const from = vi.fn((table) => table === issueApprovals
? { where: async () => [] }
: table === companies
? { where: () => ({ for: async () => [{ id: "company-1" }] }) }
: { where: selectWhere });
const select = vi.fn(() => ({ from }));
const select = vi.fn((fields?: { id?: unknown }) => fields?.id === approvals.id
? { from: () => ({ where: () => ({ for: async () => [{ id: "approval-1" }] }) }) }
: { from });
const returning = vi.fn(async () => updateResults);
const updateWhere = vi.fn(() => ({ returning }));
@@ -1,3 +1,4 @@
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
@@ -29,7 +30,8 @@ vi.mock("../telemetry.js", () => ({
getTelemetryClient: telemetryMocks.getTelemetryClient,
}));
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_INTERACTION_TELEMETRY_TEST_DATABASE_URL");
const embeddedPostgresSupport = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
describeEmbeddedPostgres("issueThreadInteractionService telemetry", () => {
@@ -38,8 +40,8 @@ describeEmbeddedPostgres("issueThreadInteractionService telemetry", () => {
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
beforeAll(async () => {
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-interaction-telemetry-");
db = createDb(tempDb.connectionString);
if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-interaction-telemetry-");
db = createDb(externalUrl ?? tempDb!.connectionString);
interactionsSvc = issueThreadInteractionService(db);
}, 20_000);
@@ -65,6 +67,7 @@ describeEmbeddedPostgres("issueThreadInteractionService telemetry", () => {
});
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await tempDb?.cleanup();
});
@@ -1,3 +1,7 @@
import { nativeReviewResponseWork } from "../../services/native-runtime/native-review-response-work.js";
import { nativeResponsePresentationWork } from "../../services/native-runtime/native-response-presentation-work.js";
import { DELIVERY_QUEUES } from "../../services/delivery-work-notifications.js";
import { idleWorkSnapshot } from "../../services/task-admission.js";
import { createDeliveryWorkCoordinator } from "../../services/delivery-work-coordinator.js";
import { resolveExternalTestDatabaseUrl } from "../helpers/external-test-database.js";
import { Readable } from "node:stream";
@@ -24,6 +28,7 @@ import {
import { and, eq, sql } from "drizzle-orm";
import {
applyPendingMigrations,
subscribeDatabaseWork,
createDb,
companies,
agents,
@@ -372,6 +377,53 @@ describe("iMessage Photon channel control plane", () => {
},
};
}
it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse])(
"keeps repeated active Photon ownership assertions out of %s reconstruction and wakes once on recovery", async queue => {
const t = await setup();
await t.start();
await t.qualify();
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
coordinators.push(coordinator);
const work = queue === DELIVERY_QUEUES.nativeResponsePresentation ? nativeResponsePresentationWork(db) : nativeReviewResponseWork(db);
const worker = work.register(coordinator, () => true);
const debug = db.$client.options.debug;
const hints: unknown[] = [];
const queries: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, queue, (event, values) => {
if (event === "settled") for (const value of values ?? []) {
const scope = JSON.parse(value);
if (scope.companyId === t.companyId) hints.push(scope);
}
});
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 })
.toEqual({ active: 0, next: null });
db.$client.options.debug = (_connection, query) => {
if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query);
};
const assertOwned = t.callbacks().onPhotonAssertOwned!;
for (let i = 0; i < 5; i++) await assertOwned();
expect(hints).toEqual([]);
expect(queries).toEqual([]);
await db.update(chatEndpoints).set({ status: "attention", healthMessage: "Recovering transport" })
.where(eq(chatEndpoints.id, t.endpoint.id));
await Promise.all([assertOwned(), assertOwned(), assertOwned()]);
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 })
.toEqual({ active: 0, next: null });
expect(hints).toEqual([{ companyId: t.companyId }]);
expect(queries).toHaveLength(1);
expect((await t.service.get(t.endpoint.id)).status).toBe("active");
hints.length = 0;
queries.length = 0;
for (let i = 0; i < 5; i++) await assertOwned();
expect(hints).toEqual([]);
expect(queries).toEqual([]);
expect(errors).toEqual([]);
} finally { unsubscribe(); db.$client.options.debug = debug; if ("stop" in worker) worker.stop(); }
});
it("recovers a Photon transport loss through the receiver worker and cancels fatal recovery", async () => {
const t = await setup();
const onError = vi.fn();
@@ -21,19 +21,16 @@ import { idleWorkSnapshot } from "../services/task-admission.js";
import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";
import { reserveRunBudget } from "../services/budget-reservations.js";
const externalTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_SATISFIED_CONNECTION_TEST_DATABASE_URL");
const support = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_SATISFIED_CONNECTION_TEST_DATABASE_URL");
const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
const describePostgres = support.supported ? describe : describe.skip;
describePostgres("already available connection requests", () => {
let db!: ReturnType<typeof createDb>;
let temp!: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
beforeAll(async () => {
if (externalTestDatabaseUrl) db = createDb(externalTestDatabaseUrl);
else {
temp = await startEmbeddedPostgresTestDatabase("paperclip-satisfied-connections-");
db = createDb(temp.connectionString);
}
if (!externalUrl) temp = await startEmbeddedPostgresTestDatabase("paperclip-satisfied-connections-");
db = createDb(externalUrl ?? temp.connectionString);
await db.insert(authUsers).values({ id: "connection-owner", name: "Owner", email: "owner@example.test", createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing();
}, 20_000);
beforeEach(async () => { await db.execute(sql`truncate table companies cascade`); });
@@ -65,6 +62,19 @@ describePostgres("already available connection requests", () => {
return run!;
}
it("wakes governed response recovery only when automatic cleanup retires a connection card", async () => {
const f = await seed();
const notified = vi.fn();
const unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeReviewResponse, notified);
try {
expect(await satisfiedConnectionIntentService(db).sweepPending()).toMatchObject({ satisfied: 1 });
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, f.intent.id)))[0].status).toBe("expired");
expect(notified).toHaveBeenCalledOnce();
expect(await satisfiedConnectionIntentService(db).sweepPending()).toMatchObject({ satisfied: 0 });
expect(notified).toHaveBeenCalledOnce();
} finally { unsubscribe(); }
});
it("notifies terminal confirmation recovery after existing-access retirement commits", async () => {
const f = await seed();
const notified = vi.fn();
@@ -398,7 +408,7 @@ describePostgres("already available connection requests", () => {
it("records a system completion and one durable continuation with a one-connection pool", async () => {
const f = await seed();
const single = createDb(externalTestDatabaseUrl ?? temp.connectionString, { maxConnections: 1 });
const single = createDb(externalUrl ?? temp.connectionString, { maxConnections: 1 });
try {
const outcomes = await Promise.all([satisfiedConnectionIntentService(single).sweepPending(), satisfiedConnectionIntentService(db).sweepPending()]);
expect(outcomes.reduce((sum, outcome) => sum + outcome.satisfied, 0)).toBe(1);
@@ -116,6 +116,7 @@ const {
reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })),
registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerNativeReviewResponseWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerNativeResponsePresentationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })),
registerNativePendingEffectWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerUnsafeWorkspaceExportWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
@@ -570,6 +571,13 @@ describe("startServer feedback export wiring", () => {
}
});
it.each([false, true])("registers governed-response recovery only when maintenance is enabled (%s)", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
await startServer(); await settleStartupHousekeeping();
expect(heartbeatServiceMock.registerNativeReviewResponseWork).toHaveBeenCalledTimes(enabled ? 1 : 0);
if (enabled) expect(heartbeatServiceMock.registerNativeReviewResponseWork.mock.calls[0][1]()).toBe(true);
});
it.each([false, true])("registers native pending effects only when maintenance is enabled (%s)", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
await startServer(); await settleStartupHousekeeping();
@@ -1017,7 +1025,7 @@ describe("startServer feedback export wiring", () => {
intervalCallback?.();
await new Promise<void>((resolve) => setImmediate(resolve));
expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2);
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true });
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true });
expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1);
expect(retiredDetector).not.toHaveBeenCalled();
} finally {
+3 -1
View File
@@ -1825,6 +1825,8 @@ async function startServerWithDatabaseTeardown(
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeReviewResponseWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) {
const worker = heartbeat.registerNativeResponsePresentationWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
@@ -1972,7 +1974,7 @@ async function startServerWithDatabaseTeardown(
// Periodically reap orphaned runs (5-min staleness threshold) and make sure
// persisted queued work is still being driven forward.
runPeriodicRecovery(() => heartbeat
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true })
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true })
.then(() => heartbeat.promoteDueScheduledRetries())
.then(async (promotion) => {
await heartbeat.resumeQueuedRuns();
+1 -1
View File
@@ -52,7 +52,7 @@ export function approvalResolutionRecords(db: Db) {
.where(and(eq(approvals.id, id), inArray(approvals.status, resolvableStatuses)))
.returning()
.then((rows) => rows[0] ?? null);
});
}, { approvalId: id });
if (updated) {
return { approval: updated, applied: true };
@@ -1224,12 +1224,13 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => {
expect(wakeRow?.payload).not.toHaveProperty("foo");
});
it("scopes a real coalesced-input response wake to its active run after commit", async () => {
it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse])(
"scopes a real coalesced-input %s response wake to its active run after commit", async queue => {
const companyId = await seedCompany();
const agentId = await seedAgent({ companyId });
const runId = await seedRun({ companyId, agentId, status: "running" });
const hints: Array<Record<string, unknown>> = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeResponsePresentation, (event, values) => {
const unsubscribe = subscribeDatabaseWork(db, queue, (event, values) => {
if (event === "settled") for (const value of values ?? []) hints.push(JSON.parse(value));
});
try {
+2
View File
@@ -1,3 +1,4 @@
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
import { isAgentStatusInvokable } from "@paperclipai/shared";
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
@@ -39,6 +40,7 @@ export function createAgentLifecycleEffects(_hooks: BudgetServiceHooks = {}): li
async beforeStatusChange(tx, companyId, agentId, before, after) {
if (!isAgentStatusInvokable(before) && isAgentStatusInvokable(after)) {
await notifyNativeResponsePresentation(tx, { companyId, agentId });
await notifyNativeReviewResponse(tx, { companyId, agentId });
await notifyChatInboundWakeNotices(tx, { companyId });
}
},
@@ -1,16 +1,32 @@
import type { Db } from "@paperclipai/db";
import { and, eq } from "drizzle-orm";
import { approvals, issueApprovals, type Db } from "@paperclipai/db";
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export const notifyApprovalQueueWork = (transaction: object, companyId: string) => notifyDeliveryWork(transaction, DELIVERY_QUEUES.approvalQueueSeeding, companyId);
export async function notifyApprovalWork(transaction: object, companyId: string) {
type ApprovalChangeScope = { approvalId?: string; issueIds?: string[] };
export async function notifyApprovalWork(transaction: object, companyId: string, scope?: ApprovalChangeScope) {
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.approvalRetention);
await notifyApprovalQueueWork(transaction, companyId);
const issueIds = new Set(scope?.issueIds);
if (scope?.approvalId) {
await (transaction as Db).select({ id: approvals.id }).from(approvals)
.where(and(eq(approvals.companyId, companyId), eq(approvals.id, scope.approvalId))).for("update");
const links = await (transaction as Db).select({ issueId: issueApprovals.issueId }).from(issueApprovals)
.where(and(eq(issueApprovals.companyId, companyId), eq(issueApprovals.approvalId, scope.approvalId)));
for (const link of links) issueIds.add(link.issueId);
}
for (const issueId of issueIds) await notifyNativeReviewResponse(transaction, { companyId, issueId });
}
export function writeApprovalChange<T>(db: Db, companyId: string, write: (transaction: Db) => Promise<T>): Promise<T> {
export function writeApprovalChange<T>(db: Db, companyId: string, write: (transaction: Db) => Promise<T>, scope?: ApprovalChangeScope): Promise<T> {
return db.transaction(async tx => {
await notifyApprovalWork(tx, companyId);
await notifyApprovalWork(tx, companyId, scope);
return write(tx as unknown as Db);
});
}
+3 -3
View File
@@ -80,7 +80,7 @@ export function approvalRecords(db: Db) {
.where(and(eq(approvals.id, id), inArray(approvals.status, resolvableStatuses)))
.returning()
.then((rows) => rows[0] ?? null);
});
}, { approvalId: id });
return updated;
},
@@ -102,7 +102,7 @@ export function approvalRecords(db: Db) {
})
.where(eq(approvals.id, id))
.returning()
.then((rows) => rows[0]));
.then((rows) => rows[0]), { approvalId: id });
},
resubmit: async (id: string, payload?: Record<string, unknown>) => {
@@ -124,7 +124,7 @@ export function approvalRecords(db: Db) {
})
.where(eq(approvals.id, id))
.returning()
.then((rows) => rows[0]));
.then((rows) => rows[0]), { approvalId: id });
},
listComments: async (approvalId: string) => {
+1 -1
View File
@@ -262,7 +262,7 @@ async function markApprovalStatus(
if (!approvalId) return;
const [approval] = await db.select({ companyId: approvals.companyId }).from(approvals).where(eq(approvals.id, approvalId));
if (!approval) return;
await notifyApprovalWork(db, approval.companyId);
await notifyApprovalWork(db, approval.companyId, { approvalId });
await db
.update(approvals)
.set({
@@ -1,8 +1,10 @@
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export type ChatResponseScope = { companyId: string; issueId?: string; deliveryId?: string; conversationId?: string };
export async function notifyChatResponseAvailability(tx: object, scope: ChatResponseScope) {
if (scope.issueId) await notifyNativeReviewResponse(tx, { companyId: scope.companyId, issueId: scope.issueId });
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubTerminalResponses, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, deliveryId: scope.deliveryId, conversationId: scope.conversationId }));
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubAcknowledgements, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, deliveryId: scope.deliveryId, conversationId: scope.conversationId }));
}
@@ -1,3 +1,4 @@
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
import { notifyChatConfirmationWork } from "./chat-confirmation-notifications.js";
@@ -11,6 +12,8 @@ export type ChatRunMilestoneScope = { companyId: string; issueId?: string; runId
export async function notifyChatRunMilestones(tx: object, scope: ChatRunMilestoneScope) {
if (scope.issueId || scope.runId) {
await notifyNativeReviewResponse(tx, { companyId: scope.companyId, issueId: scope.issueId,
runId: scope.issueId ? undefined : scope.runId });
await notifyNativeResponsePresentation(tx, { companyId: scope.companyId, issueId: scope.issueId,
runId: scope.issueId ? undefined : scope.runId });
}
@@ -30,4 +33,5 @@ export async function notifyChatRunMilestones(tx: object, scope: ChatRunMileston
export async function notifyChatCompanyAuthorityChange(tx: object, scope: { companyId: string }) {
await notifyChatRunMilestones(tx, scope);
await notifyNativeResponsePresentation(tx, scope);
await notifyNativeReviewResponse(tx, scope);
}
@@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
/** Topics for existing durable queues; register intent before their writes. */
export const DELIVERY_QUEUES = {
nativeReviewResponse: "native-review-response",
nativeResponsePresentation: "native-response-presentation",
nativePendingEffects: "native-pending-effects",
unsafeWorkspaceExportRecovery: "unsafe-workspace-export-recovery",
+3
View File
@@ -1,3 +1,4 @@
import { registerNativeReviewResponseWork } from "./native-runtime/native-review-response-work.js";
import { registerNativeResponsePresentationWork } from "./native-runtime/native-response-presentation-work.js";
import { registerNativePendingEffectWork } from "./native-runtime/native-pending-effect-work.js";
import { registerUnsafeWorkspaceExportWork } from "./native-runtime/native-workspace-export-recovery.js";
@@ -4090,6 +4091,8 @@ export function heartbeatService(
triggerIssueMonitor,
reportRunActivity: clearDetachedRunWarning,
registerNativeReviewResponseWork: (coordinator: Parameters<typeof registerNativeReviewResponseWork>[1], canRun: () => boolean) =>
registerNativeReviewResponseWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
registerNativeResponsePresentationWork: (coordinator: Parameters<typeof registerNativeResponsePresentationWork>[1], canRun: () => boolean) =>
registerNativeResponsePresentationWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
registerNativePendingEffectWork: (coordinator: Parameters<typeof registerNativePendingEffectWork>[1], canRun: () => boolean) =>
+2 -1
View File
@@ -1947,7 +1947,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
let settledProcessesDiscovered = false;
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean }) {
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean }) {
if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db);
const staleThresholdMs = opts?.staleThresholdMs ?? 0;
const now = new Date();
@@ -1970,6 +1970,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
skipUnsafeWorkspaceExports: opts?.skipUnsafeWorkspaceExports,
skipNativePendingEffects: opts?.skipNativePendingEffects,
skipNativeResponsePresentation: opts?.skipNativeResponsePresentation,
skipNativeReviewResponse: opts?.skipNativeReviewResponse,
environmentRuntime,
pluginWorkerManager: dependencies.pluginWorkerManager,
onWorkspaceSettled: settleRecoveredNativeWorkspace,
+3 -3
View File
@@ -117,7 +117,7 @@ export function issueApprovalService(db: Db) {
linkedByAgentId: actor?.agentId ?? null,
linkedByUserId: actor?.userId ?? null,
})
.onConflictDoNothing());
.onConflictDoNothing(), { issueIds: [issueId] });
return db
.select()
@@ -130,7 +130,7 @@ export function issueApprovalService(db: Db) {
const { issue } = await assertIssueAndApprovalSameCompany(issueId, approvalId);
await writeApprovalChange(db, issue.companyId, tx => tx
.delete(issueApprovals)
.where(and(eq(issueApprovals.issueId, issueId), eq(issueApprovals.approvalId, approvalId))));
.where(and(eq(issueApprovals.issueId, issueId), eq(issueApprovals.approvalId, approvalId))), { issueIds: [issueId] });
},
linkManyForApproval: async (approvalId: string, issueIds: string[], actor?: LinkActor) => {
@@ -169,7 +169,7 @@ export function issueApprovalService(db: Db) {
linkedByUserId: actor?.userId ?? null,
})),
)
.onConflictDoNothing());
.onConflictDoNothing(), { issueIds: uniqueIssueIds });
},
};
}
@@ -1,3 +1,4 @@
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyToolReviewChange } from "./tool-review-work-notifications.js";
@@ -1944,6 +1945,7 @@ async function expireStaleRequestConfirmationTarget(
const expireInTransaction = async (tx: Db | any) => {
await notifyChatConfirmationResolution(tx, args.row);
if (args.row.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.row);
else await notifyNativeReviewResponse(tx, args.row);
const [updated] = await tx
.update(issueThreadInteractions)
.set({
@@ -2223,6 +2225,7 @@ export function issueThreadInteractionService(
await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor);
await notifyChatConfirmationResolution(tx, args.current);
if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current);
else await notifyNativeReviewResponse(tx, args.current);
await args.mutationOptions?.beforeResolveInTransaction?.(tx);
// Policy mutations and review transitions use the same issue-row lock,
// so the authoritative review policy and requester are stable through
@@ -2513,6 +2516,7 @@ export function issueThreadInteractionService(
await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor);
await notifyChatConfirmationResolution(tx, args.current);
if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current);
else await notifyNativeReviewResponse(tx, args.current);
await args.mutationOptions?.beforeResolveInTransaction?.(tx);
const issueContext = await tx
.select({
@@ -2905,6 +2909,7 @@ export function issueThreadInteractionService(
const updated = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, current);
await mutationOptions.beforeResolveInTransaction?.(tx);
await notifyNativeReviewResponse(tx, current);
const [row] = await tx
.update(issueThreadInteractions)
.set({
@@ -3340,6 +3345,7 @@ export function issueThreadInteractionService(
const updated = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, row);
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
else await notifyNativeReviewResponse(tx, row);
if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]);
const [resolved] = await tx
.update(issueThreadInteractions)
@@ -3427,6 +3433,7 @@ export function issueThreadInteractionService(
const updated = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, row);
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
else await notifyNativeReviewResponse(tx, row);
if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]);
const [updatedRow] = await tx
.update(issueThreadInteractions)
@@ -4208,6 +4215,7 @@ export function issueThreadInteractionService(
}
}
}
await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id });
const resolvedAt = new Date();
const [claimed] = await tx
.update(issueThreadInteractions)
@@ -4471,6 +4479,7 @@ export function issueThreadInteractionService(
complete,
items,
} satisfies RequestItemVerdictsResult;
if (complete) await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id });
const [updated] = await tx
.update(issueThreadInteractions)
.set({
@@ -4537,6 +4546,7 @@ export function issueThreadInteractionService(
await notifyChatConfirmationResolution(tx, current);
await mutationOptions.beforeResolveInTransaction?.(tx);
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id });
const rows = await tx.update(issueThreadInteractions)
.set({
status: "rejected",
@@ -4628,6 +4638,7 @@ export function issueThreadInteractionService(
const updated = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, row);
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
else await notifyNativeReviewResponse(tx, row);
if (row.kind === "connection_intent") {
await tx
.delete(toolOauthStates)
@@ -4805,6 +4816,7 @@ export function issueThreadInteractionService(
const updatedInteractions = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, sampleConfirmationRow);
if (commentRows.some(row => row.kind === "request_confirmation")) await notifyChatRunMilestones(tx, sampleConfirmationRow);
else await notifyNativeReviewResponse(tx, sampleConfirmationRow);
await notifyToolProjectionForInteractions(tx, issue.companyId, confirmationRowIds);
const updatedRows = await tx
.update(issueThreadInteractions)
@@ -4840,6 +4852,7 @@ export function issueThreadInteractionService(
for (const row of itemVerdictRows) {
const updated = await db.transaction(async tx => {
await notifyNativeReviewResponse(tx, row);
const [retired] = await tx
.update(issueThreadInteractions)
.set({
@@ -4865,6 +4878,7 @@ export function issueThreadInteractionService(
for (const row of connectionIntentRows) {
const [updated] = await db.transaction(async (tx) => {
await notifyNativeReviewResponse(tx, row);
await notifyChatConfirmationResolution(tx, row);
await tx
.delete(toolOauthStates)
@@ -4953,6 +4967,7 @@ export function issueThreadInteractionService(
const updated = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, row);
if (row.kind === "request_confirmation") await notifyChatRunMilestones(tx, row);
else await notifyNativeReviewResponse(tx, row);
const [resolved] = await tx
.update(issueThreadInteractions)
.set({
@@ -5016,6 +5031,7 @@ export function issueThreadInteractionService(
? "the AI account"
: `the ${payload?.serviceName?.trim() || "requested"} connection`;
const updated = await db.transaction(async tx => {
await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id });
await notifyChatConfirmationResolution(tx, { companyId: issue.companyId, issueId: issue.id, kind: "connection_intent" });
const [resolved] = await tx
.update(issueThreadInteractions)
@@ -5077,6 +5093,7 @@ export function issueThreadInteractionService(
.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, row);
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
else await notifyNativeReviewResponse(tx, row);
if (row.kind === "connection_intent") {
await tx
.delete(toolOauthStates)
@@ -5169,6 +5186,7 @@ export function issueThreadInteractionService(
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
await notifyChatConfirmationResolution(tx, current);
if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current);
else await notifyNativeReviewResponse(tx, current);
await resolveLinkedToolActionRequests(tx, current, {
status: "cancelled",
fromStatuses: ["pending", "approved"],
@@ -5431,6 +5449,7 @@ export function issueThreadInteractionService(
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
await notifyChatConfirmationResolution(tx, current);
if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current);
else await notifyNativeReviewResponse(tx, current);
await resolveLinkedToolActionRequests(tx, current, {
status: "cancelled",
fromStatuses: ["pending", "approved"],
+3 -1
View File
@@ -11453,7 +11453,9 @@ export function issueService(db: Db) {
await notifyDecisionTargetWork(tx, `issue:${id}`);
}
if ((patch.status !== undefined && patch.status !== receiptExisting.status) ||
(patch.assigneeAgentId !== undefined && patch.assigneeAgentId !== receiptExisting.assigneeAgentId)) {
(patch.assigneeAgentId !== undefined && patch.assigneeAgentId !== receiptExisting.assigneeAgentId) ||
(patch.executionState !== undefined && parseObject(receiptExisting.executionState).status === "pending" &&
parseObject(patch.executionState).status !== "pending")) {
await notifyChatResponseAvailability(tx, { companyId: receiptExisting.companyId, issueId: id });
}
if (patch.status !== undefined && patch.status !== receiptExisting.status) {
@@ -1,10 +1,14 @@
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export type NativeFinalizationWorkScope = { companyId: string; runId?: string; issueId?: string };
export async function notifyNativeFinalizationWork(transaction: object, scope: NativeFinalizationWorkScope) {
if (scope.issueId || scope.runId) await notifyNativeResponsePresentation(transaction, scope);
if (scope.issueId || scope.runId) {
await notifyNativeResponsePresentation(transaction, scope);
await notifyNativeReviewResponse(transaction, scope);
}
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativePendingEffects,
JSON.stringify({ companyId: scope.companyId, runId: scope.runId }));
}
@@ -0,0 +1,8 @@
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export type NativeReviewResponseScope = { companyId: string; issueId?: string; runId?: string; agentId?: string };
export function notifyNativeReviewResponse(tx: object, scope: NativeReviewResponseScope) {
return notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeReviewResponse,
JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId, agentId: scope.agentId }));
}
@@ -1,3 +1,6 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
import { accessService } from "../access.js";
import { initializeRunIdentity } from "../run-identity.js";
import { createHash, randomUUID } from "node:crypto";
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
@@ -28,6 +31,7 @@ import {
completionContracts,
closeRegisteredClients,
createDb,
subscribeDatabaseWork,
heartbeatRuns,
issueComments,
issueAttachments,
@@ -41,6 +45,7 @@ import {
statusDecisions,
toolApplications,
toolConnections,
toolOauthStates,
chatVoiceSessions, chatVoiceReplies, chatVoiceToolCalls,
workspaceOperations,
} from "@paperclipai/db";
@@ -51,7 +56,17 @@ import type {
} from "../../vendor/paperclip-runner/index.js";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { finalizeNativeRun } from "./native-run-finalizer.js";
import { finalizeNativeRun, recoverCommittedNativeReviewResponse } from "./native-run-finalizer.js";
import { commitNativeStatusDecision } from "./status-decision-committer.js";
import { configureAgentLifecycle, createAgentLifecycle, processAgentLifecycle } from "../agent-lifecycle.js";
import { createOAuthStateStore } from "../tool-access/oauth/state.js";
import { approvalRecords } from "../approvals.js";
import { issueApprovalService } from "../issue-approvals.js";
import { NativeRunCoordinatorStore } from "./native-run-coordinator-store.js";
import { nativeReviewResponseWork } from "./native-review-response-work.js";
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
import { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import { idleWorkSnapshot } from "../task-admission.js";
import { nativeCompletionFeedback } from "./native-completion-feedback.js";
import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js";
import * as chatWaitAuthority from "./chat-attachment-reuse.js";
@@ -422,6 +437,10 @@ describe("native external-chat response wait", () => {
userId,
commentId,
principalId,
storeBinding: { companyId, issueId, runId, agentId, normalizedSessionId: sessionId,
runnerSourceInstanceId: runnerInstanceId, completionContractId: contractId,
completionContractSha256: contractSha256, completionContractRevision: "external-chat-wait-v1",
completionContractCriterionIds: ["response"] },
};
}
@@ -2869,6 +2888,653 @@ describe("native external-chat response wait", () => {
return run!.resultJson!;
}
async function seedRecoverableReviewResponse() {
const fixture = await seedWaitTurn();
const gate = await seedPriorCompletionReview(fixture);
await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: false });
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() })
.where(eq(heartbeatRuns.id, fixture.runId));
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(coordinator!.phase).toBe("committed");
const input = { companyId: fixture.companyId, issueId: fixture.issueId, runId: fixture.runId,
decisionId: coordinator!.decisionId!, resultId: coordinator!.resultId!, assessmentId: coordinator!.assessmentId!, onlyWhenDue: true };
return { ...fixture, gate, input };
}
it.each(["reconciler", "finalizer"] as const)("defers periodic governed-response %s repair to its registered worker", async path => {
const fixture = await seedRecoverableReviewResponse();
await db.update(heartbeatRuns).set({ resultJson: sql`${heartbeatRuns.resultJson} - 'externalChatReviewPresentation'` })
.where(eq(heartbeatRuns.id, fixture.runId));
if (path === "reconciler") await reconcileNativeFinalizations(db, [fixture.runId], { skipNativeReviewResponse: true });
else await finalizeNativeRun({ db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded",
projectRunStatus: true, skipNativeReviewResponse: true });
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).toHaveLength(0);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect(run.resultJson?.externalChatReviewPresentation).toBeUndefined();
const work = nativeReviewResponseWork(db);
const [entry] = await work.load({ companyId: fixture.companyId, runId: fixture.runId });
expect(entry?.id).toBe(fixture.runId);
expect(await work.check(entry)).toBeNull();
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).toHaveLength(1);
});
it("recovers scheduled governed responses once under concurrent callers and preserves deleted receipts", async () => {
const fixture = await seedRecoverableReviewResponse();
const work = nativeReviewResponseWork(db);
const [entry] = await work.load({ companyId: fixture.companyId });
expect(entry?.id).toBe(fixture.runId);
expect(await Promise.all([recoverCommittedNativeReviewResponse(db, fixture.input),
recoverCommittedNativeReviewResponse(db, fixture.input)])).toEqual(expect.arrayContaining([true, false]));
const comments = await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId));
expect(comments).toHaveLength(1);
expect(await db.select().from(chatPublications).where(eq(chatPublications.commentId, comments[0].id))).toHaveLength(1);
expect(await work.load({ companyId: fixture.companyId })).toEqual([]);
await db.update(issueComments).set({ deletedAt: new Date() }).where(eq(issueComments.id, comments[0].id));
expect(await work.load({ companyId: fixture.companyId })).toEqual([]);
expect(await recoverCommittedNativeReviewResponse(db, fixture.input)).toBe(false);
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).toHaveLength(1);
});
it("lets status replay finish while governed response recovery waits for its coordinator", async () => {
const fixture = await seedRecoverableReviewResponse();
const [issue] = await db.select().from(issues).where(eq(issues.id, fixture.issueId));
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.id, fixture.input.decisionId));
let recovery: Promise<{ value?: boolean; error?: unknown }> | undefined;
try {
await db.transaction(async tx => {
await tx.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId)).for("update");
const [backend] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
recovery = recoverCommittedNativeReviewResponse(db, fixture.input)
.then(value => ({ value }), error => ({ error }));
await vi.waitFor(async () => {
const [waiting] = await db.execute<{ count: number }>(sql`
select count(*)::int as count from pg_stat_activity
where datname = current_database() and wait_event_type = 'Lock'
and ${backend.pid} = any(pg_blocking_pids(pid))`);
expect(waiting.count).toBe(1);
});
await tx.execute(sql`set local lock_timeout = '500ms'`);
const replay = await commitNativeStatusDecision({
db: tx as unknown as typeof db, companyId: fixture.companyId,
issueId: fixture.issueId, runId: fixture.runId,
assessmentId: fixture.input.assessmentId,
priorStatus: issue.status, priorStatusVersion: issue.statusVersion,
priorDecisionId: issue.lastStatusDecisionId,
decision: decision.decisionJson as unknown as Parameters<typeof commitNativeStatusDecision>[0]["decision"],
});
expect(replay.replayed).toBe(true);
});
} finally {
if (recovery) {
const result = await recovery;
expect(result.error).toBeUndefined();
expect(result.value).toBe(true);
}
}
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).toHaveLength(1);
expect(await recoverCommittedNativeReviewResponse(db, fixture.input)).toBe(false);
});
it.each(["future", "claim"])("defers scheduled governed responses behind a new %s under the owner lock", async kind => {
const fixture = await seedRecoverableReviewResponse();
const work = nativeReviewResponseWork(db);
const [entry] = await work.load({ companyId: fixture.companyId });
const at = new Date(Date.now() + 60_000);
const transaction = db.transaction.bind(db);
const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => {
await transaction(async tx => {
await tx.update(nativeRunFinalizations).set(kind === "future" ? { nextAttemptAt: at }
: { leaseOwner: "another-finalizer", leaseExpiresAt: at }).where(eq(nativeRunFinalizations.runId, fixture.runId));
});
return transaction(callback, options);
}) as typeof db.transaction);
try {
expect(await work.check(entry)).toEqual(kind === "future" ? { ...entry, dueAt: at.getTime() } : null);
} finally { intercept.mockRestore(); }
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).toHaveLength(0);
await db.update(nativeRunFinalizations).set({ nextAttemptAt: null, leaseOwner: null, leaseExpiresAt: null })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(await recoverCommittedNativeReviewResponse(db, fixture.input)).toBe(true);
});
it("propagates scheduled governed response contention and rolls back its proof restoration", async () => {
const fixture = await seedRecoverableReviewResponse();
await db.update(heartbeatRuns).set({ resultJson: sql`${heartbeatRuns.resultJson} - 'externalChatReviewPresentation'` })
.where(eq(heartbeatRuns.id, fixture.runId));
let release!: () => void, observed!: () => void;
const hold = new Promise<void>(resolve => { release = resolve; });
const locked = new Promise<void>(resolve => { observed = resolve; });
const holder = db.transaction(async tx => {
await tx.select().from(chatEndpoints).where(eq(chatEndpoints.id, fixture.endpointId)).for("update");
observed(); await hold;
});
await locked;
try {
await expect(recoverCommittedNativeReviewResponse(db, fixture.input)).rejects.toBeDefined();
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
expect(run.resultJson?.externalChatReviewPresentation).toBeUndefined();
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).toHaveLength(0);
} finally { release(); await holder; }
expect(await recoverCommittedNativeReviewResponse(db, fixture.input)).toBe(true);
});
it.each(["gate", "agent", "principal", "company"])("leaves scheduled governed response authority with its owner after %s changes", async kind => {
const fixture = await seedRecoverableReviewResponse();
if (kind === "gate") await db.update(issueThreadInteractions).set({ status: "accepted" }).where(eq(issueThreadInteractions.id, fixture.gate.id));
if (kind === "agent") await db.update(agents).set({ status: "paused" }).where(eq(agents.id, fixture.agentId));
if (kind === "principal") await db.update(companyMemberships).set({ status: "suspended" })
.where(and(eq(companyMemberships.companyId, fixture.companyId), eq(companyMemberships.principalId, fixture.userId)));
expect(await recoverCommittedNativeReviewResponse(db, { ...fixture.input,
...(kind === "company" ? { companyId: randomUUID() } : {}) })).toBe(false);
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).toHaveLength(0);
});
it.each(["milestone", "finalization"] as const)("ignores repeated unscoped routine %s hints without reconstructing dormant governed responses", async source => {
const fixture = await seedRecoverableReviewResponse();
await db.update(companyMemberships).set({ status: "suspended" }).where(and(
eq(companyMemberships.companyId, fixture.companyId), eq(companyMemberships.principalId, fixture.userId)));
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => true);
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 })
.toEqual({ active: 0, next: null });
const debug = db.$client.options.debug;
const queries: string[] = [];
db.$client.options.debug = (_connection, query) => {
if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query);
};
try {
for (let i = 0; i < 3; i++) {
await db.transaction(tx => source === "milestone" ? notifyChatRunMilestones(tx, { companyId: fixture.companyId })
: notifyNativeFinalizationWork(tx, { companyId: fixture.companyId }));
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 })
.toEqual({ active: 0, next: null });
}
expect(queries).toEqual([]);
} finally { db.$client.options.debug = debug; }
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).toHaveLength(0);
expect(errors).toEqual([]);
} finally { await coordinator.stop(); }
});
it("wakes dormant governed responses through the real company membership restoration owner", async () => {
const fixture = await seedRecoverableReviewResponse();
const [member] = await db.select().from(companyMemberships).where(and(
eq(companyMemberships.companyId, fixture.companyId), eq(companyMemberships.principalId, fixture.userId)));
await db.update(companyMemberships).set({ status: "suspended" }).where(eq(companyMemberships.id, member.id));
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => true);
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 })
.toEqual({ active: 0, next: null });
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).toHaveLength(0);
await accessService(db).updateMember(fixture.companyId, member.id, { status: "active" });
await expect.poll(async () => ({ count: (await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).length,
active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 })
.toEqual({ count: 1, active: 0, next: null });
expect(errors).toEqual([]);
} finally { await coordinator.stop(); }
});
it.each(["approve", "reject", "cancel", "unlink"] as const)("reactivates dormant governed responses after approval %s", async action => {
const fixture = await seedRecoverableReviewResponse();
const [approval] = await db.insert(approvals).values({ companyId: fixture.companyId,
type: "hire_agent", status: "pending", payload: {} }).returning();
await issueApprovalService(db).link(fixture.issueId, approval.id);
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => true);
const comments = () => db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId));
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await comments()).toHaveLength(0);
if (action === "unlink") await issueApprovalService(db).unlink(fixture.issueId, approval.id);
else if (action === "cancel") await approvalRecords(db).cancel(approval.id);
else await approvalRecords(db).resolveApproval(approval.id, action === "approve" ? "approved" : "rejected", fixture.userId, null);
await expect.poll(async () => (await comments()).length, { timeout: 5_000 }).toBe(1);
expect(errors).toEqual([]);
} finally { await coordinator.stop(); }
});
it.each([
["resolver", "single"], ["resolver", "bulk"],
["linker", "single"], ["linker", "bulk"],
] as const)("retains a settled task wake when %s starts first during concurrent %s approval linking", async (first, kind) => {
const fixture = await seedWaitTurn();
const [approval] = await db.insert(approvals).values({ companyId: fixture.companyId,
type: "request_board_approval", status: "pending", payload: {} }).returning();
const observed: string[] = [];
const reads: Promise<void>[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeReviewResponse, (event, hints) => {
if (event !== "settled" || ![...(hints ?? [])].some(hint => {
const scope = JSON.parse(hint);
return scope.companyId === fixture.companyId && scope.issueId === fixture.issueId;
})) return;
reads.push(db.select({ status: approvals.status }).from(issueApprovals)
.innerJoin(approvals, eq(issueApprovals.approvalId, approvals.id))
.where(and(eq(issueApprovals.issueId, fixture.issueId), eq(approvals.id, approval.id)))
.then(rows => { for (const row of rows) observed.push(row.status); }));
});
let release!: () => void, ready!: () => void;
const hold = new Promise<void>(resolve => { release = resolve; });
const held = new Promise<void>(resolve => { ready = resolve; });
let holderPid = 0, resolverPid = 0, linkerPid = 0;
let linked = false, resolved = false;
const link = async (tx: typeof db) => {
if (kind === "single") await issueApprovalService(tx).link(fixture.issueId, approval.id);
else await issueApprovalService(tx).linkManyForApproval(approval.id, [fixture.issueId]);
};
const holder = db.transaction(async tx => {
const [backend] = await tx.execute(sql`select pg_backend_pid() as pid`);
holderPid = Number(backend.pid);
if (first === "resolver") {
await tx.execute(sql`lock table approvals in share mode`);
} else {
await link(tx as unknown as typeof db);
}
ready();
await hold;
});
let resolver: Promise<unknown> | undefined, linker: Promise<unknown> | undefined;
const blocked = async (pid: number, by: number) => {
if (!pid || !by) return false;
const [row] = await db.execute(sql`select ${by} = any(pg_blocking_pids(${pid})) as blocked`);
return row.blocked === true;
};
try {
await held;
resolver = db.transaction(async tx => {
const [backend] = await tx.execute(sql`select pg_backend_pid() as pid`);
resolverPid = Number(backend.pid);
await approvalRecords(tx as unknown as typeof db).resolveApproval(approval.id, "approved", fixture.userId, null);
}).then(() => { resolved = true; });
await expect.poll(async () => resolved || await blocked(resolverPid, holderPid), { timeout: 5000 }).toBe(true);
if (first === "resolver") {
linker = db.transaction(async tx => {
const [backend] = await tx.execute(sql`select pg_backend_pid() as pid`);
linkerPid = Number(backend.pid);
await link(tx as unknown as typeof db);
}).then(() => { linked = true; });
await expect.poll(async () => linked || await blocked(linkerPid, resolverPid), { timeout: 5000 }).toBe(true);
}
await Promise.all(reads);
release();
await Promise.all([holder, resolver, linker]);
await Promise.all(reads);
expect(observed).toContain("approved");
} finally {
release();
await Promise.allSettled([holder, resolver, linker]);
await Promise.all(reads);
unsubscribe();
}
});
it.each(["cancel", "unlink"] as const)("keeps governed response approval %s behind outer commit and scoped to linked tasks", async action => {
const fixture = await seedRecoverableReviewResponse();
const [approval, unrelated] = await db.insert(approvals).values([
{ companyId: fixture.companyId, type: "hire_agent", status: "pending", payload: {} },
{ companyId: fixture.companyId, type: "hire_agent", status: "pending", payload: {} },
]).returning();
await issueApprovalService(db).link(fixture.issueId, approval.id);
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => true);
const comments = () => db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId));
const mutate = async (tx: typeof db) => {
if (action === "cancel") await approvalRecords(tx).cancel(approval.id);
else await issueApprovalService(tx).unlink(fixture.issueId, approval.id);
};
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
const previousDebug = db.$client.options.debug;
const queries: string[] = [];
db.$client.options.debug = (_connection, query) => { if (query.includes('"native_run_finalizations"')) queries.push(query); };
try {
await approvalRecords(db).cancel(unrelated.id);
await expect.poll(() => idleWorkSnapshot().active).toBe(0);
expect(queries).toEqual([]);
} finally { db.$client.options.debug = previousDebug; }
const rollback = new Error("rollback governed approval change");
await expect(db.transaction(async tx => {
await mutate(tx as unknown as typeof db);
expect(await comments()).toHaveLength(0);
throw rollback;
})).rejects.toBe(rollback);
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await comments()).toHaveLength(0);
await db.transaction(async tx => {
await mutate(tx as unknown as typeof db);
expect(await comments()).toHaveLength(0);
});
await expect.poll(async () => (await comments()).length, { timeout: 5_000 }).toBe(1);
expect(errors).toEqual([]);
} finally { await coordinator.stop(); }
});
it("wakes dormant governed response recovery after real agent resume", async () => {
const fixture = await seedRecoverableReviewResponse();
await db.update(agents).set({ status: "paused", lifecycleState: "paused", lifecycleHolds: ["manual"], lifecycleOperation: null })
.where(eq(agents.id, fixture.agentId));
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => true);
const lifecycle = configureAgentLifecycle(db, {
requiredPluginIds: async () => [], runHost: async () => "complete", runPlugin: async () => "complete",
});
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).toHaveLength(0);
await createAgentLifecycle(db).resumeAgent(fixture.agentId);
await processAgentLifecycle(db, fixture.agentId);
await expect.poll(async () => (await db.select().from(agents).where(eq(agents.id, fixture.agentId)))[0].status).toBe("idle");
await expect.poll(async () => (await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).length,
{ timeout: 5_000 }).toBe(1);
expect(errors).toEqual([]);
} finally { await lifecycle.stop(); await coordinator.stop(); }
});
it.each(["skip", "withdraw", "reject_suggestions", "complete_verdicts", "decline_connection"] as const)(
"reactivates governed responses after competing interaction %s", async action => {
const fixture = await seedRecoverableReviewResponse();
const service = issueThreadInteractionService(db);
const issue = { id: fixture.issueId, companyId: fixture.companyId };
const actor = { userId: fixture.userId };
const competing = action === "decline_connection"
? await service.createConnectionIntent(issue, { sourceRunId: fixture.runId,
addresseeUserId: fixture.userId, idempotencyKey: `review-competing:${randomUUID()}`,
payload: { version: 1, serviceSlug: "notion", serviceName: "Notion", serviceLogoUrl: null,
requestingAgentId: fixture.agentId, requestingAgentName: "Agent", phase: "requested" } })
: action === "complete_verdicts"
? await service.create(issue, { kind: "request_item_verdicts", payload: { version: 1,
prompt: "Review this item", items: [{ id: "one", label: "One" }] } }, actor)
: await service.create(issue, { kind: "suggest_tasks", payload: { version: 1,
tasks: [{ clientKey: "child", title: "A suggested task" }] } }, actor);
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => true);
const comments = () => db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId));
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await comments()).toHaveLength(0);
if (action === "skip") await service.skipInteraction(issue, competing.id, {}, actor);
else if (action === "withdraw") await service.withdrawInteraction(issue, competing.id, {}, actor);
else if (action === "reject_suggestions") await service.rejectInteraction(issue, competing.id, {}, actor);
else if (action === "complete_verdicts") await service.submitItemVerdicts(issue, competing.id,
{ verdicts: [{ id: "one", verdict: "approve" }] }, actor);
else await service.resolveConnectionIntent(issue, competing.id, { version: 1, outcome: "declined" }, actor);
await expect.poll(async () => (await comments()).length, { timeout: 5_000 }).toBe(1);
expect(errors).toEqual([]);
} finally { await coordinator.stop(); }
});
it.each(["status", "assignment"] as const)("preserves governed response policy through task %s changes", async change => {
const fixture = await seedRecoverableReviewResponse();
await db.update(issues).set(change === "status" ? { status: "blocked" } : { assigneeAgentId: null })
.where(eq(issues.id, fixture.issueId));
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => true);
const comments = () => db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId));
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await comments()).toHaveLength(0);
if (change === "assignment") {
await expect(issueService(db).update(fixture.issueId, { assigneeAgentId: fixture.agentId }))
.rejects.toThrow("Agent assignment cannot change while this task is bound to an external channel");
expect(await comments()).toHaveLength(0);
} else {
await issueService(db).update(fixture.issueId, { status: "in_review" });
await expect.poll(async () => (await comments()).length, { timeout: 5_000 }).toBe(1);
}
expect(errors).toEqual([]);
} finally { await coordinator.stop(); }
});
it("reactivates governed responses after OAuth confirmation denial", async () => {
const fixture = await seedRecoverableReviewResponse();
const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.companyId, fixture.companyId));
const competing = await issueThreadInteractionService(db).create({ id: fixture.issueId, companyId: fixture.companyId },
{ kind: "request_confirmation", payload: { version: 1, prompt: "Connect your account", target: {
type: "custom", key: `connection:${connection.id}`, revisionId: randomUUID(), label: "Connect account" } } },
{ userId: fixture.userId });
const [state] = await db.insert(toolOauthStates).values({ state: randomUUID(), companyId: fixture.companyId,
connectionId: connection.id, codeVerifier: "test-verifier", issueId: fixture.issueId, interactionId: competing.id,
subjectUserId: fixture.userId, expiresAt: new Date(Date.now() + 60_000) }).returning();
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => true);
const comments = () => db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId));
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await comments()).toHaveLength(0);
const rollback = new Error("rollback OAuth denial");
await expect(db.transaction(async tx => {
await createOAuthStateStore(tx as unknown as typeof db).rejectPendingOAuthInteraction(state,
{ actorType: "user", actorId: fixture.userId });
expect(await comments()).toHaveLength(0);
throw rollback;
})).rejects.toBe(rollback);
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, competing.id)))[0].status).toBe("pending");
await db.transaction(async tx => {
await createOAuthStateStore(tx as unknown as typeof db).rejectPendingOAuthInteraction(state,
{ actorType: "user", actorId: fixture.userId });
expect(await comments()).toHaveLength(0);
});
await expect.poll(async () => (await comments()).length, { timeout: 5_000 }).toBe(1);
expect(errors).toEqual([]);
} finally { await coordinator.stop(); }
});
it("reactivates governed responses when pending execution policy is cleared without changing task status", async () => {
const fixture = await seedRecoverableReviewResponse();
await db.update(issues).set({ executionState: { status: "pending", currentStageId: randomUUID(),
currentStageIndex: 0, currentStageType: "review", currentParticipant: { type: "user", userId: fixture.userId, agentId: null },
returnAssignee: { type: "agent", agentId: fixture.agentId, userId: null }, reviewRequest: null,
completedStageIds: [], lastDecisionId: null, lastDecisionOutcome: null, monitor: null, changesRequestedCount: 0 } })
.where(eq(issues.id, fixture.issueId));
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => true);
const comments = () => db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId));
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await comments()).toHaveLength(0);
await issueService(db).updateForCompany(fixture.issueId, fixture.companyId, { executionState: null });
expect((await db.select().from(issues).where(eq(issues.id, fixture.issueId)))[0].status).toBe("in_review");
await expect.poll(async () => (await comments()).length, { timeout: 5_000 }).toBe(1);
expect(errors).toEqual([]);
} finally { await coordinator.stop(); }
});
it("wakes governed responses from the real coordinator retry release", async () => {
const fixture = await seedRecoverableReviewResponse();
const store = new NativeRunCoordinatorStore(db, fixture.storeBinding);
await store.claimFinalization({ leaseOwner: "review-response-fixture", leaseTtlMs: 30_000 });
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => true);
const comments = () => db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId));
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await comments()).toHaveLength(0);
await store.markFinalizationRetry({ leaseOwner: "review-response-fixture", failureCode: "review_response_retry", retryAfterMs: 1000 });
expect(await comments()).toHaveLength(0);
await expect.poll(async () => (await comments()).length, { timeout: 5_000 }).toBe(1);
expect(errors).toEqual([]);
} finally { await coordinator.stop(); }
});
it("retains the first governed-response wake through idle drain without passive SQL", async () => {
const fixture = await seedRecoverableReviewResponse();
const [approval] = await db.insert(approvals).values({ companyId: fixture.companyId,
type: "hire_agent", status: "pending", payload: {} }).returning();
await issueApprovalService(db).link(fixture.issueId, approval.id);
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
let canRun = true;
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun,
canReconcile: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => canRun);
const comments = () => db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId));
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
canRun = false;
await approvalRecords(db).cancel(approval.id);
await expect.poll(() => idleWorkSnapshot().active).toBe(0);
expect(await comments()).toHaveLength(0);
expect(coordinator.nextWakeAt()).not.toBeNull();
const previousDebug = db.$client.options.debug;
const queries: string[] = [];
db.$client.options.debug = (_connection, query) => { queries.push(query); };
try { await vi.advanceTimersByTimeAsync(120_000); expect(queries).toEqual([]); }
finally { db.$client.options.debug = previousDebug; }
canRun = true; worker.wake();
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(await comments()).toHaveLength(1);
expect(idleWorkSnapshot().active).toBe(0);
expect(coordinator.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
expect(errors).toEqual([]);
} finally { await coordinator.stop(); vi.useRealTimers(); }
});
it("retains failed governed-response recovery across drain and retries after resume", async () => {
const fixture = await seedRecoverableReviewResponse();
let release!: () => void, observed!: () => void;
const hold = new Promise<void>(resolve => { release = resolve; });
const locked = new Promise<void>(resolve => { observed = resolve; });
const holder = db.transaction(async tx => {
await tx.select().from(chatEndpoints).where(eq(chatEndpoints.id, fixture.endpointId)).for("update");
observed(); await hold;
});
await locked;
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
let canRun = true;
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun,
canReconcile: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => canRun);
const comments = () => db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId));
try {
await worker.ready;
await expect.poll(() => ({ errors: errors.length, active: idleWorkSnapshot().active }),
{ timeout: 10_000 }).toEqual({ errors: 1, active: 0 });
expect(await comments()).toHaveLength(0);
expect(coordinator.nextWakeAt()).toBeGreaterThan(Date.now());
canRun = false;
worker.wake();
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
const previousDebug = db.$client.options.debug;
const queries: string[] = [];
db.$client.options.debug = (_connection, query) => { queries.push(query); };
try { await vi.advanceTimersByTimeAsync(120_000); expect(queries).toEqual([]); }
finally { db.$client.options.debug = previousDebug; }
release(); await holder;
canRun = true; worker.wake();
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(await comments()).toHaveLength(1);
expect(idleWorkSnapshot().active).toBe(0);
expect(coordinator.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
expect(errors).toHaveLength(1);
} finally { release(); await holder; await coordinator.stop(); vi.useRealTimers(); }
});
it("reconstructs the governed-response deadline after shutdown without early presentation", async () => {
const fixture = await seedRecoverableReviewResponse();
const store = new NativeRunCoordinatorStore(db, fixture.storeBinding);
await store.claimFinalization({ leaseOwner: "review-future", leaseTtlMs: 30_000 });
await store.markFinalizationRetry({ leaseOwner: "review-future", failureCode: "review_fixture", retryAfterMs: 60_000 });
const [state] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
const at = state.nextAttemptAt!.getTime();
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => true);
const comments = () => db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId));
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: at });
await coordinator.stop();
expect(coordinator.nextWakeAt()).toBeNull();
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
const previousDebug = db.$client.options.debug;
const queries: string[] = [];
db.$client.options.debug = (_connection, query) => { queries.push(query); };
try { await vi.advanceTimersByTimeAsync(30_000); expect(queries).toEqual([]); }
finally { db.$client.options.debug = previousDebug; }
const resumed = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
try {
const restarted = nativeReviewResponseWork(db).register(resumed, () => true);
await restarted.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: resumed.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: at });
expect(await comments()).toHaveLength(0);
await vi.advanceTimersByTimeAsync(Math.max(0, at - Date.now()));
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(await comments()).toHaveLength(1);
expect(idleWorkSnapshot().active).toBe(0);
expect(resumed.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
expect(errors).toEqual([]);
} finally { await resumed.stop(); vi.useRealTimers(); }
});
it("reconstructs scheduled governed responses at startup and leaves their history quiet", async () => {
const fixture = await seedRecoverableReviewResponse();
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeReviewResponseWork(db).register(coordinator, () => true);
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, fixture.runId))).toHaveLength(1);
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
const previousDebug = db.$client.options.debug;
const queries: string[] = [];
db.$client.options.debug = (_connection, query) => { queries.push(query); };
try { await vi.advanceTimersByTimeAsync(120_000); expect(queries).toEqual([]); }
finally { db.$client.options.debug = previousDebug; }
expect(errors).toEqual([]);
} finally { await coordinator.stop(); vi.useRealTimers(); }
});
it.each(["telegram", "discord"] as const)(
"presents the exact %s response without resolving a pre-existing native completion review",
async (provider) => {
@@ -500,6 +500,7 @@ export async function restoreNativeChatReviewPresentationInTransaction(
decisionId: string;
resultId: string;
assessmentId: string;
onlyWhenDue?: boolean;
},
): Promise<boolean> {
const [coordinator] = await db
@@ -519,6 +520,8 @@ export async function restoreNativeChatReviewPresentationInTransaction(
.for("update")
.limit(1);
if (!coordinator) return false;
if (input.onlyWhenDue && (coordinator.leaseOwner !== null
|| coordinator.nextAttemptAt !== null && coordinator.nextAttemptAt.getTime() > Date.now())) return false;
const [decision] = await db
.select()
.from(statusDecisions)
@@ -645,6 +645,7 @@ type NativeReconciliationOptions = {
skipUnsafeWorkspaceExports?: boolean;
skipNativePendingEffects?: boolean;
skipNativeResponsePresentation?: boolean;
skipNativeReviewResponse?: boolean;
pluginWorkerManager?: WorkspaceWorkers;
environmentRuntime?: EnvironmentRuntimeService;
onWorkspaceSettled?: (input: {
@@ -875,7 +876,8 @@ async function reconcileNativeFinalizationsPass(
row.decisionId &&
row.resultId &&
row.assessmentId &&
currentDecisionJson.externalChatReviewPresentation
currentDecisionJson.externalChatReviewPresentation &&
!options.skipNativeReviewResponse
) {
// A later chat turn may retain the same pending review. Recover the
// earlier response independently before status reconciliation skips
@@ -1125,6 +1127,7 @@ async function reconcileNativeFinalizationsPass(
workspaceFinalizeStatus,
projectRunStatus: true,
skipNativeResponsePresentation: options.skipNativeResponsePresentation,
skipNativeReviewResponse: options.skipNativeReviewResponse,
preserveProviderAttempt: Boolean(
readNativeWorkspaceSyncReference(
row.nativeWorkspaceSync,
@@ -1184,6 +1187,7 @@ async function reconcileNativeFinalizationsPass(
workspaceFinalizeStatus: barrier.status as "succeeded" | "failed",
projectRunStatus: true,
skipNativeResponsePresentation: options.skipNativeResponsePresentation,
skipNativeReviewResponse: options.skipNativeReviewResponse,
preserveProviderAttempt: Boolean(
readNativeWorkspaceSyncReference(
row.nativeWorkspaceSync,
@@ -0,0 +1,88 @@
import { and, asc, eq, gt, isNotNull, isNull, notExists, sql } from "drizzle-orm";
import { heartbeatRuns, issueComments, nativeRunFinalizations, statusDecisions, type Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
import type { NativeReviewResponseScope } from "../native-review-response-notifications.js";
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
import { recoverCommittedNativeReviewResponse } from "./native-run-finalizer.js";
export function registerNativeReviewResponseWork(db: Db,
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
return nativeReviewResponseWork(db).register(coordinator, canRun);
}
type Entry = { id: string; companyId: string; issueId: string; agentId: string; dueAt: number };
export function nativeReviewResponseWork(db: Db) {
function candidates(scope?: NativeReviewResponseScope, after?: string, runId?: string) {
return db.select({ id: nativeRunFinalizations.runId, companyId: nativeRunFinalizations.companyId,
issueId: nativeRunFinalizations.issueId, agentId: heartbeatRuns.agentId,
decisionId: nativeRunFinalizations.decisionId, resultId: nativeRunFinalizations.resultId,
assessmentId: nativeRunFinalizations.assessmentId, nextAttemptAt: nativeRunFinalizations.nextAttemptAt,
}).from(nativeRunFinalizations)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.nativeIssueId, nativeRunFinalizations.issueId)))
.innerJoin(statusDecisions, and(eq(statusDecisions.id, nativeRunFinalizations.decisionId),
eq(statusDecisions.companyId, nativeRunFinalizations.companyId), eq(statusDecisions.issueId, nativeRunFinalizations.issueId),
eq(statusDecisions.runId, nativeRunFinalizations.runId), eq(statusDecisions.assessmentId, nativeRunFinalizations.assessmentId)))
.where(and(eq(nativeRunFinalizations.phase, "committed"), isNull(nativeRunFinalizations.leaseOwner),
isNotNull(nativeRunFinalizations.resultId), isNotNull(nativeRunFinalizations.assessmentId),
eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "succeeded"),
eq(statusDecisions.reasonCode, "governed_response_waiting"), eq(statusDecisions.applicationState, "applied"),
sql`${statusDecisions.decisionJson}->'externalChatReviewPresentation'->>'schema' = 'paperclip.chat_review_response_presentation.v1'`,
notExists(db.select({ id: issueComments.id }).from(issueComments).where(and(
eq(issueComments.companyId, nativeRunFinalizations.companyId), eq(issueComments.issueId, nativeRunFinalizations.issueId),
eq(issueComments.createdByRunId, nativeRunFinalizations.runId), eq(issueComments.authorAgentId, heartbeatRuns.agentId),
eq(issueComments.authorType, "agent"), sql`${issueComments.metadata}->>'authorizationReason' = 'allow_chat_run_presentation'`,
sql`${heartbeatRuns.resultJson}->'externalChatReviewPresentation'->>'decisionId' = ${nativeRunFinalizations.decisionId}::text`,
))),
scope ? eq(nativeRunFinalizations.companyId, scope.companyId) : undefined,
scope?.issueId ? eq(nativeRunFinalizations.issueId, scope.issueId) : undefined,
scope?.runId ? eq(nativeRunFinalizations.runId, scope.runId) : undefined,
scope?.agentId ? eq(heartbeatRuns.agentId, scope.agentId) : undefined,
runId ? eq(nativeRunFinalizations.runId, runId) : undefined,
after ? gt(nativeRunFinalizations.runId, after) : undefined,
)).orderBy(asc(nativeRunFinalizations.runId)).limit(100);
}
async function load(scope?: NativeReviewResponseScope, signal?: AbortSignal, canRun = () => true): Promise<Entry[]> {
const entries: Entry[] = [];
let after: string | undefined;
while (canRun() && !signal?.aborted) {
const rows = await candidates(scope, after);
for (const row of rows) entries.push({ id: row.id, companyId: row.companyId,
issueId: row.issueId, agentId: row.agentId, dueAt: row.nextAttemptAt?.getTime() ?? 0 });
if (rows.length < 100) return entries;
after = rows[rows.length - 1].id;
}
throw new Error("Governed review response discovery stopped");
}
async function check(entry: Entry, signal?: AbortSignal): Promise<Entry | null> {
if (signal?.aborted) throw new Error("Governed review response recovery stopped");
const scope = { companyId: entry.companyId, issueId: entry.issueId };
const [current] = await candidates(scope, undefined, entry.id);
if (!current) return null;
if (current.nextAttemptAt && current.nextAttemptAt.getTime() > Date.now()) return { ...entry, dueAt: current.nextAttemptAt.getTime() };
if (signal?.aborted) throw new Error("Governed review response recovery stopped");
const recoveryStartedAt = Date.now();
await recoverCommittedNativeReviewResponse(db, { ...scope, runId: entry.id,
decisionId: current.decisionId!, resultId: current.resultId!, assessmentId: current.assessmentId!, onlyWhenDue: true });
const [remaining] = await candidates(scope, undefined, entry.id);
return remaining?.nextAttemptAt && remaining.nextAttemptAt.getTime() > recoveryStartedAt
? { ...entry, dueAt: remaining.nextAttemptAt.getTime() } : null;
}
function register(coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
return registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.nativeReviewResponse,
decodeScope: (hint: string): NativeReviewResponseScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (entry: Entry, scope?: NativeReviewResponseScope) => !scope || entry.companyId === scope.companyId
&& (!scope.issueId || entry.issueId === scope.issueId) && (!scope.runId || entry.id === scope.runId)
&& (!scope.agentId || entry.agentId === scope.agentId),
load: (scope, signal) => load(scope, signal, canRun),
check: (entry, signal) => {
if (!canRun()) throw new Error("Governed review response recovery paused");
return check(entry, signal);
},
}, canRun);
}
return { load, check, register };
}
@@ -723,65 +723,62 @@ async function projectCommittedRun(input: {
}
}
async function materializeCommittedReviewResponse(db: Db, runId: string) {
try {
const [run] = await db
.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
nativeIssueId: heartbeatRuns.nativeIssueId, resultJson: heartbeatRuns.resultJson })
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, runId))
.limit(1);
if (!run?.nativeIssueId || !run.resultJson?.externalChatReviewPresentation)
return;
const decisionId = record(
run.resultJson.externalChatReviewPresentation,
).decisionId;
if (
typeof decisionId !== "string" ||
(await hasMaterializedNativeReviewResponse(db, {
companyId: run.companyId,
issueId: run.nativeIssueId,
runId,
decisionId,
}))
)
return;
const summary = record(run.resultJson.nativeResult).summary;
if (
typeof summary !== "string" ||
!(await authorizeNativeChatReviewPresentation(db, {
companyId: run.companyId,
issueId: run.nativeIssueId,
runId,
resultJson: run.resultJson,
}))
)
return;
// The coordinator transaction is already committed. addComment owns a
// fresh issue transaction, rechecks the proof and carries selected files;
// its same-run/text and publication keys make recovery replay idempotent.
await issueService(db).addComment(
run.nativeIssueId,
summary,
{ agentId: run.agentId, runId },
{
authorizationReason: "allow_chat_run_presentation",
},
);
publishChatPublicationCommitSignal({
async function materializeNativeReviewResponse(db: Db, runId: string, dbOrTx: Db = db) {
const [run] = await dbOrTx
.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
nativeIssueId: heartbeatRuns.nativeIssueId, resultJson: heartbeatRuns.resultJson })
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, runId))
.limit(1);
if (!run?.nativeIssueId || !run.resultJson?.externalChatReviewPresentation)
return null;
const decisionId = record(
run.resultJson.externalChatReviewPresentation,
).decisionId;
if (
typeof decisionId !== "string" ||
(await hasMaterializedNativeReviewResponse(dbOrTx, {
companyId: run.companyId,
issueId: run.nativeIssueId,
runId: run.id,
agentId: run.agentId,
eventType: "run.presentation.resolved",
});
runId,
decisionId,
}))
)
return null;
const summary = record(run.resultJson.nativeResult).summary;
if (
typeof summary !== "string" ||
!(await authorizeNativeChatReviewPresentation(dbOrTx, {
companyId: run.companyId,
issueId: run.nativeIssueId,
runId,
resultJson: run.resultJson,
}))
)
return null;
await issueService(db).addComment(
run.nativeIssueId,
summary,
{ agentId: run.agentId, runId },
{
authorizationReason: "allow_chat_run_presentation",
},
dbOrTx,
);
return run;
}
function publishNativeReviewResponse(run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "agentId" | "nativeIssueId">) {
publishChatPublicationCommitSignal({ companyId: run.companyId, issueId: run.nativeIssueId!,
runId: run.id, agentId: run.agentId, eventType: "run.presentation.resolved" });
}
async function materializeCommittedReviewResponse(db: Db, runId: string) {
try {
const run = await materializeNativeReviewResponse(db, runId);
if (run) publishNativeReviewResponse(run);
} catch (error) {
// The durable committed proof remains retryable by the next reconciler
// sweep. Presentation failure must not reclassify a successful native run.
logger.warn(
{ runId, error },
"Committed chat review response is awaiting presentation retry",
);
logger.warn({ runId, error }, "Committed chat review response is awaiting presentation retry");
}
}
@@ -1183,6 +1180,28 @@ export async function repairCommittedNativeReviewResponse(
);
if (restored) await materializeCommittedReviewResponse(db, input.runId);
}
export async function recoverCommittedNativeReviewResponse(db: Db, input: {
companyId: string; issueId: string; runId: string; decisionId: string;
resultId: string; assessmentId: string; onlyWhenDue?: boolean;
}): Promise<boolean> {
const materialized = await db.transaction(async transaction => {
const tx = transaction as unknown as Db;
const [coordinator] = await tx.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.where(and(eq(nativeRunFinalizations.companyId, input.companyId),
eq(nativeRunFinalizations.issueId, input.issueId), eq(nativeRunFinalizations.runId, input.runId))).for("update");
if (!coordinator) return null;
const [issue] = await tx.select({ id: issues.id }).from(issues)
.where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update");
if (!issue || await hasMaterializedNativeReviewResponse(tx, input)) return null;
if (!await restoreNativeChatReviewPresentationInTransaction(tx, input)) return null;
return materializeNativeReviewResponse(db, input.runId, tx);
});
if (materialized) publishNativeReviewResponse(materialized);
return materialized !== null;
}
export async function finalizeNativeRun(input: {
db: Db;
runId: string;
@@ -1190,6 +1209,7 @@ export async function finalizeNativeRun(input: {
/** Reconciliation owns terminal run projection; the live heartbeat does it afterward. */
projectRunStatus?: boolean;
skipNativeResponsePresentation?: boolean;
skipNativeReviewResponse?: boolean;
/** Workspace-only replay must not consume the provider recovery budget. */
preserveProviderAttempt?: boolean;
failpoint?: NativeStatusCommitFailpoint;
@@ -1214,14 +1234,14 @@ export async function finalizeNativeRun(input: {
const coordinator = claim.coordinator;
if (!claim.leaseOwner && coordinator.phase === "committed") {
if (isNativeRunnerOwnershipHeld(admittedRun)) return coordinator;
const presentationAlreadyMaterialized = coordinator.decisionId
const presentationAlreadyMaterialized = input.skipNativeReviewResponse || (coordinator.decisionId
? await hasMaterializedNativeReviewResponse(input.db, {
companyId: admittedRun.companyId,
issueId: coordinator.issueId,
runId: admittedRun.id,
decisionId: coordinator.decisionId,
})
: false;
: false);
if (
!presentationAlreadyMaterialized &&
coordinator.decisionId &&
@@ -1,3 +1,4 @@
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js";
import { enqueueTerminalIssueInteractionChatPublications } from "./chat-interaction-publications.js";
import { connectionManagementService } from "./connection-management.js";
@@ -155,6 +156,7 @@ export function satisfiedConnectionIntentService(db: Db) {
}
const connection = await findSatisfiedToolConnection(txDb, task, intent);
if (!connection) return false;
await notifyNativeReviewResponse(tx, candidate);
const now = new Date();
// Retire the obsolete request without inventing a human approval.
await notifyChatConfirmationResolution(tx, intent);
@@ -1,3 +1,4 @@
import { notifyNativeReviewResponse } from "../../native-review-response-notifications.js";
import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js";
import { managedConnectorProfile } from "./client-settings.js";
import type { ActorInfo } from "../types.js";
@@ -408,18 +409,19 @@ export function createCloudOAuthCallback(db: Db, dependencies: Dependencies) {
personalCredential || agentCredential ? credentialSecretRefs : [],
tx,
);
const linkedInteractionKind = stateRow.interactionId
const linkedInteraction = stateRow.interactionId
? await tx
.select({ kind: issueThreadInteractions.kind })
.select({ kind: issueThreadInteractions.kind, issueId: issueThreadInteractions.issueId })
.from(issueThreadInteractions)
.where(eq(issueThreadInteractions.id, stateRow.interactionId))
.limit(1)
.then((rows) => rows[0]?.kind ?? null)
.then((rows) => rows[0] ?? null)
: null;
if (
stateRow.interactionId &&
linkedInteractionKind === "request_confirmation"
linkedInteraction?.kind === "request_confirmation"
) {
await notifyNativeReviewResponse(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId });
await tx
.update(issueThreadInteractions)
.set({
@@ -1,3 +1,4 @@
import { notifyNativeReviewResponse } from "../../native-review-response-notifications.js";
import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js";
import { credentialScope } from "./client-settings.js";
import { withoutOAuthRefreshLease } from "./refresh-leases.js";
@@ -374,18 +375,19 @@ export function createDirectOAuthCallback(db: Db, dependencies: Dependencies) {
updatedAt: new Date(),
})
.where(eq(toolApplications.id, connection.applicationId));
const linkedInteractionKind = stateRow.interactionId
const linkedInteraction = stateRow.interactionId
? await tx
.select({ kind: issueThreadInteractions.kind })
.select({ kind: issueThreadInteractions.kind, issueId: issueThreadInteractions.issueId })
.from(issueThreadInteractions)
.where(eq(issueThreadInteractions.id, stateRow.interactionId))
.limit(1)
.then((rows) => rows[0]?.kind ?? null)
.then((rows) => rows[0] ?? null)
: null;
if (
stateRow.interactionId &&
linkedInteractionKind === "request_confirmation"
linkedInteraction?.kind === "request_confirmation"
) {
await notifyNativeReviewResponse(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId });
await tx
.update(issueThreadInteractions)
.set({
+33 -26
View File
@@ -1,3 +1,4 @@
import { notifyNativeReviewResponse } from "../../native-review-response-notifications.js";
import type { ActorInfo } from "../types.js";
import { and, eq } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
@@ -124,17 +125,19 @@ async function rejectPendingOAuthInteraction(
stateRow: typeof toolOauthStates.$inferSelect,
actor: ActorInfo | undefined,
) {
if (!stateRow.interactionId) return;
const interactionId = stateRow.interactionId;
if (!interactionId) return;
const now = new Date();
const linked = await db
.select({
kind: issueThreadInteractions.kind,
issueId: issueThreadInteractions.issueId,
payload: issueThreadInteractions.payload,
})
.from(issueThreadInteractions)
.where(
and(
eq(issueThreadInteractions.id, stateRow.interactionId),
eq(issueThreadInteractions.id, interactionId),
eq(issueThreadInteractions.companyId, stateRow.companyId),
),
)
@@ -152,36 +155,40 @@ async function rejectPendingOAuthInteraction(
})
.where(
and(
eq(issueThreadInteractions.id, stateRow.interactionId),
eq(issueThreadInteractions.id, interactionId),
eq(issueThreadInteractions.companyId, stateRow.companyId),
eq(issueThreadInteractions.status, "pending"),
),
);
return;
}
await db
.update(issueThreadInteractions)
.set({
status: "rejected",
result: {
version: 1,
outcome: "rejected",
// Paperclip's own words: the provider's explanation is untrusted and
// this reason is rendered in the thread (PAP-17108).
reason:
"Authorization was declined or cancelled in the provider's window",
},
resolvedByUserId: actor?.actorType === "user" ? actor.actorId : null,
resolvedAt: now,
updatedAt: now,
})
.where(
and(
eq(issueThreadInteractions.id, stateRow.interactionId),
eq(issueThreadInteractions.companyId, stateRow.companyId),
eq(issueThreadInteractions.status, "pending"),
),
);
if (!linked) return;
await db.transaction(async tx => {
await notifyNativeReviewResponse(tx, { companyId: stateRow.companyId, issueId: linked.issueId });
await tx
.update(issueThreadInteractions)
.set({
status: "rejected",
result: {
version: 1,
outcome: "rejected",
reason:
"Authorization was declined or cancelled in the provider's window",
},
resolvedByUserId: actor?.actorType === "user" ? actor.actorId : null,
resolvedAt: now,
updatedAt: now,
})
.where(
and(
eq(issueThreadInteractions.id, interactionId),
eq(issueThreadInteractions.companyId, stateRow.companyId),
eq(issueThreadInteractions.status, "pending"),
),
);
});
}
return { peekOAuthState, validateOAuthState, consumeOAuthState, rejectPendingOAuthInteraction };
}
+1 -1
View File
@@ -2689,7 +2689,7 @@ export function createToolGatewayService(
approvalId: approval.id,
linkedByAgentId: input.session.agentId,
})
.onConflictDoNothing());
.onConflictDoNothing(), { issueIds: [approvalIssueId] });
}
const interaction = await interactions.create(