mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 12:59:13 +01:00
fix(server): wake terminal confirmation recovery from committed changes (#17651)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/202/changes/paperclip/202 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip persists confirmation clicks and interaction decisions before dispatch completes. > - A crash after a decision commits can leave a processing confirmation action. > - Periodic publication maintenance repeatedly scans those actions even when idle. > - Saved decisions and the supported board/admin writers already identify when settlement can change. > - This owner reconstructs durable actions, reconciles real changes and then becomes quiet. ## Linked Issues or Issue Description This existing terminal-confirmation slice follows merged #17649. It removes only terminal-confirmation recovery from periodic chat maintenance while retaining the explicit manual API and existing durable action/interaction/publication rows. **Problem:** An accepted response can commit before its dispatch settles. Removing periodic recovery without covering board decisions, expiry and ownership changes would strand processing actions. Pending human decisions must remain recoverable without polling indefinitely. **Expected behavior:** Committed supported changes wake the scoped owner. It reconciles terminal responses once, parks unresolved human decisions, preserves concurrent commits/restart/drain and becomes quiet when settled. ## What Changed - Register terminal-confirmation work with the existing delivery coordinator and scoped lifecycle. Reconstruct processing action IDs, bound selection pages, preserve concurrent wakes and keep current authorization in the existing interaction owner. - Add the existing transaction-aware confirmation notification topic at real resolution writes and callback settlement. Reuse milestone signals instead of adding per-caller recovery wrappers. - Cover ordinary and checkbox confirmations, suggestions and connection intents through their owning service. Cover board accept/reject, cancellation/withdrawal, opt-in comment supersession, expiry, reassignment and already-satisfied connection resolution. - Preserve task/actor authority, one-use callback tokens and immutable provider publication semantics. Explicitly fail malformed missing bindings instead of silently retaining unrecoverable processing actions. - Remove only this duty from recurring publication maintenance. Add no table or scheduler infrastructure. A concurrent partial index on existing action rows prevents recovery discovery from scanning completed history. ## Verification Published repair head: `a35e78d076c1b618bc643391b7520fa6952fc996`, targeting master after normal integration of `644937c93150c423f12674992c7ea1e8b5dbc98c`. [Detailed acceptance and preserved failures](https://github.com/paperclipai/paperclip/blob/a35e78d076c1b618bc643391b7520fa6952fc996/doc/plans/2026-10-25-terminal-confirmation-sleep-audit.md#bound-retained-history-reads--october-28-review-repair). - **New query-cost regression:** Capture the actual SQL emitted by startup and company-scoped confirmation discovery. Seed 20,000 completed/unrelated action rows and one pending human decision. Before the index, the regression fails because recovery examines 20,009 action rows. After the partial index, startup and scoped discovery must examine at most two action rows and fewer than 20 table/index buffers; the no-pending scoped path must also stay below 20 buffers. Planner settings are unchanged, and returned IDs remain correct. - Match startup keyset ordering to `(company_id, id)`, the partial index order. The index alone still let PostgreSQL choose the primary key and scan 20,206 rows for `ORDER BY id`; the final query fixes this without forcing a planner setting or adding a second index. Fixed kind/status predicates stay literal for prepared-plan index eligibility. - Add the generated `0398_yielding_random.sql` migration, using `CREATE INDEX CONCURRENTLY IF NOT EXISTS` and the existing top-level migration runner. Keep the five newest generated snapshots and preserve every prior SQL/journal entry. No work table is introduced. - **110 distinct cases pass:** 17 final confirmation lifecycle/query-cost cases (1,671 skipped), seven real Runner descriptor variants (295 skipped), and all 86 app/startup/bundled-plugin/plugin-database cases. Server/database typechecking and builds, migration safety/numbering, module boundaries and a fresh PostgreSQL migration through 0398 pass. The final test-only vacuum precondition is followed by another 17-case confirmation pass and server typecheck. Counts are not added across repeated runs. Normal integration of master `644937c93150c423f12674992c7ea1e8b5dbc98c` preserves the complete owned patch and all 18 unrelated master paths. - Previous landed-parent acceptance: 29 distinct cases (16 confirmation lifecycle/producer cases with 1,671 skipped, plus all 13 startup cases) passed on `32e64a7e9ea5b457e54ca7b268be90e1e155b451`; the earlier 80-case matrix remains scoped to `c1e8542b588a4fa59b62f9829541d3daea4f1ddc`. Counts are not added across repeated runs. - Actual producer cases prove parking/quiet, caller-owned outer transaction rollback, commit-driven automatic settlement, restart/drain and concurrent commits. Checkbox, suggestion, connection-intent and Telegram/Slack callback paths retain existing authority and immutable publication semantics. - **Fresh exact-head CI and Greptile 5/5 with no unresolved actionable findings are required.** The preceding head's Runner CI failed its timeout-attachment durable-suspension case (1 failed/1,955 passed/4 skipped); that failure is retained and not counted as a pass. The new head must pass this job. No full local repository, live provider or physical suspension claim is made. The owner audit retains the earlier missing-board-producer regressions and their corrections, including the invalid supersession fixture that assumed a new checkbox expires an older one. Existing behavior was preserved by using the supported opt-in human-comment path. Prior scheduler/backoff, route and plugin test failures remain at their original revisions; no old pass is substituted for current remote gates. ## Risks - Process-local notifications cover supported application writers and their shared pools, not independent SQL writers or other replicas. - Processing actions waiting on a human response are deliberately parked. Supported interaction resolution/expiry writers must use the notification boundary; a missing producer delays settlement until restart. - Invalid bindings have explicit failure evidence. Source, actor, task and publication authority remain checked by their existing owners. - Following continuation/GitHub duties remain separate. Cloud sleep gates and external alarms are unchanged; local worker quiet does not establish physical host/database suspension or fleet savings. ## Model Used OpenAI GPT-6 / Codex, with reasoning, repository inspection, code execution, and PostgreSQL tests. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
60b8089a44
commit
ee67d2ece9
19 files changed
+1192
-39
No files matched your search
@@ -92,6 +92,8 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
chat.indexOf("async function schedulePendingPublications("));
|
||||
expect(publicationMaintenance).not.toContain("await enqueueInboundWakeupPublications(");
|
||||
expect(publicationMaintenance).not.toContain("await enqueueFailedChatRetryPublications(");
|
||||
expect(publicationMaintenance).not.toContain("reconcileTerminalConfirmationActions(");
|
||||
expect(source).toContain("chatChannels.registerTerminalConfirmationWork(deliveryWork,");
|
||||
expect(source).toContain("chatFailedRetryNoticeWork(chatChannels).register(deliveryWork,");
|
||||
});
|
||||
|
||||
|
||||
@@ -41873,6 +41873,99 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
await service.shutdown();
|
||||
});
|
||||
|
||||
it.each([
|
||||
["request_checkbox_confirmation", "accept"],
|
||||
["request_checkbox_confirmation", "reject"],
|
||||
["request_checkbox_confirmation", "withdraw"],
|
||||
["request_checkbox_confirmation", "supersede"],
|
||||
["suggest_tasks", "accept"],
|
||||
["suggest_tasks", "reject"],
|
||||
["connection_intent", "reject"],
|
||||
["connection_intent", "reassign"],
|
||||
] as const)("wakes parked Slack %s actions after board %s commits", async (kind, decision) => {
|
||||
const previousPublicUrl = process.env.PAPERCLIP_PUBLIC_URL;
|
||||
process.env.PAPERCLIP_PUBLIC_URL = "https://paperclip.example";
|
||||
const fixture = await seedCompany();
|
||||
const { callbacks, endpoint, service } = await configuredSlackEndpoint(fixture);
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
try {
|
||||
const thread = makeThread({ channelId: "C-REVIEW", id: "slack:C-REVIEW:4500.1", name: "reviews" });
|
||||
await deliverMessage({ callbacks, endpointId: endpoint.id, thread: thread.thread, message: makeMessage({ id: "4500.1", text: "@maya review this change", mentioned: true, userId: `U-REVIEW-${randomUUID()}` }), trigger: "mention" });
|
||||
await qualifySetupRoundTrip(service, endpoint.id);
|
||||
await service.test(endpoint.id, "owner-user");
|
||||
const [conversation] = await readChatConversationsByEndpointId(endpoint.id);
|
||||
const sourceRunId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId: fixture.companyId, agentId: fixture.assignedAgentId, status: "succeeded", responsibleUserId: "owner-user", contextSnapshot: { issueId: conversation.issueId, taskId: conversation.issueId, source: "automation" } });
|
||||
const interactions = issueThreadInteractionService(db);
|
||||
const issue = { id: conversation.issueId, companyId: fixture.companyId };
|
||||
const interaction = kind === "connection_intent"
|
||||
? await interactions.createConnectionIntent(issue, { sourceRunId, addresseeUserId: "owner-user", idempotencyKey: `slack-review:${sourceRunId}`, payload: { version: 1, serviceSlug: "notion", serviceName: "Notion", requestingAgentId: fixture.assignedAgentId, requestingAgentName: "Maya", phase: "requested" } })
|
||||
: await interactions.create(issue, { kind, sourceRunId, payload: kind === "suggest_tasks" ? { version: 1, tasks: [{ clientKey: "docs", title: "Review migration notes" }] } : { version: 1, prompt: "Review migration notes", ...(kind === "request_checkbox_confirmation" ? { options: [{ id: "docs", label: "Migration notes" }], supersedeOnUserComment: decision === "supersede" } : {}) } } as Parameters<typeof interactions.create>[1], { agentId: fixture.assignedAgentId });
|
||||
await service.processPendingPublications();
|
||||
const rows = () => db.select().from(chatPublications).where(and(eq(chatPublications.endpointId, endpoint.id), eq(sql<string>`${chatPublications.payload}->>'interactionId'`, interaction.id)));
|
||||
const [original] = await rows();
|
||||
expect(original?.state).toBe("published");
|
||||
expect(original?.payload.card?.actions).toEqual(expect.arrayContaining([expect.objectContaining({ type: "callback" })]));
|
||||
|
||||
const [action] = await db.select().from(chatActions).where(and(
|
||||
eq(chatActions.endpointId, endpoint.id), eq(chatActions.kind, "confirmation_response"),
|
||||
eq(sql<string>`${chatActions.payload}->>'interactionId'`, interaction.id),
|
||||
eq(sql<string>`${chatActions.payload}->>'decision'`, decision === "accept" ? "accept" : "reject"),
|
||||
));
|
||||
expect(action).toBeDefined();
|
||||
await db.update(chatActions).set({ status: "processing" }).where(eq(chatActions.id, action.id));
|
||||
const errors: unknown[] = [];
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
const settle = async () => vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
const quiet = async () => {
|
||||
const select = vi.spyOn(db, "select"), transaction = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(select).not.toHaveBeenCalled(); expect(transaction).not.toHaveBeenCalled();
|
||||
} finally { select.mockRestore(); transaction.mockRestore(); }
|
||||
};
|
||||
await service.registerTerminalConfirmationWork(coordinator, () => true).ready;
|
||||
await settle(); await quiet();
|
||||
expect((await db.select().from(chatActions).where(eq(chatActions.id, action.id)))[0].status).toBe("processing");
|
||||
const resolve = async (tx: Db) => {
|
||||
const board = issueThreadInteractionService(tx);
|
||||
if (decision === "withdraw") await board.withdrawInteraction(issue, interaction.id, {}, { userId: "owner-user" });
|
||||
else if (decision === "reassign") await board.expireConnectionIntentsForOwnershipChange(issue);
|
||||
else if (decision === "supersede") {
|
||||
const [comment] = await tx.insert(issueComments).values({ companyId: issue.companyId, issueId: issue.id,
|
||||
authorUserId: "owner-user", body: "Use the revised migration plan", createdAt: new Date(Date.now() + 1000) }).returning();
|
||||
await board.expireRequestConfirmationsSupersededByComment(issue, comment, { userId: "owner-user" });
|
||||
}
|
||||
else if (kind === "connection_intent") await board.resolveConnectionIntent(issue, interaction.id,
|
||||
{ version: 1, outcome: "declined" }, { userId: "owner-user" });
|
||||
else if (decision === "accept") await board.acceptInteraction(issue, interaction.id,
|
||||
{ selectedOptionIds: ["docs"], selectedClientKeys: ["docs"] }, { userId: "owner-user" });
|
||||
else await board.rejectInteraction(issue, interaction.id, {}, { userId: "owner-user" });
|
||||
expect((await db.select().from(chatActions).where(eq(chatActions.id, action.id)))[0].status).toBe("processing");
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0].status).toBe("pending");
|
||||
};
|
||||
await expect(db.transaction(async tx => { await resolve(tx as unknown as Db); throw new Error("rollback board decision"); }))
|
||||
.rejects.toThrow("rollback board decision");
|
||||
await settle(); await quiet();
|
||||
await db.transaction(tx => resolve(tx as unknown as Db));
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect((await db.select().from(chatActions).where(eq(chatActions.id, action.id)))[0].status).toBe(["accept", "reject"].includes(decision) ? "processed" : "expired");
|
||||
}, { timeout: 5_000 });
|
||||
await settle(); await quiet();
|
||||
expect(errors).toEqual([]);
|
||||
} finally {
|
||||
await coordinator?.stop(); vi.useRealTimers(); await service.shutdown();
|
||||
if (previousPublicUrl === undefined) delete process.env.PAPERCLIP_PUBLIC_URL;
|
||||
else process.env.PAPERCLIP_PUBLIC_URL = previousPublicUrl;
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["request_confirmation", "request_checkbox_confirmation", "request_item_verdicts", "suggest_tasks", "connection_intent"] as const)("delivers and replaces a Slack %s card through the publication service", async kind => {
|
||||
const previousPublicUrl = process.env.PAPERCLIP_PUBLIC_URL;
|
||||
process.env.PAPERCLIP_PUBLIC_URL = "https://paperclip.example";
|
||||
@@ -45407,7 +45500,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
expect(stillPending.status).toBe("pending");
|
||||
});
|
||||
|
||||
it("renders safe Telegram confirmations as opaque one-use buttons and wakes the resolved task", async () => {
|
||||
async function telegramConfirmationRecoveryFixture(decision: "accept" | "reject" = "accept") {
|
||||
const fixture = await seedCompany();
|
||||
let injectSettlementCrash = true;
|
||||
const settlementBarrier = vi.fn(async () => {
|
||||
@@ -45565,7 +45658,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
),
|
||||
).toBe(true);
|
||||
const acceptAction = confirmationActions.find(
|
||||
(action) => action.payload.decision === "accept",
|
||||
(action) => action.payload.decision === decision,
|
||||
);
|
||||
if (!acceptAction)
|
||||
throw new Error("Telegram accept action was not persisted");
|
||||
@@ -45597,7 +45690,286 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
value: undefined,
|
||||
},
|
||||
};
|
||||
await expect(callbacks.onAction(actionEvent)).rejects.toThrow(
|
||||
return { fixture, callbacks, onAction: callbacks.onAction, endpoint, service, wakeup, settlementBarrier, dm, conversation, linkedUserId, principal, interaction, publication, acceptAction, actionEvent };
|
||||
}
|
||||
|
||||
it.each(["accept", "reject", "drain", "restart", "failed selection"] as const)("recovers committed terminal confirmations without polling through %s", async mode => {
|
||||
const context = await telegramConfirmationRecoveryFixture(mode === "reject" ? "reject" : "accept");
|
||||
let canRun = true;
|
||||
const errors: unknown[] = [];
|
||||
let coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
const settle = async () => vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
const quiet = async () => {
|
||||
const select = vi.spyOn(db, "select"), transaction = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(select).not.toHaveBeenCalled(); expect(transaction).not.toHaveBeenCalled();
|
||||
} finally { select.mockRestore(); transaction.mockRestore(); }
|
||||
};
|
||||
try {
|
||||
await context.service.registerTerminalConfirmationWork(coordinator, () => canRun).ready;
|
||||
await settle(); await quiet();
|
||||
if (["drain", "failed selection"].includes(mode)) canRun = false;
|
||||
if (mode === "restart") await coordinator.stop();
|
||||
await expect(context.onAction(context.actionEvent)).rejects.toThrow("injected post-interaction confirmation settlement crash");
|
||||
if (["drain", "failed selection", "restart"].includes(mode)) {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
|
||||
expect(await db.select({ status: chatActions.status }).from(chatActions)
|
||||
.where(eq(chatActions.id, context.acceptAction.id))).toEqual([{ status: "processing" }]);
|
||||
await quiet();
|
||||
if (mode === "failed selection") {
|
||||
const select = vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("temporary confirmation selection failure"); });
|
||||
try {
|
||||
canRun = true;
|
||||
await vi.advanceTimersByTimeAsync(Math.max(0, coordinator.nextWakeAt()! - Date.now()));
|
||||
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(10); expect(errors).toHaveLength(1); });
|
||||
canRun = false;
|
||||
} finally { select.mockRestore(); }
|
||||
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
|
||||
await quiet();
|
||||
}
|
||||
canRun = true;
|
||||
if (coordinator.nextWakeAt() !== null) await vi.advanceTimersByTimeAsync(Math.max(0, coordinator.nextWakeAt()! - Date.now()));
|
||||
if (mode === "restart") {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
await context.service.registerTerminalConfirmationWork(coordinator, () => canRun).ready;
|
||||
}
|
||||
}
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(await db.select({ status: chatActions.status }).from(chatActions)
|
||||
.where(eq(chatActions.id, context.acceptAction.id))).toEqual([{ status: "processed" }]);
|
||||
}, { timeout: 10_000 });
|
||||
await settle(); await quiet();
|
||||
expect(await db.select({ id: chatActions.id }).from(chatActions).where(and(
|
||||
eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "interaction_wakeup"),
|
||||
eq(sql<string>`${chatActions.payload}->>'interactionId'`, context.interaction.id),
|
||||
))).toHaveLength(1);
|
||||
expect(await db.select({ id: activityLog.id }).from(activityLog).where(and(
|
||||
eq(activityLog.companyId, context.fixture.companyId),
|
||||
eq(sql<string>`${activityLog.details}->>'interactionId'`, context.interaction.id),
|
||||
eq(sql<string>`${activityLog.details}->>'source'`, "external_chat"),
|
||||
))).toHaveLength(1);
|
||||
expect(errors).toEqual(mode === "failed selection" ? [expect.objectContaining({
|
||||
errors: [expect.objectContaining({ message: "temporary confirmation selection failure" })],
|
||||
})] : []);
|
||||
} finally { await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("parks pending confirmation history, passes malformed prefixes, and wakes on the outer interaction commit", async () => {
|
||||
const waiting = await telegramConfirmationRecoveryFixture();
|
||||
const terminal = await telegramConfirmationRecoveryFixture();
|
||||
const pendingIds = Array.from({ length: 101 }, (_, index) => `00000000-0000-4000-8000-${String(index + 1).padStart(12, "0")}`);
|
||||
const invalidIds = Array.from({ length: 101 }, (_, index) => `00000000-0000-4000-8001-${String(index + 1).padStart(12, "0")}`);
|
||||
await db.insert(chatActions).values([...pendingIds, ...invalidIds].map(id => ({
|
||||
...waiting.acceptAction, id, providerActionId: `confirmation-history:${id}`, status: "processing",
|
||||
conversationId: invalidIds.includes(id) ? null : waiting.conversation.id, result: null,
|
||||
})));
|
||||
await expect(terminal.onAction(terminal.actionEvent)).rejects.toThrow("injected post-interaction confirmation settlement crash");
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
const settle = async () => vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 30_000 });
|
||||
const quiet = async () => {
|
||||
const select = vi.spyOn(db, "select"), transaction = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(select).not.toHaveBeenCalled(); expect(transaction).not.toHaveBeenCalled();
|
||||
} finally { select.mockRestore(); transaction.mockRestore(); }
|
||||
};
|
||||
try {
|
||||
await waiting.service.registerTerminalConfirmationWork(coordinator, () => true).ready;
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(await db.select({ status: chatActions.status }).from(chatActions)
|
||||
.where(eq(chatActions.id, terminal.acceptAction.id))).toEqual([{ status: "processed" }]);
|
||||
}, { timeout: 10_000 });
|
||||
await settle();
|
||||
expect(await db.select({ id: chatActions.id }).from(chatActions)
|
||||
.where(and(inArray(chatActions.id, pendingIds), eq(chatActions.status, "processing")))).toHaveLength(101);
|
||||
expect(await db.select({ id: chatActions.id }).from(chatActions).where(and(
|
||||
inArray(chatActions.id, invalidIds), eq(chatActions.status, "failed"),
|
||||
eq(sql<string>`${chatActions.result}->>'code'`, "confirmation_action_binding_invalid"),
|
||||
))).toHaveLength(101);
|
||||
await quiet();
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, waiting.conversation.issueId));
|
||||
await expect(db.transaction(async tx => {
|
||||
await issueThreadInteractionService(tx as unknown as Db).acceptInteraction(issue, waiting.interaction.id, {}, { userId: waiting.linkedUserId });
|
||||
expect(await db.select({ status: issueThreadInteractions.status }).from(issueThreadInteractions)
|
||||
.where(eq(issueThreadInteractions.id, waiting.interaction.id))).toEqual([{ status: "pending" }]);
|
||||
throw new Error("rollback outer confirmation decision");
|
||||
})).rejects.toThrow("rollback outer confirmation decision");
|
||||
await settle(); await quiet();
|
||||
expect(await db.select({ id: chatActions.id }).from(chatActions)
|
||||
.where(and(inArray(chatActions.id, pendingIds), eq(chatActions.status, "processing")))).toHaveLength(101);
|
||||
await db.transaction(async tx => {
|
||||
await issueThreadInteractionService(tx as unknown as Db).acceptInteraction(issue, waiting.interaction.id, {}, { userId: waiting.linkedUserId });
|
||||
expect(await db.select({ status: issueThreadInteractions.status }).from(issueThreadInteractions)
|
||||
.where(eq(issueThreadInteractions.id, waiting.interaction.id))).toEqual([{ status: "pending" }]);
|
||||
expect(await db.select({ id: chatActions.id }).from(chatActions)
|
||||
.where(and(inArray(chatActions.id, pendingIds), eq(chatActions.status, "processing")))).toHaveLength(101);
|
||||
});
|
||||
await settle(); await quiet();
|
||||
expect(await db.select({ id: chatActions.id }).from(chatActions)
|
||||
.where(and(inArray(chatActions.id, pendingIds), eq(chatActions.status, "processing")))).toHaveLength(0);
|
||||
expect(await db.select({ id: chatActions.id }).from(chatActions)
|
||||
.where(and(inArray(chatActions.id, pendingIds), eq(chatActions.status, "processed")))).toHaveLength(1);
|
||||
expect(await db.select({ id: activityLog.id }).from(activityLog).where(and(
|
||||
eq(activityLog.companyId, waiting.fixture.companyId),
|
||||
eq(sql<string>`${activityLog.details}->>'interactionId'`, waiting.interaction.id),
|
||||
eq(sql<string>`${activityLog.details}->>'source'`, "external_chat"),
|
||||
))).toHaveLength(0);
|
||||
expect(errors).toEqual([]);
|
||||
} finally {
|
||||
await coordinator.stop(); vi.useRealTimers();
|
||||
await db.delete(chatActions).where(inArray(chatActions.id, [...pendingIds, ...invalidIds]));
|
||||
}
|
||||
});
|
||||
|
||||
it("bounds terminal confirmation recovery reads with completed history", async () => {
|
||||
const waiting = await telegramConfirmationRecoveryFixture();
|
||||
const prefix = `confirmation-plan:${randomUUID()}:`;
|
||||
await db.execute(sql`
|
||||
INSERT INTO chat_actions (company_id, endpoint_id, kind, provider_action_id, status)
|
||||
SELECT ${waiting.fixture.companyId}::uuid, ${waiting.endpoint.id}::uuid,
|
||||
CASE WHEN n <= 10000 THEN 'confirmation_response' ELSE 'unrelated_action' END,
|
||||
${prefix} || n,
|
||||
CASE WHEN n <= 10000 THEN 'processed' ELSE 'processing' END
|
||||
FROM generate_series(1, 20000) AS n
|
||||
`);
|
||||
await db.update(chatActions).set({ status: "processing", result: null })
|
||||
.where(eq(chatActions.id, waiting.acceptAction.id));
|
||||
await db.execute(sql`VACUUM (ANALYZE) chat_actions`);
|
||||
const statements: { sql: string; params: unknown[] }[] = [];
|
||||
const select = db.select.bind(db);
|
||||
const reads = vi.spyOn(db, "select").mockImplementation((...args) => {
|
||||
const selection = select(...args), from = selection.from.bind(selection);
|
||||
vi.spyOn(selection, "from").mockImplementation(table => {
|
||||
const query = from(table);
|
||||
if (table === chatActions) {
|
||||
const execute = query.execute.bind(query);
|
||||
vi.spyOn(query, "execute").mockImplementation(async () => {
|
||||
const statement = query.toSQL();
|
||||
if (statement.sql.includes("'confirmation_response'") && statement.sql.includes("'processing'")) {
|
||||
statements.push(statement);
|
||||
}
|
||||
return execute();
|
||||
});
|
||||
}
|
||||
return query;
|
||||
});
|
||||
return selection;
|
||||
});
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
try {
|
||||
await waiting.service.registerTerminalConfirmationWork(coordinator, () => true).ready;
|
||||
await vi.waitFor(() => expect(statements.length).toBeGreaterThan(0));
|
||||
const startup = statements[0];
|
||||
const beforeHint = statements.length;
|
||||
await db.transaction(async tx => {
|
||||
await signalDatabaseWork(tx, DELIVERY_QUEUES.chatTerminalConfirmations,
|
||||
JSON.stringify({ companyId: waiting.fixture.companyId }));
|
||||
});
|
||||
await vi.waitFor(() => expect(statements.length).toBeGreaterThan(beforeHint), { timeout: 5_000 });
|
||||
const scoped = statements.at(-1)!;
|
||||
expect(scoped.params).toContain(waiting.fixture.companyId);
|
||||
await coordinator.stop();
|
||||
reads.mockRestore();
|
||||
for (const statement of [startup, scoped]) {
|
||||
expect(await db.$client.unsafe(statement.sql, statement.params as Parameters<typeof db.$client.unsafe>[1]))
|
||||
.toMatchObject([{ id: waiting.acceptAction.id }]);
|
||||
const result = await db.$client.unsafe(
|
||||
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${statement.sql}`, statement.params as Parameters<typeof db.$client.unsafe>[1],
|
||||
);
|
||||
const plan = result[0]["QUERY PLAN"][0].Plan;
|
||||
type PlanNode = { Plans?: PlanNode[]; [key: string]: unknown };
|
||||
const nodes = (node: PlanNode): PlanNode[] => [node, ...(node.Plans ?? []).flatMap(nodes)];
|
||||
const actionReads = nodes(plan).filter(node => node["Relation Name"] === "chat_actions");
|
||||
expect(actionReads).toHaveLength(1);
|
||||
expect(Number(actionReads[0]["Actual Rows"]) + Number(actionReads[0]["Rows Removed by Filter"] ?? 0),
|
||||
JSON.stringify({ params: statement.params, actionReads }))
|
||||
.toBeLessThanOrEqual(2);
|
||||
expect(Number(actionReads[0]["Shared Hit Blocks"]) + Number(actionReads[0]["Shared Read Blocks"]))
|
||||
.toBeLessThan(20);
|
||||
}
|
||||
await db.update(chatActions).set({ status: "processed" }).where(eq(chatActions.id, waiting.acceptAction.id));
|
||||
await db.execute(sql`ANALYZE chat_actions`);
|
||||
expect(await db.$client.unsafe(scoped.sql, scoped.params as Parameters<typeof db.$client.unsafe>[1])).toHaveLength(0);
|
||||
const empty = await db.$client.unsafe(
|
||||
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${scoped.sql}`, scoped.params as Parameters<typeof db.$client.unsafe>[1],
|
||||
);
|
||||
expect(empty[0]["QUERY PLAN"][0].Plan["Shared Hit Blocks"]
|
||||
+ empty[0]["QUERY PLAN"][0].Plan["Shared Read Blocks"]).toBeLessThan(20);
|
||||
expect(errors).toEqual([]);
|
||||
} finally {
|
||||
await coordinator.stop(); reads.mockRestore();
|
||||
await db.delete(chatActions).where(sql`${chatActions.providerActionId} LIKE ${`${prefix}%`}`);
|
||||
}
|
||||
});
|
||||
|
||||
it("retains a confirmation committed while another settlement is in flight", async () => {
|
||||
const first = await telegramConfirmationRecoveryFixture();
|
||||
const second = await telegramConfirmationRecoveryFixture();
|
||||
await expect(first.onAction(first.actionEvent)).rejects.toThrow("injected post-interaction confirmation settlement crash");
|
||||
let locked!: () => void, release!: () => void;
|
||||
const lockReady = new Promise<void>(resolve => { locked = resolve; });
|
||||
const held = new Promise<void>(resolve => { release = resolve; });
|
||||
const transaction = db.transaction(async tx => {
|
||||
await tx.select({ id: chatActions.id }).from(chatActions).where(eq(chatActions.id, first.acceptAction.id)).for("update");
|
||||
locked(); await held;
|
||||
});
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
let ready: Promise<unknown> | undefined;
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
try {
|
||||
await lockReady;
|
||||
ready = first.service.registerTerminalConfirmationWork(coordinator, () => true).ready;
|
||||
await vi.waitFor(async () => {
|
||||
const blocked = await db.execute(sql`select pid from pg_stat_activity
|
||||
where datname = current_database() and wait_event_type = 'Lock'
|
||||
and query like 'update "chat_actions"%'`);
|
||||
expect(blocked.length).toBeGreaterThan(0);
|
||||
}, { timeout: 10_000 });
|
||||
await expect(second.onAction(second.actionEvent)).rejects.toThrow("injected post-interaction confirmation settlement crash");
|
||||
expect(await db.select({ status: chatActions.status }).from(chatActions)
|
||||
.where(eq(chatActions.id, second.acceptAction.id))).toEqual([{ status: "processing" }]);
|
||||
release(); await transaction; await ready;
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(await db.select({ status: chatActions.status }).from(chatActions)
|
||||
.where(inArray(chatActions.id, [first.acceptAction.id, second.acceptAction.id])))
|
||||
.toEqual([{ status: "processed" }, { status: "processed" }]);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
const select = vi.spyOn(db, "select"), queryTransaction = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(select).not.toHaveBeenCalled(); expect(queryTransaction).not.toHaveBeenCalled();
|
||||
} finally { select.mockRestore(); queryTransaction.mockRestore(); }
|
||||
expect(errors).toEqual([]);
|
||||
} finally {
|
||||
release(); await transaction; await ready; await coordinator.stop(); vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it("renders safe Telegram confirmations as opaque one-use buttons and wakes the resolved task", async () => {
|
||||
const { fixture, onAction, endpoint, service, wakeup, settlementBarrier, dm, conversation, linkedUserId, principal, interaction, publication, acceptAction, actionEvent } = await telegramConfirmationRecoveryFixture();
|
||||
await expect(onAction(actionEvent)).rejects.toThrow(
|
||||
"injected post-interaction confirmation settlement crash",
|
||||
);
|
||||
expect(settlementBarrier).toHaveBeenCalledTimes(1);
|
||||
@@ -45776,7 +46148,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
),
|
||||
)
|
||||
.then((rows) => rows.length);
|
||||
await callbacks.onAction(actionEvent);
|
||||
await onAction(actionEvent);
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
expect(dm.post).toHaveBeenCalledTimes(providerPostCountBeforeRedelivery);
|
||||
await expect(
|
||||
@@ -64915,6 +65287,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
await quiet(); canRun = true;
|
||||
if (coordinator.nextWakeAt() !== null) await vi.advanceTimersByTimeAsync(Math.max(0, coordinator.nextWakeAt()! - Date.now()));
|
||||
if (mode === "stopped") {
|
||||
await context.service.processPublicationMaintenance();
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
@@ -64924,7 +65297,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
}
|
||||
}
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(30_000);
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([expect.objectContaining({
|
||||
payload: mode === "disabled"
|
||||
? { text: "This retry was not started. Open the task in Paperclip for details.", progressState: "failed" }
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { enqueueIssueInteractionChatPublications } from "../services/chat-interaction-publications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
@@ -20,16 +21,20 @@ import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";
|
||||
import { reserveRunBudget } from "../services/budget-reservations.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const externalTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_SATISFIED_CONNECTION_TEST_DATABASE_URL");
|
||||
const support = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
const describePostgres = support.supported ? describe : describe.skip;
|
||||
|
||||
describePostgres("already available connection requests", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let temp!: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
beforeAll(async () => {
|
||||
temp = await startEmbeddedPostgresTestDatabase("paperclip-satisfied-connections-");
|
||||
db = createDb(temp.connectionString);
|
||||
await db.insert(authUsers).values({ id: "connection-owner", name: "Owner", email: "owner@example.test", createdAt: new Date(), updatedAt: new Date() });
|
||||
if (externalTestDatabaseUrl) db = createDb(externalTestDatabaseUrl);
|
||||
else {
|
||||
temp = await startEmbeddedPostgresTestDatabase("paperclip-satisfied-connections-");
|
||||
db = createDb(temp.connectionString);
|
||||
}
|
||||
await db.insert(authUsers).values({ id: "connection-owner", name: "Owner", email: "owner@example.test", createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing();
|
||||
}, 20_000);
|
||||
beforeEach(async () => { await db.execute(sql`truncate table companies cascade`); });
|
||||
afterAll(async () => { await db.$client.end({ timeout: 1 }); await temp?.cleanup(); });
|
||||
@@ -60,6 +65,21 @@ describePostgres("already available connection requests", () => {
|
||||
return run!;
|
||||
}
|
||||
|
||||
it("notifies terminal confirmation recovery after existing-access retirement commits", async () => {
|
||||
const f = await seed();
|
||||
const notified = vi.fn();
|
||||
const unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.chatTerminalConfirmations, notified);
|
||||
try {
|
||||
await db.transaction(async tx => {
|
||||
expect(await satisfiedConnectionIntentService(tx as unknown as typeof db).sweepPending()).toMatchObject({ satisfied: 1 });
|
||||
expect(notified).not.toHaveBeenCalled();
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, f.intent.id)))[0].status).toBe("pending");
|
||||
});
|
||||
expect(notified).toHaveBeenCalledOnce();
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, f.intent.id)))[0].status).toBe("expired");
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
|
||||
it("settles a published Slack card when existing access retires the connection request", async () => {
|
||||
const f = await seed();
|
||||
const endpointId = randomUUID();
|
||||
@@ -378,7 +398,7 @@ describePostgres("already available connection requests", () => {
|
||||
|
||||
it("records a system completion and one durable continuation with a one-connection pool", async () => {
|
||||
const f = await seed();
|
||||
const single = createDb(temp.connectionString, { maxConnections: 1 });
|
||||
const single = createDb(externalTestDatabaseUrl ?? temp.connectionString, { maxConnections: 1 });
|
||||
try {
|
||||
const outcomes = await Promise.all([satisfiedConnectionIntentService(single).sweepPending(), satisfiedConnectionIntentService(db).sweepPending()]);
|
||||
expect(outcomes.reduce((sum, outcome) => sum + outcome.satisfied, 0)).toBe(1);
|
||||
|
||||
@@ -1288,6 +1288,7 @@ export async function createApp(
|
||||
chatChannels.registerReceiverWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
chatChannels.registerTeamsConsentExpiryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
chatChannels.registerTeamsAttemptRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
chatChannels.registerTerminalConfirmationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
chatChannels.registerCommandWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
nativeChatProgressWork(db).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
chatInboundWakeNoticeWork(chatChannels).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
|
||||
@@ -3350,7 +3350,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
||||
|
||||
async function processPublicationMaintenance(limit = 25) {
|
||||
scheduleGitHubMaintenance();
|
||||
await reconcileTerminalConfirmationActions(limit);
|
||||
await processPendingInteractionWakeups(limit);
|
||||
}
|
||||
|
||||
@@ -3479,7 +3478,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
||||
const { processPendingInteractionWakeups } = createChatInteractionContinuations(db, {
|
||||
redactError, CONFIRMATION_WAKEUP_RETRY_BACKOFF_MS, get heartbeat() { return options.heartbeat; },
|
||||
});
|
||||
const { handleAction, handleModalSubmit, reconcileTerminalConfirmationActions } = createChatInteractionCallbacks(db, {
|
||||
const { handleAction, handleModalSubmit, reconcileTerminalConfirmationActions, registerTerminalConfirmationWork } = createChatInteractionCallbacks(db, {
|
||||
heartbeat: options.heartbeat,
|
||||
CAPABILITIES, MAX_ERROR_TEXT, actionThreadMatchesConversation, nonDirectDestinationAllowed, sameTeamsThreadIdentity,
|
||||
ensurePrincipal, endpointRecord, runtimeCallbackRecord, runtimeCallbackEndpoint,
|
||||
@@ -3503,6 +3502,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
||||
const { prepareTeamsBoardSend } = createTeamsBoardSendPreparation(db, { endpointRecord, runtimeFor });
|
||||
const { getPublicationBatchStatus, publishComment, publishBoardMessage } = createChatPublicationRequests(db, { endpointRecord, stageTeamsBoardFileIntent, processPendingSlackBoardMessages, processPendingPublications, prepareTeamsBoardSend });
|
||||
return {
|
||||
registerTerminalConfirmationWork,
|
||||
loadFailedRetryNoticeScopes,
|
||||
reconcileFailedRetryNotices,
|
||||
registerTeamsAttemptRecoveryWork,
|
||||
|
||||
@@ -145,7 +145,7 @@ type InteractionCallbacksOptions = {
|
||||
export function createChatInteractionCallbacks(db: Db, options: InteractionCallbacksOptions) {
|
||||
const { CAPABILITIES, MAX_ERROR_TEXT, actionThreadMatchesConversation, nonDirectDestinationAllowed, sameTeamsThreadIdentity, ensurePrincipal, endpointRecord, runtimeCallbackRecord, runtimeCallbackEndpoint, requireCurrentExternalActionAuthorization, isExternalActionAuthorizationChange, recordCurrentMicrosoftTeamsRoute, lockTelegramCallbackNoticeAuthority, stageProviderEffect, scheduleProviderEffect, invalidateRuntime, redactError, persistence, questionResponses, scheduleMessageProcessing, processPendingPublications } = options;
|
||||
const { handleDiscordFormCorrection, acceptDiscordQuestionForm } = createDiscordQuestionForms(db, { requireCurrentExternalActionAuthorization, isExternalActionAuthorizationChange, persistence });
|
||||
const { reconcileTerminalConfirmationActions, handleChatConfirmation } = createChatConfirmationActions(db, {
|
||||
const { reconcileTerminalConfirmationActions, handleChatConfirmation, registerTerminalConfirmationWork } = createChatConfirmationActions(db, {
|
||||
heartbeat: options.heartbeat, requireCurrentExternalActionAuthorization, isExternalActionAuthorizationChange, scheduleMessageProcessing, processPendingPublications,
|
||||
get confirmationResolutionPersistBarrier() { return options.confirmationResolutionPersistBarrier; },
|
||||
});
|
||||
@@ -2029,5 +2029,5 @@ export function createChatInteractionCallbacks(db: Db, options: InteractionCallb
|
||||
});
|
||||
return acceptedResponse();
|
||||
}
|
||||
return { handleAction, handleModalSubmit, reconcileTerminalConfirmationActions };
|
||||
return { handleAction, handleModalSubmit, reconcileTerminalConfirmationActions, registerTerminalConfirmationWork };
|
||||
}
|
||||
@@ -3,12 +3,20 @@ import { queueIssueAssignmentWakeup, type IssueAssignmentWakeupDeps } from "../.
|
||||
import { chatInteractionPresentation } from "../../chat-interaction-presentation.js";
|
||||
import type { DbTransaction, RuntimeContext, LifecycleRuntimeFence } from "../runtime-types.js";
|
||||
|
||||
import type { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js";
|
||||
import { registerScopedDeadlineWork } from "../../scoped-deadline-work.js";
|
||||
import { DELIVERY_QUEUES } from "../../delivery-work-notifications.js";
|
||||
import { notifyChatConfirmationWork, type ChatConfirmationScope } from "../../chat-confirmation-notifications.js";
|
||||
|
||||
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
|
||||
|
||||
import {
|
||||
and,
|
||||
asc,
|
||||
eq,
|
||||
gt,
|
||||
isNull,
|
||||
or,
|
||||
inArray,
|
||||
ne,
|
||||
sql,
|
||||
@@ -176,6 +184,7 @@ export function createChatConfirmationActions(db: Db, options: ChatConfirmationO
|
||||
});
|
||||
},
|
||||
afterResolveInTransaction: async (tx: DbTransaction, resolved: import("@paperclipai/shared").IssueThreadInteraction) => {
|
||||
await notifyChatConfirmationWork(tx, { companyId: action.companyId, issueId: conversation.issueId });
|
||||
const [owned] = await tx
|
||||
.update(chatActions)
|
||||
.set({
|
||||
@@ -286,8 +295,13 @@ export function createChatConfirmationActions(db: Db, options: ChatConfirmationO
|
||||
)
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (action?.status === "processed") return true;
|
||||
if (!action || action.status !== "processing" || !action.conversationId)
|
||||
if (!action || action.status !== "processing") return false;
|
||||
if (!action.conversationId) {
|
||||
await db.update(chatActions).set({
|
||||
status: "failed", result: { code: "confirmation_action_binding_invalid" }, updatedAt: new Date(),
|
||||
}).where(and(eq(chatActions.id, action.id), eq(chatActions.status, "processing")));
|
||||
return false;
|
||||
}
|
||||
const token = action.payload;
|
||||
if (
|
||||
token.version !== 1 ||
|
||||
@@ -647,5 +661,44 @@ export function createChatConfirmationActions(db: Db, options: ChatConfirmationO
|
||||
}
|
||||
return settled;
|
||||
}
|
||||
return { settleTerminalConfirmationAction, reconcileTerminalConfirmationActions, handleChatConfirmation };
|
||||
function registerTerminalConfirmationWork(
|
||||
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
|
||||
canRun: () => boolean,
|
||||
) {
|
||||
type Entry = { id: string; companyId: string; issueId: string | null; dueAt: number };
|
||||
return registerScopedDeadlineWork(coordinator, {
|
||||
queue: DELIVERY_QUEUES.chatTerminalConfirmations,
|
||||
decodeScope: (hint: string): ChatConfirmationScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
|
||||
matches: (entry: Entry, scope?: ChatConfirmationScope) => !scope ||
|
||||
(entry.companyId === scope.companyId && (!scope.issueId || !entry.issueId || entry.issueId === scope.issueId)),
|
||||
load: async (scope?: ChatConfirmationScope, signal?: AbortSignal): Promise<Entry[]> => {
|
||||
const entries: Entry[] = [];
|
||||
let cursor: { companyId: string; id: string } | undefined;
|
||||
while (!signal?.aborted) {
|
||||
const rows = await db.select({
|
||||
id: chatActions.id, companyId: chatActions.companyId, issueId: chatConversations.issueId,
|
||||
}).from(chatActions).leftJoin(chatConversations, and(
|
||||
eq(chatConversations.companyId, chatActions.companyId), eq(chatConversations.id, chatActions.conversationId),
|
||||
)).where(and(
|
||||
sql`${chatActions.kind} = 'confirmation_response' AND ${chatActions.status} = 'processing'`,
|
||||
scope ? eq(chatActions.companyId, scope.companyId) : undefined,
|
||||
scope?.issueId ? or(eq(chatConversations.issueId, scope.issueId), isNull(chatConversations.id)) : undefined,
|
||||
cursor ? or(
|
||||
gt(chatActions.companyId, cursor.companyId),
|
||||
and(eq(chatActions.companyId, cursor.companyId), gt(chatActions.id, cursor.id)),
|
||||
) : undefined,
|
||||
)).orderBy(asc(chatActions.companyId), asc(chatActions.id)).limit(100);
|
||||
entries.push(...rows.map(row => ({ ...row, dueAt: 0 })));
|
||||
if (rows.length < 100) return entries;
|
||||
cursor = rows[rows.length - 1];
|
||||
}
|
||||
throw signal?.reason ?? new Error("Confirmation recovery stopped");
|
||||
},
|
||||
check: async (entry: Entry) => {
|
||||
await settleTerminalConfirmationAction(entry.id);
|
||||
return null;
|
||||
},
|
||||
}, canRun);
|
||||
}
|
||||
return { settleTerminalConfirmationAction, reconcileTerminalConfirmationActions, handleChatConfirmation, registerTerminalConfirmationWork };
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type ChatConfirmationScope = { companyId: string; issueId?: string };
|
||||
|
||||
export async function notifyChatConfirmationWork(tx: object, scope: ChatConfirmationScope) {
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatTerminalConfirmations,
|
||||
JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }));
|
||||
}
|
||||
|
||||
/** Interaction resolution can release a callback parked while its human input
|
||||
* was pending. Register on the owning transaction before its resolution write. */
|
||||
export async function notifyChatConfirmationResolution(tx: object, interaction: ChatConfirmationScope & { kind: string }) {
|
||||
if (["request_confirmation", "request_checkbox_confirmation", "suggest_tasks", "connection_intent"].includes(interaction.kind)) {
|
||||
await notifyChatConfirmationWork(tx, interaction);
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyChatConfirmationWork } from "./chat-confirmation-notifications.js";
|
||||
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
|
||||
import { notifyChatNativeProgress } from "./chat-native-progress-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
@@ -13,4 +14,5 @@ export async function notifyChatRunMilestones(tx: object, scope: ChatRunMileston
|
||||
await notifyChatNativeProgress(tx, scope);
|
||||
// Admission and current chat authority also govern saved queue notices.
|
||||
await notifyChatInboundWakeNotices(tx, scope);
|
||||
await notifyChatConfirmationWork(tx, scope);
|
||||
}
|
||||
@@ -59,6 +59,7 @@ export const DELIVERY_QUEUES = {
|
||||
chatPublications: "chat-publications",
|
||||
chatNativeProgress: "chat-native-progress",
|
||||
chatInboundWakeNotices: "chat-inbound-wake-notices",
|
||||
chatTerminalConfirmations: "chat-terminal-confirmations",
|
||||
chatFailedRetryNotices: "chat-failed-retry-notices",
|
||||
chatRunMilestones: "chat-run-milestones",
|
||||
chatReceipts: "chat-slack-receipts",
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js";
|
||||
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
|
||||
import { notifyToolReviewChange } from "./tool-review-work-notifications.js";
|
||||
import { normalizeEscapedLineBreaks } from "@paperclipai/shared/validators/text";
|
||||
@@ -1913,6 +1914,7 @@ async function expireStaleRequestConfirmationTarget(
|
||||
snapshot,
|
||||
});
|
||||
const expireInTransaction = async (tx: Db | any) => {
|
||||
await notifyChatConfirmationResolution(tx, args.row);
|
||||
if (args.row.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.row);
|
||||
const [updated] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
@@ -2191,6 +2193,7 @@ export function issueThreadInteractionService(
|
||||
const postCommitActivityPublications: ActivityPublication[] = [];
|
||||
const result = await db.transaction(async (tx) => {
|
||||
await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor);
|
||||
await notifyChatConfirmationResolution(tx, args.current);
|
||||
if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current);
|
||||
await args.mutationOptions?.beforeResolveInTransaction?.(tx);
|
||||
// Policy mutations and review transitions use the same issue-row lock,
|
||||
@@ -2475,6 +2478,7 @@ export function issueThreadInteractionService(
|
||||
const now = new Date();
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor);
|
||||
await notifyChatConfirmationResolution(tx, args.current);
|
||||
if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current);
|
||||
await args.mutationOptions?.beforeResolveInTransaction?.(tx);
|
||||
const issueContext = await tx
|
||||
@@ -2762,6 +2766,7 @@ export function issueThreadInteractionService(
|
||||
})
|
||||
.map((candidate) => candidate.id);
|
||||
if (supersededIds.length > 0) {
|
||||
await notifyChatConfirmationResolution(tx, { companyId: issue.companyId, issueId: issue.id, kind: "connection_intent" });
|
||||
await tx
|
||||
.delete(toolOauthStates)
|
||||
.where(inArray(toolOauthStates.interactionId, supersededIds));
|
||||
@@ -2860,6 +2865,7 @@ export function issueThreadInteractionService(
|
||||
: "expired";
|
||||
const resolvedAt = now();
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, current);
|
||||
await mutationOptions.beforeResolveInTransaction?.(tx);
|
||||
const [row] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
@@ -3294,6 +3300,7 @@ export function issueThreadInteractionService(
|
||||
const cancelled: IssueThreadInteraction[] = [];
|
||||
for (const row of rows) {
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, row);
|
||||
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
|
||||
if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]);
|
||||
const [resolved] = await tx
|
||||
@@ -3380,6 +3387,7 @@ export function issueThreadInteractionService(
|
||||
const expired: IssueThreadInteraction[] = [];
|
||||
for (const { row, replacementInteractionId } of supersededRows) {
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, row);
|
||||
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
|
||||
if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]);
|
||||
const [updatedRow] = await tx
|
||||
@@ -4071,6 +4079,7 @@ export function issueThreadInteractionService(
|
||||
const createdWakeTargets: IssueWakeTarget[] = [];
|
||||
|
||||
await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, current);
|
||||
await mutationOptions.beforeResolveInTransaction?.(tx);
|
||||
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
|
||||
const resolvedAt = new Date();
|
||||
@@ -4377,6 +4386,7 @@ export function issueThreadInteractionService(
|
||||
}
|
||||
|
||||
const [updated] = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, current);
|
||||
await mutationOptions.beforeResolveInTransaction?.(tx);
|
||||
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
|
||||
const rows = await tx.update(issueThreadInteractions)
|
||||
@@ -4468,6 +4478,7 @@ export function issueThreadInteractionService(
|
||||
const expired: IssueThreadInteraction[] = [];
|
||||
for (const row of superseded) {
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, row);
|
||||
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
|
||||
if (row.kind === "connection_intent") {
|
||||
await tx
|
||||
@@ -4644,6 +4655,7 @@ export function issueThreadInteractionService(
|
||||
);
|
||||
if (!sampleConfirmationRow) continue;
|
||||
const updatedInteractions = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, sampleConfirmationRow);
|
||||
if (commentRows.some(row => row.kind === "request_confirmation")) await notifyChatRunMilestones(tx, sampleConfirmationRow);
|
||||
await notifyToolProjectionForInteractions(tx, issue.companyId, confirmationRowIds);
|
||||
const updatedRows = await tx
|
||||
@@ -4705,6 +4717,7 @@ export function issueThreadInteractionService(
|
||||
|
||||
for (const row of connectionIntentRows) {
|
||||
const [updated] = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, row);
|
||||
await tx
|
||||
.delete(toolOauthStates)
|
||||
.where(eq(toolOauthStates.interactionId, row.id));
|
||||
@@ -4790,6 +4803,7 @@ export function issueThreadInteractionService(
|
||||
document,
|
||||
});
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, row);
|
||||
if (row.kind === "request_confirmation") await notifyChatRunMilestones(tx, row);
|
||||
const [resolved] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
@@ -4853,27 +4867,29 @@ export function issueThreadInteractionService(
|
||||
const subject = payload?.purpose === "ai"
|
||||
? "the AI account"
|
||||
: `the ${payload?.serviceName?.trim() || "requested"} connection`;
|
||||
const [updated] = await db
|
||||
.update(issueThreadInteractions)
|
||||
.set({
|
||||
status: "expired",
|
||||
result: {
|
||||
version: 1,
|
||||
outcome: "expired",
|
||||
reason: `The task was reassigned before the connection request was answered, so ${subject} is still not connected. Reconnect it or choose an available connection.`,
|
||||
},
|
||||
resolvedAt: now(),
|
||||
updatedAt: now(),
|
||||
})
|
||||
.where(and(
|
||||
eq(issueThreadInteractions.id, row.id),
|
||||
eq(issueThreadInteractions.status, "pending"),
|
||||
))
|
||||
.returning();
|
||||
if (updated) {
|
||||
await enqueueTerminalIssueInteractionChatPublications(db, hydrateInteraction(updated));
|
||||
expired.push(updated);
|
||||
}
|
||||
const updated = await db.transaction(async tx => {
|
||||
await notifyChatConfirmationResolution(tx, { companyId: issue.companyId, issueId: issue.id, kind: "connection_intent" });
|
||||
const [resolved] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
.set({
|
||||
status: "expired",
|
||||
result: {
|
||||
version: 1,
|
||||
outcome: "expired",
|
||||
reason: `The task was reassigned before the connection request was answered, so ${subject} is still not connected. Reconnect it or choose an available connection.`,
|
||||
},
|
||||
resolvedAt: now(),
|
||||
updatedAt: now(),
|
||||
})
|
||||
.where(and(
|
||||
eq(issueThreadInteractions.id, row.id),
|
||||
eq(issueThreadInteractions.status, "pending"),
|
||||
))
|
||||
.returning();
|
||||
if (resolved) await enqueueTerminalIssueInteractionChatPublications(tx as unknown as Db, hydrateInteraction(resolved));
|
||||
return resolved;
|
||||
});
|
||||
if (updated) expired.push(updated);
|
||||
}
|
||||
if (expired.length) await db.delete(toolOauthStates).where(inArray(toolOauthStates.interactionId, expired.map((item) => item.id)));
|
||||
return expired;
|
||||
@@ -4911,6 +4927,7 @@ export function issueThreadInteractionService(
|
||||
// lifecycle reflection.
|
||||
const updated = await db
|
||||
.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, row);
|
||||
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
|
||||
if (row.kind === "connection_intent") {
|
||||
await tx
|
||||
@@ -5002,6 +5019,7 @@ export function issueThreadInteractionService(
|
||||
// request must not outlive a withdrawn card.
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
|
||||
await notifyChatConfirmationResolution(tx, current);
|
||||
if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current);
|
||||
await resolveLinkedToolActionRequests(tx, current, {
|
||||
status: "cancelled",
|
||||
@@ -5263,6 +5281,7 @@ export function issueThreadInteractionService(
|
||||
const now = new Date();
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
|
||||
await notifyChatConfirmationResolution(tx, current);
|
||||
if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current);
|
||||
await resolveLinkedToolActionRequests(tx, current, {
|
||||
status: "cancelled",
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js";
|
||||
import { enqueueTerminalIssueInteractionChatPublications } from "./chat-interaction-publications.js";
|
||||
import { connectionManagementService } from "./connection-management.js";
|
||||
import { chatCredentialEntryGenerations } from "./chat-channels/credentials.js";
|
||||
@@ -156,6 +157,7 @@ export function satisfiedConnectionIntentService(db: Db) {
|
||||
if (!connection) return false;
|
||||
const now = new Date();
|
||||
// Retire the obsolete request without inventing a human approval.
|
||||
await notifyChatConfirmationResolution(tx, intent);
|
||||
const [retired] = await tx.update(issueThreadInteractions).set({ status: "expired", resolvedAt: now, updatedAt: now,
|
||||
result: { version: 1, outcome: "expired", connectionId: connection.id,
|
||||
reason: "An already authorized connection is available to the requesting agent." },
|
||||
|
||||
Reference in new issue
Block a user