fix(server): wake terminal confirmation recovery from committed changes (#17651)

<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/202/changes/paperclip/202
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip persists confirmation clicks and interaction decisions
before dispatch completes.
> - A crash after a decision commits can leave a processing confirmation
action.
> - Periodic publication maintenance repeatedly scans those actions even
when idle.
> - Saved decisions and the supported board/admin writers already
identify when settlement can change.
> - This owner reconstructs durable actions, reconciles real changes and
then becomes quiet.

## Linked Issues or Issue Description

This existing terminal-confirmation slice follows merged #17649. It
removes only terminal-confirmation recovery from periodic chat
maintenance while retaining the explicit manual API and existing durable
action/interaction/publication rows.

**Problem:** An accepted response can commit before its dispatch
settles. Removing periodic recovery without covering board decisions,
expiry and ownership changes would strand processing actions. Pending
human decisions must remain recoverable without polling indefinitely.

**Expected behavior:** Committed supported changes wake the scoped
owner. It reconciles terminal responses once, parks unresolved human
decisions, preserves concurrent commits/restart/drain and becomes quiet
when settled.

## What Changed

- Register terminal-confirmation work with the existing delivery
coordinator and scoped lifecycle. Reconstruct processing action IDs,
bound selection pages, preserve concurrent wakes and keep current
authorization in the existing interaction owner.
- Add the existing transaction-aware confirmation notification topic at
real resolution writes and callback settlement. Reuse milestone signals
instead of adding per-caller recovery wrappers.
- Cover ordinary and checkbox confirmations, suggestions and connection
intents through their owning service. Cover board accept/reject,
cancellation/withdrawal, opt-in comment supersession, expiry,
reassignment and already-satisfied connection resolution.
- Preserve task/actor authority, one-use callback tokens and immutable
provider publication semantics. Explicitly fail malformed missing
bindings instead of silently retaining unrecoverable processing actions.
- Remove only this duty from recurring publication maintenance. Add no
table or scheduler infrastructure. A concurrent partial index on
existing action rows prevents recovery discovery from scanning completed
history.

## Verification

Published repair head: `a35e78d076c1b618bc643391b7520fa6952fc996`,
targeting master after normal integration of
`644937c93150c423f12674992c7ea1e8b5dbc98c`. [Detailed acceptance and
preserved
failures](https://github.com/paperclipai/paperclip/blob/a35e78d076c1b618bc643391b7520fa6952fc996/doc/plans/2026-10-25-terminal-confirmation-sleep-audit.md#bound-retained-history-reads--october-28-review-repair).

- **New query-cost regression:** Capture the actual SQL emitted by
startup and company-scoped confirmation discovery. Seed 20,000
completed/unrelated action rows and one pending human decision. Before
the index, the regression fails because recovery examines 20,009 action
rows. After the partial index, startup and scoped discovery must examine
at most two action rows and fewer than 20 table/index buffers; the
no-pending scoped path must also stay below 20 buffers. Planner settings
are unchanged, and returned IDs remain correct.
- Match startup keyset ordering to `(company_id, id)`, the partial index
order. The index alone still let PostgreSQL choose the primary key and
scan 20,206 rows for `ORDER BY id`; the final query fixes this without
forcing a planner setting or adding a second index. Fixed kind/status
predicates stay literal for prepared-plan index eligibility.
- Add the generated `0398_yielding_random.sql` migration, using `CREATE
INDEX CONCURRENTLY IF NOT EXISTS` and the existing top-level migration
runner. Keep the five newest generated snapshots and preserve every
prior SQL/journal entry. No work table is introduced.
- **110 distinct cases pass:** 17 final confirmation
lifecycle/query-cost cases (1,671 skipped), seven real Runner descriptor
variants (295 skipped), and all 86
app/startup/bundled-plugin/plugin-database cases. Server/database
typechecking and builds, migration safety/numbering, module boundaries
and a fresh PostgreSQL migration through 0398 pass. The final test-only
vacuum precondition is followed by another 17-case confirmation pass and
server typecheck. Counts are not added across repeated runs. Normal
integration of master `644937c93150c423f12674992c7ea1e8b5dbc98c`
preserves the complete owned patch and all 18 unrelated master paths.
- Previous landed-parent acceptance: 29 distinct cases (16 confirmation
lifecycle/producer cases with 1,671 skipped, plus all 13 startup cases)
passed on `32e64a7e9ea5b457e54ca7b268be90e1e155b451`; the earlier
80-case matrix remains scoped to
`c1e8542b588a4fa59b62f9829541d3daea4f1ddc`. Counts are not added across
repeated runs.
- Actual producer cases prove parking/quiet, caller-owned outer
transaction rollback, commit-driven automatic settlement, restart/drain
and concurrent commits. Checkbox, suggestion, connection-intent and
Telegram/Slack callback paths retain existing authority and immutable
publication semantics.
- **Fresh exact-head CI and Greptile 5/5 with no unresolved actionable
findings are required.** The preceding head's Runner CI failed its
timeout-attachment durable-suspension case (1 failed/1,955 passed/4
skipped); that failure is retained and not counted as a pass. The new
head must pass this job. No full local repository, live provider or
physical suspension claim is made.

The owner audit retains the earlier missing-board-producer regressions
and their corrections, including the invalid supersession fixture that
assumed a new checkbox expires an older one. Existing behavior was
preserved by using the supported opt-in human-comment path. Prior
scheduler/backoff, route and plugin test failures remain at their
original revisions; no old pass is substituted for current remote gates.

## Risks

- Process-local notifications cover supported application writers and
their shared pools, not independent SQL writers or other replicas.
- Processing actions waiting on a human response are deliberately
parked. Supported interaction resolution/expiry writers must use the
notification boundary; a missing producer delays settlement until
restart.
- Invalid bindings have explicit failure evidence. Source, actor, task
and publication authority remain checked by their existing owners.
- Following continuation/GitHub duties remain separate. Cloud sleep
gates and external alarms are unchanged; local worker quiet does not
establish physical host/database suspension or fleet savings.

## Model Used

OpenAI GPT-6 / Codex, with reasoning, repository inspection, code
execution, and PostgreSQL tests. The exact runtime model identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-28 15:00:31 -05:00
1 parent 60b8089a44
commit ee67d2ece9
19 files changed
+1192 -39

No files matched your search

@@ -92,6 +92,8 @@ describe("createChatReconciliationCoordinator", () => {
chat.indexOf("async function schedulePendingPublications("));
expect(publicationMaintenance).not.toContain("await enqueueInboundWakeupPublications(");
expect(publicationMaintenance).not.toContain("await enqueueFailedChatRetryPublications(");
expect(publicationMaintenance).not.toContain("reconcileTerminalConfirmationActions(");
expect(source).toContain("chatChannels.registerTerminalConfirmationWork(deliveryWork,");
expect(source).toContain("chatFailedRetryNoticeWork(chatChannels).register(deliveryWork,");
});
@@ -41873,6 +41873,99 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
await service.shutdown();
});
it.each([
["request_checkbox_confirmation", "accept"],
["request_checkbox_confirmation", "reject"],
["request_checkbox_confirmation", "withdraw"],
["request_checkbox_confirmation", "supersede"],
["suggest_tasks", "accept"],
["suggest_tasks", "reject"],
["connection_intent", "reject"],
["connection_intent", "reassign"],
] as const)("wakes parked Slack %s actions after board %s commits", async (kind, decision) => {
const previousPublicUrl = process.env.PAPERCLIP_PUBLIC_URL;
process.env.PAPERCLIP_PUBLIC_URL = "https://paperclip.example";
const fixture = await seedCompany();
const { callbacks, endpoint, service } = await configuredSlackEndpoint(fixture);
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
try {
const thread = makeThread({ channelId: "C-REVIEW", id: "slack:C-REVIEW:4500.1", name: "reviews" });
await deliverMessage({ callbacks, endpointId: endpoint.id, thread: thread.thread, message: makeMessage({ id: "4500.1", text: "@maya review this change", mentioned: true, userId: `U-REVIEW-${randomUUID()}` }), trigger: "mention" });
await qualifySetupRoundTrip(service, endpoint.id);
await service.test(endpoint.id, "owner-user");
const [conversation] = await readChatConversationsByEndpointId(endpoint.id);
const sourceRunId = randomUUID();
await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId: fixture.companyId, agentId: fixture.assignedAgentId, status: "succeeded", responsibleUserId: "owner-user", contextSnapshot: { issueId: conversation.issueId, taskId: conversation.issueId, source: "automation" } });
const interactions = issueThreadInteractionService(db);
const issue = { id: conversation.issueId, companyId: fixture.companyId };
const interaction = kind === "connection_intent"
? await interactions.createConnectionIntent(issue, { sourceRunId, addresseeUserId: "owner-user", idempotencyKey: `slack-review:${sourceRunId}`, payload: { version: 1, serviceSlug: "notion", serviceName: "Notion", requestingAgentId: fixture.assignedAgentId, requestingAgentName: "Maya", phase: "requested" } })
: await interactions.create(issue, { kind, sourceRunId, payload: kind === "suggest_tasks" ? { version: 1, tasks: [{ clientKey: "docs", title: "Review migration notes" }] } : { version: 1, prompt: "Review migration notes", ...(kind === "request_checkbox_confirmation" ? { options: [{ id: "docs", label: "Migration notes" }], supersedeOnUserComment: decision === "supersede" } : {}) } } as Parameters<typeof interactions.create>[1], { agentId: fixture.assignedAgentId });
await service.processPendingPublications();
const rows = () => db.select().from(chatPublications).where(and(eq(chatPublications.endpointId, endpoint.id), eq(sql<string>`${chatPublications.payload}->>'interactionId'`, interaction.id)));
const [original] = await rows();
expect(original?.state).toBe("published");
expect(original?.payload.card?.actions).toEqual(expect.arrayContaining([expect.objectContaining({ type: "callback" })]));
const [action] = await db.select().from(chatActions).where(and(
eq(chatActions.endpointId, endpoint.id), eq(chatActions.kind, "confirmation_response"),
eq(sql<string>`${chatActions.payload}->>'interactionId'`, interaction.id),
eq(sql<string>`${chatActions.payload}->>'decision'`, decision === "accept" ? "accept" : "reject"),
));
expect(action).toBeDefined();
await db.update(chatActions).set({ status: "processing" }).where(eq(chatActions.id, action.id));
const errors: unknown[] = [];
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
canReconcile: () => true, onError: error => errors.push(error) });
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
const settle = async () => vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(10);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
const quiet = async () => {
const select = vi.spyOn(db, "select"), transaction = vi.spyOn(db, "transaction");
try {
await vi.advanceTimersByTimeAsync(120_000);
expect(select).not.toHaveBeenCalled(); expect(transaction).not.toHaveBeenCalled();
} finally { select.mockRestore(); transaction.mockRestore(); }
};
await service.registerTerminalConfirmationWork(coordinator, () => true).ready;
await settle(); await quiet();
expect((await db.select().from(chatActions).where(eq(chatActions.id, action.id)))[0].status).toBe("processing");
const resolve = async (tx: Db) => {
const board = issueThreadInteractionService(tx);
if (decision === "withdraw") await board.withdrawInteraction(issue, interaction.id, {}, { userId: "owner-user" });
else if (decision === "reassign") await board.expireConnectionIntentsForOwnershipChange(issue);
else if (decision === "supersede") {
const [comment] = await tx.insert(issueComments).values({ companyId: issue.companyId, issueId: issue.id,
authorUserId: "owner-user", body: "Use the revised migration plan", createdAt: new Date(Date.now() + 1000) }).returning();
await board.expireRequestConfirmationsSupersededByComment(issue, comment, { userId: "owner-user" });
}
else if (kind === "connection_intent") await board.resolveConnectionIntent(issue, interaction.id,
{ version: 1, outcome: "declined" }, { userId: "owner-user" });
else if (decision === "accept") await board.acceptInteraction(issue, interaction.id,
{ selectedOptionIds: ["docs"], selectedClientKeys: ["docs"] }, { userId: "owner-user" });
else await board.rejectInteraction(issue, interaction.id, {}, { userId: "owner-user" });
expect((await db.select().from(chatActions).where(eq(chatActions.id, action.id)))[0].status).toBe("processing");
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0].status).toBe("pending");
};
await expect(db.transaction(async tx => { await resolve(tx as unknown as Db); throw new Error("rollback board decision"); }))
.rejects.toThrow("rollback board decision");
await settle(); await quiet();
await db.transaction(tx => resolve(tx as unknown as Db));
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(10);
expect((await db.select().from(chatActions).where(eq(chatActions.id, action.id)))[0].status).toBe(["accept", "reject"].includes(decision) ? "processed" : "expired");
}, { timeout: 5_000 });
await settle(); await quiet();
expect(errors).toEqual([]);
} finally {
await coordinator?.stop(); vi.useRealTimers(); await service.shutdown();
if (previousPublicUrl === undefined) delete process.env.PAPERCLIP_PUBLIC_URL;
else process.env.PAPERCLIP_PUBLIC_URL = previousPublicUrl;
}
});
it.each(["request_confirmation", "request_checkbox_confirmation", "request_item_verdicts", "suggest_tasks", "connection_intent"] as const)("delivers and replaces a Slack %s card through the publication service", async kind => {
const previousPublicUrl = process.env.PAPERCLIP_PUBLIC_URL;
process.env.PAPERCLIP_PUBLIC_URL = "https://paperclip.example";
@@ -45407,7 +45500,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
expect(stillPending.status).toBe("pending");
});
it("renders safe Telegram confirmations as opaque one-use buttons and wakes the resolved task", async () => {
async function telegramConfirmationRecoveryFixture(decision: "accept" | "reject" = "accept") {
const fixture = await seedCompany();
let injectSettlementCrash = true;
const settlementBarrier = vi.fn(async () => {
@@ -45565,7 +45658,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
),
).toBe(true);
const acceptAction = confirmationActions.find(
(action) => action.payload.decision === "accept",
(action) => action.payload.decision === decision,
);
if (!acceptAction)
throw new Error("Telegram accept action was not persisted");
@@ -45597,7 +45690,286 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
value: undefined,
},
};
await expect(callbacks.onAction(actionEvent)).rejects.toThrow(
return { fixture, callbacks, onAction: callbacks.onAction, endpoint, service, wakeup, settlementBarrier, dm, conversation, linkedUserId, principal, interaction, publication, acceptAction, actionEvent };
}
it.each(["accept", "reject", "drain", "restart", "failed selection"] as const)("recovers committed terminal confirmations without polling through %s", async mode => {
const context = await telegramConfirmationRecoveryFixture(mode === "reject" ? "reject" : "accept");
let canRun = true;
const errors: unknown[] = [];
let coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun,
canReconcile: () => true, onError: error => errors.push(error) });
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
const settle = async () => vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(10);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
const quiet = async () => {
const select = vi.spyOn(db, "select"), transaction = vi.spyOn(db, "transaction");
try {
await vi.advanceTimersByTimeAsync(120_000);
expect(select).not.toHaveBeenCalled(); expect(transaction).not.toHaveBeenCalled();
} finally { select.mockRestore(); transaction.mockRestore(); }
};
try {
await context.service.registerTerminalConfirmationWork(coordinator, () => canRun).ready;
await settle(); await quiet();
if (["drain", "failed selection"].includes(mode)) canRun = false;
if (mode === "restart") await coordinator.stop();
await expect(context.onAction(context.actionEvent)).rejects.toThrow("injected post-interaction confirmation settlement crash");
if (["drain", "failed selection", "restart"].includes(mode)) {
await vi.advanceTimersByTimeAsync(1000);
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
expect(await db.select({ status: chatActions.status }).from(chatActions)
.where(eq(chatActions.id, context.acceptAction.id))).toEqual([{ status: "processing" }]);
await quiet();
if (mode === "failed selection") {
const select = vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("temporary confirmation selection failure"); });
try {
canRun = true;
await vi.advanceTimersByTimeAsync(Math.max(0, coordinator.nextWakeAt()! - Date.now()));
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(10); expect(errors).toHaveLength(1); });
canRun = false;
} finally { select.mockRestore(); }
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
await quiet();
}
canRun = true;
if (coordinator.nextWakeAt() !== null) await vi.advanceTimersByTimeAsync(Math.max(0, coordinator.nextWakeAt()! - Date.now()));
if (mode === "restart") {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun,
canReconcile: () => true, onError: error => errors.push(error) });
await context.service.registerTerminalConfirmationWork(coordinator, () => canRun).ready;
}
}
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(10);
expect(await db.select({ status: chatActions.status }).from(chatActions)
.where(eq(chatActions.id, context.acceptAction.id))).toEqual([{ status: "processed" }]);
}, { timeout: 10_000 });
await settle(); await quiet();
expect(await db.select({ id: chatActions.id }).from(chatActions).where(and(
eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "interaction_wakeup"),
eq(sql<string>`${chatActions.payload}->>'interactionId'`, context.interaction.id),
))).toHaveLength(1);
expect(await db.select({ id: activityLog.id }).from(activityLog).where(and(
eq(activityLog.companyId, context.fixture.companyId),
eq(sql<string>`${activityLog.details}->>'interactionId'`, context.interaction.id),
eq(sql<string>`${activityLog.details}->>'source'`, "external_chat"),
))).toHaveLength(1);
expect(errors).toEqual(mode === "failed selection" ? [expect.objectContaining({
errors: [expect.objectContaining({ message: "temporary confirmation selection failure" })],
})] : []);
} finally { await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
});
it("parks pending confirmation history, passes malformed prefixes, and wakes on the outer interaction commit", async () => {
const waiting = await telegramConfirmationRecoveryFixture();
const terminal = await telegramConfirmationRecoveryFixture();
const pendingIds = Array.from({ length: 101 }, (_, index) => `00000000-0000-4000-8000-${String(index + 1).padStart(12, "0")}`);
const invalidIds = Array.from({ length: 101 }, (_, index) => `00000000-0000-4000-8001-${String(index + 1).padStart(12, "0")}`);
await db.insert(chatActions).values([...pendingIds, ...invalidIds].map(id => ({
...waiting.acceptAction, id, providerActionId: `confirmation-history:${id}`, status: "processing",
conversationId: invalidIds.includes(id) ? null : waiting.conversation.id, result: null,
})));
await expect(terminal.onAction(terminal.actionEvent)).rejects.toThrow("injected post-interaction confirmation settlement crash");
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
canReconcile: () => true, onError: error => errors.push(error) });
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
const settle = async () => vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(10);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
}, { timeout: 30_000 });
const quiet = async () => {
const select = vi.spyOn(db, "select"), transaction = vi.spyOn(db, "transaction");
try {
await vi.advanceTimersByTimeAsync(120_000);
expect(select).not.toHaveBeenCalled(); expect(transaction).not.toHaveBeenCalled();
} finally { select.mockRestore(); transaction.mockRestore(); }
};
try {
await waiting.service.registerTerminalConfirmationWork(coordinator, () => true).ready;
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(10);
expect(await db.select({ status: chatActions.status }).from(chatActions)
.where(eq(chatActions.id, terminal.acceptAction.id))).toEqual([{ status: "processed" }]);
}, { timeout: 10_000 });
await settle();
expect(await db.select({ id: chatActions.id }).from(chatActions)
.where(and(inArray(chatActions.id, pendingIds), eq(chatActions.status, "processing")))).toHaveLength(101);
expect(await db.select({ id: chatActions.id }).from(chatActions).where(and(
inArray(chatActions.id, invalidIds), eq(chatActions.status, "failed"),
eq(sql<string>`${chatActions.result}->>'code'`, "confirmation_action_binding_invalid"),
))).toHaveLength(101);
await quiet();
const [issue] = await db.select().from(issues).where(eq(issues.id, waiting.conversation.issueId));
await expect(db.transaction(async tx => {
await issueThreadInteractionService(tx as unknown as Db).acceptInteraction(issue, waiting.interaction.id, {}, { userId: waiting.linkedUserId });
expect(await db.select({ status: issueThreadInteractions.status }).from(issueThreadInteractions)
.where(eq(issueThreadInteractions.id, waiting.interaction.id))).toEqual([{ status: "pending" }]);
throw new Error("rollback outer confirmation decision");
})).rejects.toThrow("rollback outer confirmation decision");
await settle(); await quiet();
expect(await db.select({ id: chatActions.id }).from(chatActions)
.where(and(inArray(chatActions.id, pendingIds), eq(chatActions.status, "processing")))).toHaveLength(101);
await db.transaction(async tx => {
await issueThreadInteractionService(tx as unknown as Db).acceptInteraction(issue, waiting.interaction.id, {}, { userId: waiting.linkedUserId });
expect(await db.select({ status: issueThreadInteractions.status }).from(issueThreadInteractions)
.where(eq(issueThreadInteractions.id, waiting.interaction.id))).toEqual([{ status: "pending" }]);
expect(await db.select({ id: chatActions.id }).from(chatActions)
.where(and(inArray(chatActions.id, pendingIds), eq(chatActions.status, "processing")))).toHaveLength(101);
});
await settle(); await quiet();
expect(await db.select({ id: chatActions.id }).from(chatActions)
.where(and(inArray(chatActions.id, pendingIds), eq(chatActions.status, "processing")))).toHaveLength(0);
expect(await db.select({ id: chatActions.id }).from(chatActions)
.where(and(inArray(chatActions.id, pendingIds), eq(chatActions.status, "processed")))).toHaveLength(1);
expect(await db.select({ id: activityLog.id }).from(activityLog).where(and(
eq(activityLog.companyId, waiting.fixture.companyId),
eq(sql<string>`${activityLog.details}->>'interactionId'`, waiting.interaction.id),
eq(sql<string>`${activityLog.details}->>'source'`, "external_chat"),
))).toHaveLength(0);
expect(errors).toEqual([]);
} finally {
await coordinator.stop(); vi.useRealTimers();
await db.delete(chatActions).where(inArray(chatActions.id, [...pendingIds, ...invalidIds]));
}
});
it("bounds terminal confirmation recovery reads with completed history", async () => {
const waiting = await telegramConfirmationRecoveryFixture();
const prefix = `confirmation-plan:${randomUUID()}:`;
await db.execute(sql`
INSERT INTO chat_actions (company_id, endpoint_id, kind, provider_action_id, status)
SELECT ${waiting.fixture.companyId}::uuid, ${waiting.endpoint.id}::uuid,
CASE WHEN n <= 10000 THEN 'confirmation_response' ELSE 'unrelated_action' END,
${prefix} || n,
CASE WHEN n <= 10000 THEN 'processed' ELSE 'processing' END
FROM generate_series(1, 20000) AS n
`);
await db.update(chatActions).set({ status: "processing", result: null })
.where(eq(chatActions.id, waiting.acceptAction.id));
await db.execute(sql`VACUUM (ANALYZE) chat_actions`);
const statements: { sql: string; params: unknown[] }[] = [];
const select = db.select.bind(db);
const reads = vi.spyOn(db, "select").mockImplementation((...args) => {
const selection = select(...args), from = selection.from.bind(selection);
vi.spyOn(selection, "from").mockImplementation(table => {
const query = from(table);
if (table === chatActions) {
const execute = query.execute.bind(query);
vi.spyOn(query, "execute").mockImplementation(async () => {
const statement = query.toSQL();
if (statement.sql.includes("'confirmation_response'") && statement.sql.includes("'processing'")) {
statements.push(statement);
}
return execute();
});
}
return query;
});
return selection;
});
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
canReconcile: () => true, onError: error => errors.push(error) });
try {
await waiting.service.registerTerminalConfirmationWork(coordinator, () => true).ready;
await vi.waitFor(() => expect(statements.length).toBeGreaterThan(0));
const startup = statements[0];
const beforeHint = statements.length;
await db.transaction(async tx => {
await signalDatabaseWork(tx, DELIVERY_QUEUES.chatTerminalConfirmations,
JSON.stringify({ companyId: waiting.fixture.companyId }));
});
await vi.waitFor(() => expect(statements.length).toBeGreaterThan(beforeHint), { timeout: 5_000 });
const scoped = statements.at(-1)!;
expect(scoped.params).toContain(waiting.fixture.companyId);
await coordinator.stop();
reads.mockRestore();
for (const statement of [startup, scoped]) {
expect(await db.$client.unsafe(statement.sql, statement.params as Parameters<typeof db.$client.unsafe>[1]))
.toMatchObject([{ id: waiting.acceptAction.id }]);
const result = await db.$client.unsafe(
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${statement.sql}`, statement.params as Parameters<typeof db.$client.unsafe>[1],
);
const plan = result[0]["QUERY PLAN"][0].Plan;
type PlanNode = { Plans?: PlanNode[]; [key: string]: unknown };
const nodes = (node: PlanNode): PlanNode[] => [node, ...(node.Plans ?? []).flatMap(nodes)];
const actionReads = nodes(plan).filter(node => node["Relation Name"] === "chat_actions");
expect(actionReads).toHaveLength(1);
expect(Number(actionReads[0]["Actual Rows"]) + Number(actionReads[0]["Rows Removed by Filter"] ?? 0),
JSON.stringify({ params: statement.params, actionReads }))
.toBeLessThanOrEqual(2);
expect(Number(actionReads[0]["Shared Hit Blocks"]) + Number(actionReads[0]["Shared Read Blocks"]))
.toBeLessThan(20);
}
await db.update(chatActions).set({ status: "processed" }).where(eq(chatActions.id, waiting.acceptAction.id));
await db.execute(sql`ANALYZE chat_actions`);
expect(await db.$client.unsafe(scoped.sql, scoped.params as Parameters<typeof db.$client.unsafe>[1])).toHaveLength(0);
const empty = await db.$client.unsafe(
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${scoped.sql}`, scoped.params as Parameters<typeof db.$client.unsafe>[1],
);
expect(empty[0]["QUERY PLAN"][0].Plan["Shared Hit Blocks"]
+ empty[0]["QUERY PLAN"][0].Plan["Shared Read Blocks"]).toBeLessThan(20);
expect(errors).toEqual([]);
} finally {
await coordinator.stop(); reads.mockRestore();
await db.delete(chatActions).where(sql`${chatActions.providerActionId} LIKE ${`${prefix}%`}`);
}
});
it("retains a confirmation committed while another settlement is in flight", async () => {
const first = await telegramConfirmationRecoveryFixture();
const second = await telegramConfirmationRecoveryFixture();
await expect(first.onAction(first.actionEvent)).rejects.toThrow("injected post-interaction confirmation settlement crash");
let locked!: () => void, release!: () => void;
const lockReady = new Promise<void>(resolve => { locked = resolve; });
const held = new Promise<void>(resolve => { release = resolve; });
const transaction = db.transaction(async tx => {
await tx.select({ id: chatActions.id }).from(chatActions).where(eq(chatActions.id, first.acceptAction.id)).for("update");
locked(); await held;
});
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
canReconcile: () => true, onError: error => errors.push(error) });
let ready: Promise<unknown> | undefined;
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
try {
await lockReady;
ready = first.service.registerTerminalConfirmationWork(coordinator, () => true).ready;
await vi.waitFor(async () => {
const blocked = await db.execute(sql`select pid from pg_stat_activity
where datname = current_database() and wait_event_type = 'Lock'
and query like 'update "chat_actions"%'`);
expect(blocked.length).toBeGreaterThan(0);
}, { timeout: 10_000 });
await expect(second.onAction(second.actionEvent)).rejects.toThrow("injected post-interaction confirmation settlement crash");
expect(await db.select({ status: chatActions.status }).from(chatActions)
.where(eq(chatActions.id, second.acceptAction.id))).toEqual([{ status: "processing" }]);
release(); await transaction; await ready;
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(10);
expect(await db.select({ status: chatActions.status }).from(chatActions)
.where(inArray(chatActions.id, [first.acceptAction.id, second.acceptAction.id])))
.toEqual([{ status: "processed" }, { status: "processed" }]);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
const select = vi.spyOn(db, "select"), queryTransaction = vi.spyOn(db, "transaction");
try {
await vi.advanceTimersByTimeAsync(120_000);
expect(select).not.toHaveBeenCalled(); expect(queryTransaction).not.toHaveBeenCalled();
} finally { select.mockRestore(); queryTransaction.mockRestore(); }
expect(errors).toEqual([]);
} finally {
release(); await transaction; await ready; await coordinator.stop(); vi.useRealTimers();
}
});
it("renders safe Telegram confirmations as opaque one-use buttons and wakes the resolved task", async () => {
const { fixture, onAction, endpoint, service, wakeup, settlementBarrier, dm, conversation, linkedUserId, principal, interaction, publication, acceptAction, actionEvent } = await telegramConfirmationRecoveryFixture();
await expect(onAction(actionEvent)).rejects.toThrow(
"injected post-interaction confirmation settlement crash",
);
expect(settlementBarrier).toHaveBeenCalledTimes(1);
@@ -45776,7 +46148,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
),
)
.then((rows) => rows.length);
await callbacks.onAction(actionEvent);
await onAction(actionEvent);
await new Promise<void>((resolve) => setImmediate(resolve));
expect(dm.post).toHaveBeenCalledTimes(providerPostCountBeforeRedelivery);
await expect(
@@ -64915,6 +65287,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
await quiet(); canRun = true;
if (coordinator.nextWakeAt() !== null) await vi.advanceTimersByTimeAsync(Math.max(0, coordinator.nextWakeAt()! - Date.now()));
if (mode === "stopped") {
await context.service.processPublicationMaintenance();
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
@@ -64924,7 +65297,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
}
}
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(30_000);
await vi.advanceTimersByTimeAsync(10);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([expect.objectContaining({
payload: mode === "disabled"
? { text: "This retry was not started. Open the task in Paperclip for details.", progressState: "failed" }
@@ -1,3 +1,4 @@
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { enqueueIssueInteractionChatPublications } from "../services/chat-interaction-publications.js";
import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
@@ -20,16 +21,20 @@ import { idleWorkSnapshot } from "../services/task-admission.js";
import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";
import { reserveRunBudget } from "../services/budget-reservations.js";
const support = await getEmbeddedPostgresTestSupport();
const externalTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_SATISFIED_CONNECTION_TEST_DATABASE_URL");
const support = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
const describePostgres = support.supported ? describe : describe.skip;
describePostgres("already available connection requests", () => {
let db!: ReturnType<typeof createDb>;
let temp!: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
beforeAll(async () => {
temp = await startEmbeddedPostgresTestDatabase("paperclip-satisfied-connections-");
db = createDb(temp.connectionString);
await db.insert(authUsers).values({ id: "connection-owner", name: "Owner", email: "owner@example.test", createdAt: new Date(), updatedAt: new Date() });
if (externalTestDatabaseUrl) db = createDb(externalTestDatabaseUrl);
else {
temp = await startEmbeddedPostgresTestDatabase("paperclip-satisfied-connections-");
db = createDb(temp.connectionString);
}
await db.insert(authUsers).values({ id: "connection-owner", name: "Owner", email: "owner@example.test", createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing();
}, 20_000);
beforeEach(async () => { await db.execute(sql`truncate table companies cascade`); });
afterAll(async () => { await db.$client.end({ timeout: 1 }); await temp?.cleanup(); });
@@ -60,6 +65,21 @@ describePostgres("already available connection requests", () => {
return run!;
}
it("notifies terminal confirmation recovery after existing-access retirement commits", async () => {
const f = await seed();
const notified = vi.fn();
const unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.chatTerminalConfirmations, notified);
try {
await db.transaction(async tx => {
expect(await satisfiedConnectionIntentService(tx as unknown as typeof db).sweepPending()).toMatchObject({ satisfied: 1 });
expect(notified).not.toHaveBeenCalled();
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, f.intent.id)))[0].status).toBe("pending");
});
expect(notified).toHaveBeenCalledOnce();
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, f.intent.id)))[0].status).toBe("expired");
} finally { unsubscribe(); }
});
it("settles a published Slack card when existing access retires the connection request", async () => {
const f = await seed();
const endpointId = randomUUID();
@@ -378,7 +398,7 @@ describePostgres("already available connection requests", () => {
it("records a system completion and one durable continuation with a one-connection pool", async () => {
const f = await seed();
const single = createDb(temp.connectionString, { maxConnections: 1 });
const single = createDb(externalTestDatabaseUrl ?? temp.connectionString, { maxConnections: 1 });
try {
const outcomes = await Promise.all([satisfiedConnectionIntentService(single).sweepPending(), satisfiedConnectionIntentService(db).sweepPending()]);
expect(outcomes.reduce((sum, outcome) => sum + outcome.satisfied, 0)).toBe(1);
+1
View File
@@ -1288,6 +1288,7 @@ export async function createApp(
chatChannels.registerReceiverWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTeamsConsentExpiryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTeamsAttemptRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTerminalConfirmationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerCommandWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
nativeChatProgressWork(db).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatInboundWakeNoticeWork(chatChannels).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
+2 -2
View File
@@ -3350,7 +3350,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
async function processPublicationMaintenance(limit = 25) {
scheduleGitHubMaintenance();
await reconcileTerminalConfirmationActions(limit);
await processPendingInteractionWakeups(limit);
}
@@ -3479,7 +3478,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
const { processPendingInteractionWakeups } = createChatInteractionContinuations(db, {
redactError, CONFIRMATION_WAKEUP_RETRY_BACKOFF_MS, get heartbeat() { return options.heartbeat; },
});
const { handleAction, handleModalSubmit, reconcileTerminalConfirmationActions } = createChatInteractionCallbacks(db, {
const { handleAction, handleModalSubmit, reconcileTerminalConfirmationActions, registerTerminalConfirmationWork } = createChatInteractionCallbacks(db, {
heartbeat: options.heartbeat,
CAPABILITIES, MAX_ERROR_TEXT, actionThreadMatchesConversation, nonDirectDestinationAllowed, sameTeamsThreadIdentity,
ensurePrincipal, endpointRecord, runtimeCallbackRecord, runtimeCallbackEndpoint,
@@ -3503,6 +3502,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
const { prepareTeamsBoardSend } = createTeamsBoardSendPreparation(db, { endpointRecord, runtimeFor });
const { getPublicationBatchStatus, publishComment, publishBoardMessage } = createChatPublicationRequests(db, { endpointRecord, stageTeamsBoardFileIntent, processPendingSlackBoardMessages, processPendingPublications, prepareTeamsBoardSend });
return {
registerTerminalConfirmationWork,
loadFailedRetryNoticeScopes,
reconcileFailedRetryNotices,
registerTeamsAttemptRecoveryWork,
@@ -145,7 +145,7 @@ type InteractionCallbacksOptions = {
export function createChatInteractionCallbacks(db: Db, options: InteractionCallbacksOptions) {
const { CAPABILITIES, MAX_ERROR_TEXT, actionThreadMatchesConversation, nonDirectDestinationAllowed, sameTeamsThreadIdentity, ensurePrincipal, endpointRecord, runtimeCallbackRecord, runtimeCallbackEndpoint, requireCurrentExternalActionAuthorization, isExternalActionAuthorizationChange, recordCurrentMicrosoftTeamsRoute, lockTelegramCallbackNoticeAuthority, stageProviderEffect, scheduleProviderEffect, invalidateRuntime, redactError, persistence, questionResponses, scheduleMessageProcessing, processPendingPublications } = options;
const { handleDiscordFormCorrection, acceptDiscordQuestionForm } = createDiscordQuestionForms(db, { requireCurrentExternalActionAuthorization, isExternalActionAuthorizationChange, persistence });
const { reconcileTerminalConfirmationActions, handleChatConfirmation } = createChatConfirmationActions(db, {
const { reconcileTerminalConfirmationActions, handleChatConfirmation, registerTerminalConfirmationWork } = createChatConfirmationActions(db, {
heartbeat: options.heartbeat, requireCurrentExternalActionAuthorization, isExternalActionAuthorizationChange, scheduleMessageProcessing, processPendingPublications,
get confirmationResolutionPersistBarrier() { return options.confirmationResolutionPersistBarrier; },
});
@@ -2029,5 +2029,5 @@ export function createChatInteractionCallbacks(db: Db, options: InteractionCallb
});
return acceptedResponse();
}
return { handleAction, handleModalSubmit, reconcileTerminalConfirmationActions };
return { handleAction, handleModalSubmit, reconcileTerminalConfirmationActions, registerTerminalConfirmationWork };
}
@@ -3,12 +3,20 @@ import { queueIssueAssignmentWakeup, type IssueAssignmentWakeupDeps } from "../.
import { chatInteractionPresentation } from "../../chat-interaction-presentation.js";
import type { DbTransaction, RuntimeContext, LifecycleRuntimeFence } from "../runtime-types.js";
import type { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js";
import { registerScopedDeadlineWork } from "../../scoped-deadline-work.js";
import { DELIVERY_QUEUES } from "../../delivery-work-notifications.js";
import { notifyChatConfirmationWork, type ChatConfirmationScope } from "../../chat-confirmation-notifications.js";
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
import {
and,
asc,
eq,
gt,
isNull,
or,
inArray,
ne,
sql,
@@ -176,6 +184,7 @@ export function createChatConfirmationActions(db: Db, options: ChatConfirmationO
});
},
afterResolveInTransaction: async (tx: DbTransaction, resolved: import("@paperclipai/shared").IssueThreadInteraction) => {
await notifyChatConfirmationWork(tx, { companyId: action.companyId, issueId: conversation.issueId });
const [owned] = await tx
.update(chatActions)
.set({
@@ -286,8 +295,13 @@ export function createChatConfirmationActions(db: Db, options: ChatConfirmationO
)
.then((rows) => rows[0] ?? null);
if (action?.status === "processed") return true;
if (!action || action.status !== "processing" || !action.conversationId)
if (!action || action.status !== "processing") return false;
if (!action.conversationId) {
await db.update(chatActions).set({
status: "failed", result: { code: "confirmation_action_binding_invalid" }, updatedAt: new Date(),
}).where(and(eq(chatActions.id, action.id), eq(chatActions.status, "processing")));
return false;
}
const token = action.payload;
if (
token.version !== 1 ||
@@ -647,5 +661,44 @@ export function createChatConfirmationActions(db: Db, options: ChatConfirmationO
}
return settled;
}
return { settleTerminalConfirmationAction, reconcileTerminalConfirmationActions, handleChatConfirmation };
function registerTerminalConfirmationWork(
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
canRun: () => boolean,
) {
type Entry = { id: string; companyId: string; issueId: string | null; dueAt: number };
return registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.chatTerminalConfirmations,
decodeScope: (hint: string): ChatConfirmationScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (entry: Entry, scope?: ChatConfirmationScope) => !scope ||
(entry.companyId === scope.companyId && (!scope.issueId || !entry.issueId || entry.issueId === scope.issueId)),
load: async (scope?: ChatConfirmationScope, signal?: AbortSignal): Promise<Entry[]> => {
const entries: Entry[] = [];
let cursor: { companyId: string; id: string } | undefined;
while (!signal?.aborted) {
const rows = await db.select({
id: chatActions.id, companyId: chatActions.companyId, issueId: chatConversations.issueId,
}).from(chatActions).leftJoin(chatConversations, and(
eq(chatConversations.companyId, chatActions.companyId), eq(chatConversations.id, chatActions.conversationId),
)).where(and(
sql`${chatActions.kind} = 'confirmation_response' AND ${chatActions.status} = 'processing'`,
scope ? eq(chatActions.companyId, scope.companyId) : undefined,
scope?.issueId ? or(eq(chatConversations.issueId, scope.issueId), isNull(chatConversations.id)) : undefined,
cursor ? or(
gt(chatActions.companyId, cursor.companyId),
and(eq(chatActions.companyId, cursor.companyId), gt(chatActions.id, cursor.id)),
) : undefined,
)).orderBy(asc(chatActions.companyId), asc(chatActions.id)).limit(100);
entries.push(...rows.map(row => ({ ...row, dueAt: 0 })));
if (rows.length < 100) return entries;
cursor = rows[rows.length - 1];
}
throw signal?.reason ?? new Error("Confirmation recovery stopped");
},
check: async (entry: Entry) => {
await settleTerminalConfirmationAction(entry.id);
return null;
},
}, canRun);
}
return { settleTerminalConfirmationAction, reconcileTerminalConfirmationActions, handleChatConfirmation, registerTerminalConfirmationWork };
}
@@ -0,0 +1,16 @@
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export type ChatConfirmationScope = { companyId: string; issueId?: string };
export async function notifyChatConfirmationWork(tx: object, scope: ChatConfirmationScope) {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatTerminalConfirmations,
JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }));
}
/** Interaction resolution can release a callback parked while its human input
* was pending. Register on the owning transaction before its resolution write. */
export async function notifyChatConfirmationResolution(tx: object, interaction: ChatConfirmationScope & { kind: string }) {
if (["request_confirmation", "request_checkbox_confirmation", "suggest_tasks", "connection_intent"].includes(interaction.kind)) {
await notifyChatConfirmationWork(tx, interaction);
}
}
@@ -1,3 +1,4 @@
import { notifyChatConfirmationWork } from "./chat-confirmation-notifications.js";
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
import { notifyChatNativeProgress } from "./chat-native-progress-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
@@ -13,4 +14,5 @@ export async function notifyChatRunMilestones(tx: object, scope: ChatRunMileston
await notifyChatNativeProgress(tx, scope);
// Admission and current chat authority also govern saved queue notices.
await notifyChatInboundWakeNotices(tx, scope);
await notifyChatConfirmationWork(tx, scope);
}
@@ -59,6 +59,7 @@ export const DELIVERY_QUEUES = {
chatPublications: "chat-publications",
chatNativeProgress: "chat-native-progress",
chatInboundWakeNotices: "chat-inbound-wake-notices",
chatTerminalConfirmations: "chat-terminal-confirmations",
chatFailedRetryNotices: "chat-failed-retry-notices",
chatRunMilestones: "chat-run-milestones",
chatReceipts: "chat-slack-receipts",
@@ -1,3 +1,4 @@
import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyToolReviewChange } from "./tool-review-work-notifications.js";
import { normalizeEscapedLineBreaks } from "@paperclipai/shared/validators/text";
@@ -1913,6 +1914,7 @@ async function expireStaleRequestConfirmationTarget(
snapshot,
});
const expireInTransaction = async (tx: Db | any) => {
await notifyChatConfirmationResolution(tx, args.row);
if (args.row.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.row);
const [updated] = await tx
.update(issueThreadInteractions)
@@ -2191,6 +2193,7 @@ export function issueThreadInteractionService(
const postCommitActivityPublications: ActivityPublication[] = [];
const result = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor);
await notifyChatConfirmationResolution(tx, args.current);
if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current);
await args.mutationOptions?.beforeResolveInTransaction?.(tx);
// Policy mutations and review transitions use the same issue-row lock,
@@ -2475,6 +2478,7 @@ export function issueThreadInteractionService(
const now = new Date();
const updated = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor);
await notifyChatConfirmationResolution(tx, args.current);
if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current);
await args.mutationOptions?.beforeResolveInTransaction?.(tx);
const issueContext = await tx
@@ -2762,6 +2766,7 @@ export function issueThreadInteractionService(
})
.map((candidate) => candidate.id);
if (supersededIds.length > 0) {
await notifyChatConfirmationResolution(tx, { companyId: issue.companyId, issueId: issue.id, kind: "connection_intent" });
await tx
.delete(toolOauthStates)
.where(inArray(toolOauthStates.interactionId, supersededIds));
@@ -2860,6 +2865,7 @@ export function issueThreadInteractionService(
: "expired";
const resolvedAt = now();
const updated = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, current);
await mutationOptions.beforeResolveInTransaction?.(tx);
const [row] = await tx
.update(issueThreadInteractions)
@@ -3294,6 +3300,7 @@ export function issueThreadInteractionService(
const cancelled: IssueThreadInteraction[] = [];
for (const row of rows) {
const updated = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, row);
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]);
const [resolved] = await tx
@@ -3380,6 +3387,7 @@ export function issueThreadInteractionService(
const expired: IssueThreadInteraction[] = [];
for (const { row, replacementInteractionId } of supersededRows) {
const updated = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, row);
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]);
const [updatedRow] = await tx
@@ -4071,6 +4079,7 @@ export function issueThreadInteractionService(
const createdWakeTargets: IssueWakeTarget[] = [];
await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, current);
await mutationOptions.beforeResolveInTransaction?.(tx);
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
const resolvedAt = new Date();
@@ -4377,6 +4386,7 @@ export function issueThreadInteractionService(
}
const [updated] = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, current);
await mutationOptions.beforeResolveInTransaction?.(tx);
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
const rows = await tx.update(issueThreadInteractions)
@@ -4468,6 +4478,7 @@ export function issueThreadInteractionService(
const expired: IssueThreadInteraction[] = [];
for (const row of superseded) {
const updated = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, row);
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
if (row.kind === "connection_intent") {
await tx
@@ -4644,6 +4655,7 @@ export function issueThreadInteractionService(
);
if (!sampleConfirmationRow) continue;
const updatedInteractions = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, sampleConfirmationRow);
if (commentRows.some(row => row.kind === "request_confirmation")) await notifyChatRunMilestones(tx, sampleConfirmationRow);
await notifyToolProjectionForInteractions(tx, issue.companyId, confirmationRowIds);
const updatedRows = await tx
@@ -4705,6 +4717,7 @@ export function issueThreadInteractionService(
for (const row of connectionIntentRows) {
const [updated] = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, row);
await tx
.delete(toolOauthStates)
.where(eq(toolOauthStates.interactionId, row.id));
@@ -4790,6 +4803,7 @@ export function issueThreadInteractionService(
document,
});
const updated = await db.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, row);
if (row.kind === "request_confirmation") await notifyChatRunMilestones(tx, row);
const [resolved] = await tx
.update(issueThreadInteractions)
@@ -4853,27 +4867,29 @@ export function issueThreadInteractionService(
const subject = payload?.purpose === "ai"
? "the AI account"
: `the ${payload?.serviceName?.trim() || "requested"} connection`;
const [updated] = await db
.update(issueThreadInteractions)
.set({
status: "expired",
result: {
version: 1,
outcome: "expired",
reason: `The task was reassigned before the connection request was answered, so ${subject} is still not connected. Reconnect it or choose an available connection.`,
},
resolvedAt: now(),
updatedAt: now(),
})
.where(and(
eq(issueThreadInteractions.id, row.id),
eq(issueThreadInteractions.status, "pending"),
))
.returning();
if (updated) {
await enqueueTerminalIssueInteractionChatPublications(db, hydrateInteraction(updated));
expired.push(updated);
}
const updated = await db.transaction(async tx => {
await notifyChatConfirmationResolution(tx, { companyId: issue.companyId, issueId: issue.id, kind: "connection_intent" });
const [resolved] = await tx
.update(issueThreadInteractions)
.set({
status: "expired",
result: {
version: 1,
outcome: "expired",
reason: `The task was reassigned before the connection request was answered, so ${subject} is still not connected. Reconnect it or choose an available connection.`,
},
resolvedAt: now(),
updatedAt: now(),
})
.where(and(
eq(issueThreadInteractions.id, row.id),
eq(issueThreadInteractions.status, "pending"),
))
.returning();
if (resolved) await enqueueTerminalIssueInteractionChatPublications(tx as unknown as Db, hydrateInteraction(resolved));
return resolved;
});
if (updated) expired.push(updated);
}
if (expired.length) await db.delete(toolOauthStates).where(inArray(toolOauthStates.interactionId, expired.map((item) => item.id)));
return expired;
@@ -4911,6 +4927,7 @@ export function issueThreadInteractionService(
// lifecycle reflection.
const updated = await db
.transaction(async (tx) => {
await notifyChatConfirmationResolution(tx, row);
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
if (row.kind === "connection_intent") {
await tx
@@ -5002,6 +5019,7 @@ export function issueThreadInteractionService(
// request must not outlive a withdrawn card.
const updated = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
await notifyChatConfirmationResolution(tx, current);
if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current);
await resolveLinkedToolActionRequests(tx, current, {
status: "cancelled",
@@ -5263,6 +5281,7 @@ export function issueThreadInteractionService(
const now = new Date();
const updated = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
await notifyChatConfirmationResolution(tx, current);
if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current);
await resolveLinkedToolActionRequests(tx, current, {
status: "cancelled",
@@ -1,3 +1,4 @@
import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js";
import { enqueueTerminalIssueInteractionChatPublications } from "./chat-interaction-publications.js";
import { connectionManagementService } from "./connection-management.js";
import { chatCredentialEntryGenerations } from "./chat-channels/credentials.js";
@@ -156,6 +157,7 @@ export function satisfiedConnectionIntentService(db: Db) {
if (!connection) return false;
const now = new Date();
// Retire the obsolete request without inventing a human approval.
await notifyChatConfirmationResolution(tx, intent);
const [retired] = await tx.update(issueThreadInteractions).set({ status: "expired", resolvedAt: now, updatedAt: now,
result: { version: 1, outcome: "expired", connectionId: connection.id,
reason: "An already authorized connection is available to the requesting agent." },