Files
PaperClipAI/server/src/services/chat-completion-delivery.ts
T
DottaandClaude Opus 5.5 9b581d689b fix(chat): report an outcome once and create no run when an earlier reply gave it (#21489)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In agent chat, a finished delegated task or a declined tool action
wakes the chat agent so it tells the person
> - #21401 stopped a second completion report when an earlier reply
already gave it
> - But the suppressed completion still created a zero-second heartbeat
run, and declined tool actions still got a second report
> - Run accounting must count only real turns, and the person must get
each outcome once
> - This pull request settles an already-reported outcome before it
becomes a run, and applies the same rule to declined tool actions
> - The benefit is one report per outcome, no phantom run, and no extra
model call

## Linked Issues or Issue Description

Refs #21401

**What happened?**
Chat eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`)
showed three problems after #21401.
1. `questions-and-approvals.duplicate-answer`: the completion wake was
suppressed, but it still created a 0-second run (`runtimeMode: legacy`,
no usage). The bounded-native-runs check rejected that run.
2. `skills.studio-test`: the CEO's single reply reported the Studio
result after it read the test task done (0 passed, 1 failed, links
COM-2). The completion was then suppressed, but the eval waited for a
separate completion run and timed out after 240 s.
3. `questions-and-approvals.decline-tool`: the person declined the tool
review one second after the turn for their message started. That turn
read the declined action and said so. The decline continuation, which
waited behind that turn, then posted the same declined outcome again.

**Expected behavior**
Each finished task or declined tool action is reported once in the
conversation. An outcome that an earlier reply already reported creates
no heartbeat run.

**Steps to reproduce**
1. In chat, delegate a short task, or start a governed tool read that
needs approval.
2. While it waits, send a message that wakes the chat agent.
3. Let the task finish, or decline the tool review, while that turn
runs. The turn reads the outcome and reports it.
4. See the completion wake become a 0-second run, or see a second
decline report from the decline continuation.

**Paperclip version or commit**
master `df1b12ad8e`

## What Changed

- `server/src/services/chat-completion-delivery.ts`:
`settleReportedChatCompletions` marks each undispatched completion that
an earlier reply reported as delivered, with that reply as its response.
It returns true only when nothing else is owed. The outbox `deliver`
also checks this before it requests a wake, so no wake and no run exist
for an already-reported completion.
- `server/src/modules/wake-queue`: a new transaction port,
`skipAlreadyReportedChatWake`, runs in the release drain before a
deferred wake is promoted. When the earlier reply already reported every
outcome of the wake, the wake ends `skipped` ("Already reported by an
earlier reply in this conversation") and no run is created. Wakes that
carry queued comments or a failed-chat retry are never skipped.
- `server/src/services/chat-interaction-outcome-report.ts` (new): the
same structural rule for declined tool actions. A conversation agent run
that lists its interactions records `chat_interaction.declined_read` for
each declined tool action. The decline continuation is settled when the
same agent replied in the conversation after the decline, from a run
with that receipt, posted after the read. The rule covers only declined
tool-action confirmations. Answers, plan reviews and accepted actions
keep their own turn.
- `server/src/routes/issues.ts`: `GET /issues/:id/interactions` calls
the recorder for agent runs.
- The run-start check from #21401 stays as a fallback for a completion
turn that was admitted before the earlier reply landed.
- `tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.ts`:
the Studio oracle (v5) accepts either the completion turn's reply or an
earlier CEO reply as the delivery. The earlier reply counts only from
saved state: no completion run exists, the replying run has a
`chat_handoff.completed_read` receipt for the test task, the reply was
posted after that read, it is the only such reply, and it links the
saved result. No wording check is added.
- `doc/run-log-events.md`: documents both read receipts and the skipped
wake outcome.

## Verification

- `cd server && npx vitest run
src/__tests__/chat-completion-delivery.test.ts
src/modules/wake-queue/adapters/chat-declined-outcome-reported.test.ts
src/modules/wake-queue/application/use-cases.test.ts`: all pass.
- Red: with the drain check and the outbox check disabled, 5 tests fail:
"starts no completion wake or run ...", "settles a deferred completion
wake without a run ... (true)", "settles the decline continuation with
no run ...", and both "settles an already-reported chat wake without a
run" cases.
- Negative controls stay green: a linking reply without a done read, a
decline the earlier turn never read, a non-tool-action card, and an
accepted card all still run their continuation.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.test.ts`:
34 pass. Red on the old oracle: "accepts the earlier reply the server
took as the completion report" fails. New negative cases reject no done
read, a read of another task, a reply before the read, a reply from
another run, two replies, an extra completion run, and a missing task
link.
- A narrow paid eval run of `duplicate-answer`, `studio-test` and
`decline-tool` on this branch follows in a comment.

## Risks

- A completion turn admitted before the earlier reply landed (a narrow
race) still ends as a short succeeded run with no model call, as in
#21401.
- If an earlier reply read the outcome but did not mention it, the
person does not get a second message. The read receipt, the
reply-after-read order and, for completions, the task link make this
unlikely.
- The decline rule covers only declined tool-action confirmations. Other
interaction outcomes are unchanged.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:16 -06:00

442 lines
31 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
import { nativeResponsePresentationCandidates } from "./native-runtime/native-response-presentation-query.js";
import { and, asc, desc, eq, gte, inArray, isNotNull, isNull, lte, ne, sql } from "drizzle-orm";
import { agents, agentWakeupRequests, chatCompletionDeliveries as deliveries, chatTaskHandoffs as handoffs,
heartbeatRunEvents, heartbeatRuns, issueComments, issueDocuments, issueThreadInteractions, issues, type Db } from "@paperclipai/db";
import { extractIssueReferenceMatches } from "@paperclipai/shared";
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
import { instanceSettingsService } from "./instance-settings.js";
import { conflict, forbidden, notFound } from "../errors.js";
export const CHAT_COMPLETION_WAKE_REASON = "chat_task_completed";
/** Run event: the conversation agent read its handed-off task after it was done. */
export const COMPLETED_HANDOFF_READ_EVENT = "chat_handoff.completed_read";
const MAX_ATTEMPTS = 5;
const LEASE_MS = 60_000;
type Issue = typeof issues.$inferSelect;
type Run = typeof heartbeatRuns.$inferSelect;
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
type Connection = Db | Tx;
const pending = ["pending", "queued"] as const;
const activeRuns = ["queued", "scheduled_retry", "running"];
function ids(run: Run): string[] {
const value = run.contextSnapshot?.chatCompletionDeliveryIds;
return Array.isArray(value) ? value.filter((v): v is string => typeof v === "string") : [];
}
/** Human-confirmed suggestions use the same durable audience as native create_task.
* The source must be read/authorized by the caller; ownership and generation are
* rechecked under the creation transaction so reset chats cannot receive new work. */
export async function lockAcceptedChatHandoffSource(tx: Connection, input: {
companyId: string; conversationId: string; sessionGeneration: number;
userId?: string | null; agentId?: string | null;
}) {
const [source] = await tx.select().from(issues).where(and(eq(issues.id, input.conversationId),
eq(issues.companyId, input.companyId))).for("update");
if (!source?.conversationAgentId) throw notFound("Originating conversation not found");
if (source.conversationSessionGeneration !== input.sessionGeneration) {
throw conflict("The originating conversation was reset; request a new task suggestion");
}
if (input.userId ? source.conversationUserId !== input.userId : source.conversationAgentId !== input.agentId) {
throw forbidden("Task handoff requires the originating conversation's participant");
}
return source;
}
export async function recordAcceptedChatHandoff(tx: Connection, task: Issue, source: Issue) {
if (!source.conversationAgentId || source.companyId !== task.companyId || task.conversationAgentId) {
throw forbidden("Task handoff requires an execution task in the originating conversation's company");
}
await tx.insert(handoffs).values({ taskId: task.id, companyId: task.companyId,
conversationId: source.id, agentId: source.conversationAgentId,
sessionGeneration: source.conversationSessionGeneration }).onConflictDoNothing();
}
export async function recordChatHandoff(tx: Connection, task: Issue, actorRunId: string | null | undefined) {
if (!actorRunId || task.conversationAgentId || !task.createdByAgentId) return;
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, actorRunId),
eq(heartbeatRuns.companyId, task.companyId), eq(heartbeatRuns.agentId, task.createdByAgentId)));
const sourceId = run?.nativeIssueId ?? run?.contextSnapshot?.issueId;
if (typeof sourceId !== "string" || sourceId === task.id) return;
const [source] = await tx.select().from(issues).where(and(eq(issues.id, sourceId), eq(issues.companyId, task.companyId)));
if (!source?.conversationAgentId || source.conversationAgentId !== run.agentId ||
source.conversationSessionGeneration !== run.contextSnapshot?.conversationSessionGeneration) return;
await recordAcceptedChatHandoff(tx, task, source);
}
/** Must run on the same transaction as status projection (including native arbitration). */
export async function recordChatCompletion(tx: Connection, before: Issue, after: Issue): Promise<boolean> {
if (before.status === after.status) return false;
await tx.update(deliveries).set({ status: "superseded" }).where(and(eq(deliveries.taskId, after.id),
eq(deliveries.companyId, after.companyId), inArray(deliveries.status, [...pending])));
if (after.status !== "done") return false;
const [handoff] = await tx.select().from(handoffs).where(and(eq(handoffs.taskId, after.id), eq(handoffs.companyId, after.companyId)));
if (!handoff) return false;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatCompletion);
await tx.insert(deliveries).values({ companyId: after.companyId, taskId: after.id, statusVersion: after.statusVersion }).onConflictDoNothing();
return true;
}
async function loadAudience(tx: Connection, deliveryId: string) {
const [row] = await tx.select({ delivery: deliveries, handoff: handoffs, task: issues }).from(deliveries)
.innerJoin(handoffs, and(eq(handoffs.taskId, deliveries.taskId), eq(handoffs.companyId, deliveries.companyId)))
.innerJoin(issues, and(eq(issues.id, deliveries.taskId), eq(issues.companyId, deliveries.companyId)))
.where(eq(deliveries.id, deliveryId));
if (!row) return null;
const [source] = await tx.select().from(issues).where(and(eq(issues.id, row.handoff.conversationId), eq(issues.companyId, row.handoff.companyId)));
return { ...row, source };
}
function current(row: NonNullable<Awaited<ReturnType<typeof loadAudience>>>) {
return row.task.status === "done" && !["superseded", "exhausted"].includes(row.delivery.status) &&
row.source?.conversationAgentId === row.handoff.agentId &&
row.source.conversationSessionGeneration === row.handoff.sessionGeneration;
}
async function taskResult(tx: Connection, task: Issue) {
// Never copy worker-authored titles, comments or document bodies into the
// source agent's instructions. These links and lifecycle facts are generated
// by the server; the saved work remains on its normal access-controlled task.
const documents = await tx.select({ id: issueDocuments.documentId }).from(issueDocuments)
.where(and(eq(issueDocuments.companyId, task.companyId), eq(issueDocuments.issueId, task.id))).limit(1);
// A bounded receipt projection gives the coordinator actual verdicts to link,
// rather than asking it to infer whether the worker's prose needs another review.
// These are historical decisions on their named versions, not publication authority.
const reviews = await tx.select().from(issueThreadInteractions).where(and(
eq(issueThreadInteractions.companyId, task.companyId), eq(issueThreadInteractions.issueId, task.id),
inArray(issueThreadInteractions.kind, ["request_confirmation", "request_item_verdicts"]),
inArray(issueThreadInteractions.status, ["accepted", "answered"]),
)).orderBy(desc(issueThreadInteractions.resolvedAt), desc(issueThreadInteractions.id)).limit(20);
const recordedOutputDecisions = reviews.flatMap(review => {
const payload = review.payload as Record<string, any>, result = (review.result ?? {}) as Record<string, any>;
const candidates = review.kind === "request_confirmation" ? [{ target: payload.target, outcome: "approve", human: Boolean(review.resolvedByUserId) && !review.resolvedByAgentId }]
: (Array.isArray(result.items) ? result.items : []).map((item: any) => ({
target: (Array.isArray(payload.items) ? payload.items : []).find((entry: any) => entry.id === item.id)?.target ?? payload.target,
outcome: item.verdict, human: Boolean(item.resolvedByUserId) && !item.resolvedByAgentId,
}));
return candidates.filter(candidate => candidate.human && ["approve", "reject", "defer"].includes(candidate.outcome) &&
candidate.target && ["issue_document", "issue_attachment"].includes(candidate.target.type) &&
(!candidate.target.issueId || candidate.target.issueId === task.id) && candidate.target.key !== "plan")
.map(candidate => ({ interactionId: review.id, outcome: candidate.outcome,
target: candidate.target.type === "issue_document"
? { type: "issue_document", key: candidate.target.key, revisionId: candidate.target.revisionId }
: { type: "issue_attachment", attachmentId: candidate.target.attachmentId, assetId: candidate.target.assetId, sha256: candidate.target.sha256 },
url: `/issues/${task.identifier ?? task.id}#interaction-${review.id}` }));
}).slice(0, 30);
return { id: task.id, identifier: task.identifier, status: task.status,
completedAt: task.completedAt, url: `/issues/${task.identifier ?? task.id}`,
hasSavedDocuments: documents.length > 0, recordedOutputDecisions };
}
/** A completed onboarding parent still owes the result of its own child handoff.
* This permits a reporting turn without reopening Done or reviving cancellation. */
export async function isCompletedOnboardingHandoffWake(db: Connection, input: {
companyId: string; issueId: string; agentId: string; reason: string | null;
contextSnapshot: Record<string, unknown>;
}) {
if (input.reason !== "issue_children_completed" || typeof input.contextSnapshot.completedChildIssueId !== "string") return false;
const [source] = await db.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId)));
if (source?.originKind !== "onboarding_first_task" || source.status !== "done" || source.assigneeAgentId !== input.agentId) return false;
const children = await db.select({ id: issues.id, status: issues.status }).from(issues)
.where(and(eq(issues.companyId, input.companyId), eq(issues.parentId, source.id)));
return children.some(child => child.id === input.contextSnapshot.completedChildIssueId && child.status === "done") &&
children.every(child => ["done", "cancelled"].includes(child.status));
}
/** Records that the conversation agent read a handed-off task in its done state. The
* completion turn uses this to recognize a completion that an earlier reply reported. */
export async function recordCompletedHandoffRead(db: Db, input: {
companyId: string; agentId: string; runId: string;
task: { id: string; status: string; statusVersion?: number | null };
}) {
if (input.task.status !== "done" || typeof input.task.statusVersion !== "number") return;
const [handoff] = await db.select({ agentId: handoffs.agentId }).from(handoffs)
.where(and(eq(handoffs.taskId, input.task.id), eq(handoffs.companyId, input.companyId)));
if (handoff?.agentId !== input.agentId) return;
await appendHeartbeatRunEvent(db, { companyId: input.companyId, runId: input.runId, agentId: input.agentId,
eventType: COMPLETED_HANDOFF_READ_EVENT, stream: "system", level: "info",
payload: { taskId: input.task.id, statusVersion: input.task.statusVersion } });
}
/** A reply in this conversation already reported this completion when a turn of the same
* agent read the task done for this completion and then linked the task in its reply. */
async function earlierCompletionReply(tx: Connection, row: NonNullable<Awaited<ReturnType<typeof loadAudience>>>, runId: string | null) {
const { task, handoff, delivery } = row;
if (!task.completedAt || !task.identifier) return null;
// A reopen supersedes this delivery and a new completion gets a higher status
// version, so any done read at or after this version belongs to this completion,
// including after an owner change that advanced the version while still done.
const readThisCompletion = sql`exists (select 1 from ${heartbeatRunEvents} handoff_read
where handoff_read.company_id = ${handoff.companyId} and handoff_read.run_id = ${issueComments.createdByRunId}
and handoff_read.event_type = ${COMPLETED_HANDOFF_READ_EVENT} and handoff_read.payload->>'taskId' = ${task.id}
and (handoff_read.payload->>'statusVersion')::bigint >= ${delivery.statusVersion}
and handoff_read.created_at <= ${issueComments.createdAt})`;
const replies = await tx.select({ id: issueComments.id, body: issueComments.body }).from(issueComments).where(and(
eq(issueComments.companyId, handoff.companyId), eq(issueComments.issueId, handoff.conversationId),
eq(issueComments.authorAgentId, handoff.agentId), isNull(issueComments.deletedAt),
isNotNull(issueComments.createdByRunId), runId ? ne(issueComments.createdByRunId, runId) : undefined,
gte(issueComments.createdAt, task.completedAt), readThisCompletion,
)).orderBy(asc(issueComments.createdAt), asc(issueComments.id)).limit(20);
const identifier = task.identifier.toUpperCase();
return replies.find(reply => extractIssueReferenceMatches(reply.body)
.some(match => match.identifier.toUpperCase() === identifier))?.id ?? null;
}
/** Marks one delivery delivered by the earlier reply that already reported it. */
async function deliverByEarlierReply(tx: Connection, row: NonNullable<Awaited<ReturnType<typeof loadAudience>>>, runId: string | null) {
const reportedBy = await earlierCompletionReply(tx, row, runId);
if (!reportedBy) return false;
// An earlier turn already told the person; repeating it would post the same news twice.
await tx.update(deliveries).set({ status: "delivered", responseCommentId: reportedBy, error: null })
.where(and(eq(deliveries.id, row.delivery.id), inArray(deliveries.status, [...pending])));
return true;
}
/** Before a completion wake becomes a run: settles every undispatched completion of this
* conversation that an earlier reply already reported. True only when at least one was
* reported and none is left for a completion turn, so the wake needs no run at all. */
export async function settleReportedChatCompletions(tx: Connection, input: { companyId: string; conversationId: string; agentId: string }) {
const [source] = await tx.select().from(issues).where(and(eq(issues.id, input.conversationId), eq(issues.companyId, input.companyId))).for("update");
if (!source || source.conversationAgentId !== input.agentId) return false;
const rows = await tx.select({ id: deliveries.id }).from(deliveries).innerJoin(handoffs, eq(handoffs.taskId, deliveries.taskId))
.where(and(eq(deliveries.companyId, input.companyId), eq(handoffs.conversationId, source.id), eq(handoffs.agentId, input.agentId),
eq(handoffs.sessionGeneration, source.conversationSessionGeneration), inArray(deliveries.status, [...pending]), isNull(deliveries.targetRunId)))
.orderBy(asc(deliveries.createdAt)).limit(21);
if (rows.length === 0 || rows.length > 20) return false;
let reported = 0, remaining = 0;
for (const { id } of rows) {
const row = await loadAudience(tx, id);
if (!row || !current(row)) {
await tx.update(deliveries).set({ status: "superseded" }).where(eq(deliveries.id, id));
continue;
}
if (await deliverByEarlierReply(tx, row, null)) reported++; else remaining++;
}
return reported > 0 && remaining === 0;
}
/** Freeze the input at turn start. New completions cannot be consumed by an already-running turn. */
export async function prepareChatCompletionTurn(db: Db, run: Run): Promise<Run> {
const issueId = run.contextSnapshot?.issueId;
if (typeof issueId !== "string") return run;
return db.transaction(async tx => {
const [source] = await tx.select().from(issues).where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId))).for("update");
if (!source) return run;
if (source.originKind === "onboarding_first_task" && run.contextSnapshot?.wakeReason === "issue_children_completed") {
const children = await tx.select().from(issues).where(and(eq(issues.companyId, run.companyId), eq(issues.parentId, source.id), eq(issues.status, "done"))).orderBy(issues.createdAt).limit(20);
const contextSnapshot = { ...run.contextSnapshot, onboardingCompletion: true,
chatCompletionUpdates: await Promise.all(children.map(child => taskResult(tx, child))) };
await tx.update(heartbeatRuns).set({ contextSnapshot }).where(eq(heartbeatRuns.id, run.id));
return { ...run, contextSnapshot };
}
const initialIds = ids(run);
// Only completion wakes absorb pending events; ordinary user turns retain their existing input.
if (initialIds.length === 0 && run.contextSnapshot?.wakeReason !== CHAT_COMPLETION_WAKE_REASON) return run;
if (run.contextSnapshot?.conversationSessionGeneration !== source.conversationSessionGeneration) throw new Error("chat_completion_superseded");
const rows = await tx.select({ delivery: deliveries }).from(deliveries).innerJoin(handoffs, eq(handoffs.taskId, deliveries.taskId))
.where(and(eq(deliveries.companyId, run.companyId), eq(handoffs.conversationId, issueId),
eq(handoffs.agentId, run.agentId), eq(handoffs.sessionGeneration, source.conversationSessionGeneration),
inArray(deliveries.status, [...pending]),
sql`(${deliveries.targetRunId} is null or ${deliveries.targetRunId} = ${run.id})`))
.orderBy(asc(deliveries.createdAt)).limit(20);
const updates: Awaited<ReturnType<typeof taskResult>>[] = [];
const accepted: string[] = [];
const reported: string[] = [];
for (const { delivery } of rows) {
const row = await loadAudience(tx, delivery.id);
if (!row || !current(row)) {
await tx.update(deliveries).set({ status: "superseded" }).where(eq(deliveries.id, delivery.id));
continue;
}
if (await deliverByEarlierReply(tx, row, run.id)) {
reported.push(delivery.id);
continue;
}
accepted.push(delivery.id); updates.push(await taskResult(tx, row.task));
await tx.update(deliveries).set({ status: "queued", targetRunId: run.id }).where(eq(deliveries.id, delivery.id));
}
if (accepted.length === 0 && reported.length > 0) {
const contextSnapshot = { ...run.contextSnapshot, chatCompletionDeliveryIds: [], chatCompletionAlreadyReported: reported };
await tx.update(heartbeatRuns).set({ contextSnapshot }).where(eq(heartbeatRuns.id, run.id));
return { ...run, contextSnapshot };
}
if (accepted.length === 0) throw new Error("chat_completion_superseded");
const contextSnapshot = { ...run.contextSnapshot, chatCompletionDeliveryIds: accepted, chatCompletionUpdates: updates };
await tx.update(heartbeatRuns).set({ contextSnapshot }).where(eq(heartbeatRuns.id, run.id));
return { ...run, contextSnapshot };
});
}
/** True when every completion this turn was woken for had already been reported. */
export function chatCompletionAlreadyReported(run: Run) {
const reported = run.contextSnapshot?.chatCompletionAlreadyReported;
return Array.isArray(reported) && reported.length > 0 && ids(run).length === 0;
}
export function chatCompletionInstruction(context: Record<string, unknown>) {
if (!Array.isArray(context.chatCompletionUpdates) || !context.chatCompletionUpdates.length) return "";
return `\n\nDelegated work has completed. Tell the user in this conversation what finished and provide access using the supplied task links. Report completion only for the tasks listed in this update; other tasks receive their own completion updates. Use the recorded status and result locations; do not repeat a promise to do work that is already Done. Do not start more work, change these tasks, or create duplicate review requests. Reuse and link the task’s existing canonical review/decision; completion updates are notifications, not fresh quality-review authority. recordedOutputDecisions are human verdicts on the exact versions shown. Link them instead of re-requesting review of those versions; do not treat approval as permission to publish. A separate explicit user request may authorize a new review. Keep opaque IDs and hashes out of your visible reply. The following JSON contains server-recorded lifecycle facts and result locations:\n${JSON.stringify(context.chatCompletionUpdates)}`;
}
/** Called under the comment transaction, before insertion. An event can publish only once. */
export async function existingChatCompletionReply(tx: Connection, runId: string, issueId: string) {
const [run] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
if (!run || ids(run).length === 0) return null;
const rows = await tx.select().from(deliveries).where(and(eq(deliveries.companyId, run.companyId), inArray(deliveries.id, ids(run))));
for (const delivery of rows) {
// Match the status writer's task-first lock order, then reload the outbox
// row so a concurrent reopen cannot leave us with its pre-commit snapshot.
await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.id, delivery.taskId), eq(issues.companyId, run.companyId))).for("update");
const row = await loadAudience(tx, delivery.id);
if (!row || row.source?.id !== issueId || row.handoff.agentId !== run.agentId || !current(row)) throw new Error("chat_completion_superseded");
if (row.delivery.responseCommentId) {
const [comment] = await tx.select().from(issueComments).where(and(eq(issueComments.id, row.delivery.responseCommentId), eq(issueComments.issueId, issueId)));
if (comment) return comment;
}
if (row.delivery.status !== "queued" || row.delivery.targetRunId !== runId) throw new Error("chat_completion_superseded");
}
if (rows.length !== ids(run).length) throw new Error("chat_completion_superseded");
return null;
}
export async function acknowledgeChatCompletionReply(tx: Connection, runId: string, commentId: string) {
await tx.update(deliveries).set({ status: "delivered", responseCommentId: commentId, error: null })
.where(and(eq(deliveries.targetRunId, runId), eq(deliveries.status, "queued")));
}
export function chatCompletionDeliveryService(db: Db, heartbeat: { wakeup(agentId: string, options: {
source: "automation"; triggerDetail: "system"; reason: string; idempotencyKey: string; allowRunCoalescing: boolean;
requestedByActorType: "system"; requestedByActorId: string; payload: Record<string, unknown>; contextSnapshot: Record<string, unknown>;
}): Promise<{ id: string } | null> }) {
async function deliver(id: string): Promise<void> {
// Lease outbox work before leaving the transaction; recovery reuses the same wake key.
const claimed = await db.update(deliveries).set({ nextAttemptAt: new Date(Date.now() + LEASE_MS) })
.where(and(eq(deliveries.id, id), inArray(deliveries.status, [...pending]), lte(deliveries.nextAttemptAt, new Date())))
.returning().then(rows => rows[0]);
if (!claimed) return;
try {
const row = await loadAudience(db, id);
if (!row || !current(row)) {
await db.update(deliveries).set({ status: "superseded" }).where(eq(deliveries.id, id)); return;
}
if (!(await instanceSettingsService(db).getExperimental()).enableAgentChat) return;
const [agent] = await db.select().from(agents).where(and(eq(agents.id, row.handoff.agentId), eq(agents.companyId, claimed.companyId)));
if (!agent || ["paused", "terminated"].includes(agent.status)) return;
const key = `chat-completion:${id}:${claimed.attempts}`;
const wakes = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, claimed.companyId), eq(agentWakeupRequests.idempotencyKey, key)));
const runId = claimed.targetRunId ?? wakes.find(w => w.runId)?.runId;
if (runId) {
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
if (run && activeRuns.includes(run.status)) return;
const presentationDeadline = run?.runtimeMode === "native" && run.status === "succeeded" && run.finishedAt
? run.finishedAt.getTime() + MAX_ATTEMPTS * LEASE_MS : null;
if (run && presentationDeadline !== null && Date.now() < presentationDeadline) {
const scope = { companyId: claimed.companyId, issueId: row.handoff.conversationId,
runId: run.id, agentId: row.handoff.agentId };
const deferred = await db.transaction(async tx => {
const candidates = nativeResponsePresentationCandidates(scope).as("pending_native_completion_reply");
const [response] = await tx.select().from(candidates);
if (!response || Date.now() >= presentationDeadline) return false;
await notifyNativeResponsePresentation(tx, scope);
const updated = await tx.update(deliveries).set({ nextAttemptAt: new Date(Math.min(presentationDeadline, Math.max(
claimed.nextAttemptAt.getTime(), response.nextAttemptAt?.getTime() ?? 0,
))) }).where(and(eq(deliveries.id, id), eq(deliveries.companyId, claimed.companyId),
eq(deliveries.status, "queued"), eq(deliveries.targetRunId, run.id),
eq(deliveries.statusVersion, claimed.statusVersion), eq(deliveries.attempts, claimed.attempts),
eq(deliveries.nextAttemptAt, claimed.nextAttemptAt))).returning({ id: deliveries.id });
return updated.length > 0;
});
if (deferred) return;
}
const advanced = await db.transaction(async tx => {
const [task] = await tx.select({ status: issues.status }).from(issues)
.where(and(eq(issues.id, claimed.taskId), eq(issues.companyId, claimed.companyId))).for("update");
if (task?.status !== "done") return false;
const sameClaim = and(eq(deliveries.id, id), eq(deliveries.companyId, claimed.companyId),
inArray(deliveries.status, [...pending]), eq(deliveries.statusVersion, claimed.statusVersion),
eq(deliveries.attempts, claimed.attempts),
eq(deliveries.nextAttemptAt, claimed.nextAttemptAt),
claimed.targetRunId ? eq(deliveries.targetRunId, claimed.targetRunId) : isNull(deliveries.targetRunId));
const [latest] = await tx.select({ id: deliveries.id }).from(deliveries).where(sameClaim).for("update");
if (!latest) return false;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatCompletion);
const updated = await tx.update(deliveries).set({ status: claimed.attempts + 1 >= MAX_ATTEMPTS ? "exhausted" : "pending",
attempts: claimed.attempts + 1, targetRunId: null, nextAttemptAt: new Date(),
error: run?.error ?? "Completion turn ended without a reply" }).where(sameClaim).returning({ id: deliveries.id });
return updated.length > 0;
});
if (advanced && claimed.attempts + 1 < MAX_ATTEMPTS) await deliver(id);
return;
}
if (wakes.some(w => ["queued", "claimed", "deferred_issue_execution", "coalesced"].includes(w.status))) return;
if (wakes.length) {
// Terminal receipts must advance the key; reusing them collides forever.
const dailyCap = wakes.some(w => w.reason?.startsWith("heartbeat.daily_"));
const next = new Date(Date.now() + LEASE_MS);
if (dailyCap) next.setUTCHours(24, 0, 0, 0);
await db.update(deliveries).set({ attempts: claimed.attempts + 1, targetRunId: null,
status: !dailyCap && claimed.attempts + 1 >= MAX_ATTEMPTS ? "exhausted" : "pending",
nextAttemptAt: next, error: wakes[0].reason ?? "Completion wake did not start" })
.where(and(eq(deliveries.id, id), inArray(deliveries.status, [...pending])));
return;
}
// One undispatched head per conversation. Its turn absorbs the remaining
// events at admission; events arriving after admission need a later turn.
const siblings = await db.select({ delivery: deliveries, run: heartbeatRuns }).from(deliveries)
.innerJoin(handoffs, eq(handoffs.taskId, deliveries.taskId))
.leftJoin(heartbeatRuns, eq(heartbeatRuns.id, deliveries.targetRunId))
.where(and(eq(deliveries.companyId, claimed.companyId), eq(handoffs.conversationId, row.handoff.conversationId),
eq(handoffs.sessionGeneration, row.handoff.sessionGeneration), inArray(deliveries.status, [...pending])))
.orderBy(asc(deliveries.createdAt), asc(deliveries.id));
if (siblings.some(s => s.run && activeRuns.includes(s.run.status) &&
!Array.isArray(s.run.contextSnapshot?.chatCompletionUpdates))) return;
if (siblings.find(s => !s.delivery.targetRunId)?.delivery.id !== id) return;
// An earlier reply already reported this completion: no wake and no run.
// The next undispatched sibling becomes the head on the following pass.
if (await deliverByEarlierReply(db, row, null)) {
await notifyDeliveryWork(db, DELIVERY_QUEUES.chatCompletion);
return;
}
const run = await heartbeat.wakeup(row.handoff.agentId, {
source: "automation", triggerDetail: "system", reason: CHAT_COMPLETION_WAKE_REASON, idempotencyKey: key, allowRunCoalescing: false,
requestedByActorType: "system", requestedByActorId: "chat_completion_delivery",
payload: { issueId: row.handoff.conversationId, chatCompletionDeliveryIds: [id] },
contextSnapshot: { issueId: row.handoff.conversationId, taskId: row.handoff.conversationId,
wakeReason: CHAT_COMPLETION_WAKE_REASON, chatCompletionDeliveryIds: [id], conversationSessionGeneration: row.handoff.sessionGeneration },
});
if (!run) {
const receipts = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests)
.where(and(eq(agentWakeupRequests.companyId, claimed.companyId), eq(agentWakeupRequests.idempotencyKey, key)));
if (!receipts.length) throw new Error("Completion wake returned no run or durable receipt");
}
if (run) await db.update(deliveries).set({ targetRunId: run.id, status: "queued" })
.where(and(eq(deliveries.id, id), inArray(deliveries.status, [...pending]), sql`${deliveries.targetRunId} is null`));
} catch (error) {
const receipts = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests)
.where(and(eq(agentWakeupRequests.companyId, claimed.companyId), eq(agentWakeupRequests.idempotencyKey, `chat-completion:${id}:${claimed.attempts}`)));
await db.update(deliveries).set({ error: error instanceof Error ? error.message : String(error),
...(receipts.length ? {} : { attempts: claimed.attempts + 1,
status: claimed.attempts + 1 >= MAX_ATTEMPTS ? "exhausted" as const : "pending" as const }) })
.where(and(eq(deliveries.id, id), inArray(deliveries.status, [...pending])));
}
}
async function sweepPending(scope?: { companyId: string; taskId: string }) {
const due = await db.select({ id: deliveries.id }).from(deliveries)
.where(and(inArray(deliveries.status, [...pending]), lte(deliveries.nextAttemptAt, new Date()),
...(scope ? [eq(deliveries.companyId, scope.companyId), eq(deliveries.taskId, scope.taskId)] : [])))
.orderBy(asc(deliveries.nextAttemptAt)).limit(100);
for (const row of due) await deliver(row.id);
}
async function hasPending() {
return (await db.select({ id: deliveries.id }).from(deliveries)
.where(inArray(deliveries.status, [...pending])).limit(1)).length > 0;
}
return { deliver, sweepPending, hasPending };
}