mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path
> - Paperclip manages work for AI agents.
> - Native helpers read task documents through authorized API tools.
> - A scan of 146 cases needs one saved response and one download per
case.
> - The document service can read the same key across authorized tasks
with bounded queries.
> - This pull request adds a bounded bulk read with the existing task
access policy.
> - Helpers can transfer one batch and check current revisions before
fetching changed bodies.
## Linked Issues or Issue Description
**Subsystem affected**
Task document REST API, shared query contracts, and native API
discovery.
**Problem or motivation**
Per-task document reads make native sync scans cost hundreds of tool
calls. Task list timestamps do not prove that document content is
unchanged.
**Proposed solution**
Add `GET
/api/companies/{companyId}/issue-documents?key=notes&issueIds=...`. Read
up to 200 task UUIDs under current company and task permissions. Return
current bodies and revision metadata, or metadata alone with
`includeBody=false`.
**Alternatives considered**
Generic request batching would add tool execution and partial failure
rules. One bulk resource read removes the per-case relay cost with the
existing tools. Do not skip documents based only on task-row timestamps.
Keep managed skill files immutable; cleanup of copied skill test
fixtures belongs to those tests.
**Roadmap alignment**
This improves existing task documents and native API access. It adds no
new workflow or UI. The roadmap has no duplicate bulk document read
work. No duplicate open pull request was found.
## What Changed
- Add a bulk keyed document route and a company-scoped document reader.
Reuse the current batched task-read policy and completed-task read
receipts.
- Preserve request order and deduplicate IDs. Return `document: null`
for readable tasks without the key. Omit unreadable, hidden, foreign and
missing tasks without revealing which exist.
- Bound requests to 200 IDs and responses to 8 MiB of JSON bytes. Check
encoded body sizes in SQL before fetching bodies in the same read
snapshot. Build JSON within the byte budget. Oversized responses fail
with 413 and are never saved as partial artifacts.
- Admit at most two active reads per actor and eight per server
instance. Extra requests receive 429 with Retry-After. Keep the slot
until both the database work and response settle.
- Batch completed-handoff lookups and group child receipts by parent.
Extend the existing event batch writer for ordinary non-waking receipts
with atomic sequence allocation. Stored receipt shapes stay unchanged
for existing consumers.
- Expose revision-only reads without selecting document bodies. Document
native discovery, one save plus one download, and complete-inventory
checks.
- Add real HTTP, database and artifact tests for 146 open and completed
cases, 200 maximum-size bodies, metadata overhead, access failures,
query limits and JSON expansion. Cover admission limits, failure
cleanup, and concurrent event writers. Refresh the shared SDK source
pins.
## Verification
Final revision: `93dc8b029600211d79a050650f3dff82e5dad52f`.
- [CI run 44212220917, attempt
3](https://github.com/paperclipai/paperclip/actions/runs/44212220917)
passed. This includes typecheck, build, general tests, Runner checks,
serialized server suites, all e2e shards, and the final verify gate.
-
[Build](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506305267),
[e2e](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506309117),
and
[verify](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134510062263)
are green.
- [Linux CI job
134506309291](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506309291)
passed all 16 bulk document integration tests. This includes the native
transfer of all 146 documents with the saved size and hash,
completed-case receipt batching, and size-only rejection of 200
maximum-size documents.
- [Greptile
review](https://github.com/paperclipai/paperclip/pull/22105#issuecomment-6965449032)
is 5/5 on the final revision. All review threads are resolved. Security
checks pass. GitHub reports CLEAN and MERGEABLE.
- Local server typecheck passed on macOS with Node 25.6.1. The focused
run below passed 177 tests; the Linux-only transfer test was skipped
locally and then passed in CI.
```sh
pnpm --filter @paperclipai/server typecheck
cd server
pnpm exec vitest run \
src/routes/issue-document-batch.test.ts \
src/services/native-runtime/issue-document-batch.integration.test.ts \
src/__tests__/documents-service.test.ts \
src/__tests__/openapi-routes.test.ts \
src/__tests__/heartbeat-run-event-atomic-append.test.ts \
src/__tests__/chat-completion-delivery.test.ts \
src/__tests__/native-child-task-read-routes.test.ts
```
- Shared typecheck and build passed. From `packages/shared/`, `pnpm exec
vitest run src/validators/issue.test.ts` passed 61 tests before the
rebase; the batch query contract did not change in the rebase.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
passed. Runtime and lockfile pins are unchanged.
- `pnpm verify --explain` returned exit 2 because the API docs and
shared contracts require broader CI. The successful CI run above
supplies that coverage.
Failure and recovery record:
- Both performance findings reproduced before the fixes. Scale tests now
verify a size-only query for 200 large documents and one handoff lookup
plus two receipt-write statements for 146 completed cases. Concurrent
writers retain contiguous unique sequences and decoded payloads.
- Initial local runs found missing shared exports and OpenAPI route
coverage. These are fixed. Fixtures now respect Linux-only installation,
thrown cross-company denials, and the ES2022 TypeScript target. Rebase
conflicts preserved the onboarding coverage from master.
- CI attempt 1 interrupted Build, e2e shard 7 and six general-test
shards with runner shutdown signals. Attempt 2 recovered the tests but
its [Build
job](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134499680425)
also received a shutdown signal and exited 143. No compiler or test
assertion failure was found in those interrupted job logs. The reason
the runners stopped remains unknown. Attempt 3 recovered Build and its
dependent verify gate on the same revision. Earlier interruptions are
not counted as passes.
## Risks
- Additive API only. No migration or existing endpoint response change.
- Helpers must adopt the batch response to gain the tool-call reduction.
They must verify that every requested task is present before treating
the scan as complete. Unreadable tasks are intentionally omitted.
- Split large document batches when the route returns 413. Metadata
remains available without bodies. Concurrency limits are per server
instance and can refuse simultaneous reads with 429; callers retry after
an active read finishes.
- This does not change company-owned helper inventory expiry or cleanup
tests, installed skill permissions, or generic tool batching. It does
not deploy to a tenant.
## Model Used
OpenAI `gpt-6.1-sol` through Codex in T3 Code, with xhigh reasoning,
tool use, and code execution. Context window size was not exposed.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
451 lines
31 KiB
TypeScript
451 lines
31 KiB
TypeScript
import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
||
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
|
||
import { nativeResponsePresentationCandidates } from "./native-runtime/native-response-presentation-query.js";
|
||
import { and, asc, desc, eq, gte, inArray, isNotNull, isNull, lte, ne, sql } from "drizzle-orm";
|
||
import { agents, agentWakeupRequests, chatCompletionDeliveries as deliveries, chatTaskHandoffs as handoffs,
|
||
heartbeatRunEvents, heartbeatRuns, issueComments, issueDocuments, issueThreadInteractions, issues, type Db } from "@paperclipai/db";
|
||
import { extractIssueReferenceMatches } from "@paperclipai/shared";
|
||
import { appendHeartbeatRunEventBatch } from "./heartbeat-run-events.js";
|
||
import { conflict, forbidden, notFound } from "../errors.js";
|
||
|
||
export const CHAT_COMPLETION_WAKE_REASON = "chat_task_completed";
|
||
/** Run event: the conversation agent read its handed-off task after it was done. */
|
||
export const COMPLETED_HANDOFF_READ_EVENT = "chat_handoff.completed_read";
|
||
const MAX_ATTEMPTS = 5;
|
||
const LEASE_MS = 60_000;
|
||
type Issue = typeof issues.$inferSelect;
|
||
type Run = typeof heartbeatRuns.$inferSelect;
|
||
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||
type Connection = Db | Tx;
|
||
const pending = ["pending", "queued"] as const;
|
||
const activeRuns = ["queued", "scheduled_retry", "running"];
|
||
function ids(run: Run): string[] {
|
||
const value = run.contextSnapshot?.chatCompletionDeliveryIds;
|
||
return Array.isArray(value) ? value.filter((v): v is string => typeof v === "string") : [];
|
||
}
|
||
|
||
/** Human-confirmed suggestions use the same durable audience as native create_task.
|
||
* The source must be read/authorized by the caller; ownership and generation are
|
||
* rechecked under the creation transaction so reset chats cannot receive new work. */
|
||
export async function lockAcceptedChatHandoffSource(tx: Connection, input: {
|
||
companyId: string; conversationId: string; sessionGeneration: number;
|
||
userId?: string | null; agentId?: string | null;
|
||
}) {
|
||
const [source] = await tx.select().from(issues).where(and(eq(issues.id, input.conversationId),
|
||
eq(issues.companyId, input.companyId))).for("update");
|
||
if (!source?.conversationAgentId) throw notFound("Originating conversation not found");
|
||
if (source.conversationSessionGeneration !== input.sessionGeneration) {
|
||
throw conflict("The originating conversation was reset; request a new task suggestion");
|
||
}
|
||
if (input.userId ? source.conversationUserId !== input.userId : source.conversationAgentId !== input.agentId) {
|
||
throw forbidden("Task handoff requires the originating conversation's participant");
|
||
}
|
||
return source;
|
||
}
|
||
|
||
export async function recordAcceptedChatHandoff(tx: Connection, task: Issue, source: Issue) {
|
||
if (!source.conversationAgentId || source.companyId !== task.companyId || task.conversationAgentId) {
|
||
throw forbidden("Task handoff requires an execution task in the originating conversation's company");
|
||
}
|
||
await tx.insert(handoffs).values({ taskId: task.id, companyId: task.companyId,
|
||
conversationId: source.id, agentId: source.conversationAgentId,
|
||
sessionGeneration: source.conversationSessionGeneration }).onConflictDoNothing();
|
||
}
|
||
|
||
export async function recordChatHandoff(tx: Connection, task: Issue, actorRunId: string | null | undefined) {
|
||
if (!actorRunId || task.conversationAgentId || !task.createdByAgentId) return;
|
||
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, actorRunId),
|
||
eq(heartbeatRuns.companyId, task.companyId), eq(heartbeatRuns.agentId, task.createdByAgentId)));
|
||
const sourceId = run?.nativeIssueId ?? run?.contextSnapshot?.issueId;
|
||
if (typeof sourceId !== "string" || sourceId === task.id) return;
|
||
const [source] = await tx.select().from(issues).where(and(eq(issues.id, sourceId), eq(issues.companyId, task.companyId)));
|
||
if (!source?.conversationAgentId || source.conversationAgentId !== run.agentId ||
|
||
source.conversationSessionGeneration !== run.contextSnapshot?.conversationSessionGeneration) return;
|
||
await recordAcceptedChatHandoff(tx, task, source);
|
||
}
|
||
|
||
/** Must run on the same transaction as status projection (including native arbitration). */
|
||
export async function recordChatCompletion(tx: Connection, before: Issue, after: Issue): Promise<boolean> {
|
||
if (before.status === after.status) return false;
|
||
await tx.update(deliveries).set({ status: "superseded" }).where(and(eq(deliveries.taskId, after.id),
|
||
eq(deliveries.companyId, after.companyId), inArray(deliveries.status, [...pending])));
|
||
if (after.status !== "done") return false;
|
||
const [handoff] = await tx.select().from(handoffs).where(and(eq(handoffs.taskId, after.id), eq(handoffs.companyId, after.companyId)));
|
||
if (!handoff) return false;
|
||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatCompletion);
|
||
await tx.insert(deliveries).values({ companyId: after.companyId, taskId: after.id, statusVersion: after.statusVersion }).onConflictDoNothing();
|
||
return true;
|
||
}
|
||
|
||
async function loadAudience(tx: Connection, deliveryId: string) {
|
||
const [row] = await tx.select({ delivery: deliveries, handoff: handoffs, task: issues }).from(deliveries)
|
||
.innerJoin(handoffs, and(eq(handoffs.taskId, deliveries.taskId), eq(handoffs.companyId, deliveries.companyId)))
|
||
.innerJoin(issues, and(eq(issues.id, deliveries.taskId), eq(issues.companyId, deliveries.companyId)))
|
||
.where(eq(deliveries.id, deliveryId));
|
||
if (!row) return null;
|
||
const [source] = await tx.select().from(issues).where(and(eq(issues.id, row.handoff.conversationId), eq(issues.companyId, row.handoff.companyId)));
|
||
return { ...row, source };
|
||
}
|
||
function current(row: NonNullable<Awaited<ReturnType<typeof loadAudience>>>) {
|
||
return row.task.status === "done" && !["superseded", "exhausted"].includes(row.delivery.status) &&
|
||
row.source?.conversationAgentId === row.handoff.agentId &&
|
||
row.source.conversationSessionGeneration === row.handoff.sessionGeneration;
|
||
}
|
||
|
||
async function taskResult(tx: Connection, task: Issue) {
|
||
// Never copy worker-authored titles, comments or document bodies into the
|
||
// source agent's instructions. These links and lifecycle facts are generated
|
||
// by the server; the saved work remains on its normal access-controlled task.
|
||
const documents = await tx.select({ id: issueDocuments.documentId }).from(issueDocuments)
|
||
.where(and(eq(issueDocuments.companyId, task.companyId), eq(issueDocuments.issueId, task.id))).limit(1);
|
||
// A bounded receipt projection gives the coordinator actual verdicts to link,
|
||
// rather than asking it to infer whether the worker's prose needs another review.
|
||
// These are historical decisions on their named versions, not publication authority.
|
||
const reviews = await tx.select().from(issueThreadInteractions).where(and(
|
||
eq(issueThreadInteractions.companyId, task.companyId), eq(issueThreadInteractions.issueId, task.id),
|
||
inArray(issueThreadInteractions.kind, ["request_confirmation", "request_item_verdicts"]),
|
||
inArray(issueThreadInteractions.status, ["accepted", "answered"]),
|
||
)).orderBy(desc(issueThreadInteractions.resolvedAt), desc(issueThreadInteractions.id)).limit(20);
|
||
const recordedOutputDecisions = reviews.flatMap(review => {
|
||
const payload = review.payload as Record<string, any>, result = (review.result ?? {}) as Record<string, any>;
|
||
const candidates = review.kind === "request_confirmation" ? [{ target: payload.target, outcome: "approve", human: Boolean(review.resolvedByUserId) && !review.resolvedByAgentId }]
|
||
: (Array.isArray(result.items) ? result.items : []).map((item: any) => ({
|
||
target: (Array.isArray(payload.items) ? payload.items : []).find((entry: any) => entry.id === item.id)?.target ?? payload.target,
|
||
outcome: item.verdict, human: Boolean(item.resolvedByUserId) && !item.resolvedByAgentId,
|
||
}));
|
||
return candidates.filter(candidate => candidate.human && ["approve", "reject", "defer"].includes(candidate.outcome) &&
|
||
candidate.target && ["issue_document", "issue_attachment"].includes(candidate.target.type) &&
|
||
(!candidate.target.issueId || candidate.target.issueId === task.id) && candidate.target.key !== "plan")
|
||
.map(candidate => ({ interactionId: review.id, outcome: candidate.outcome,
|
||
target: candidate.target.type === "issue_document"
|
||
? { type: "issue_document", key: candidate.target.key, revisionId: candidate.target.revisionId }
|
||
: { type: "issue_attachment", attachmentId: candidate.target.attachmentId, assetId: candidate.target.assetId, sha256: candidate.target.sha256 },
|
||
url: `/issues/${task.identifier ?? task.id}#interaction-${review.id}` }));
|
||
}).slice(0, 30);
|
||
return { id: task.id, identifier: task.identifier, status: task.status,
|
||
completedAt: task.completedAt, url: `/issues/${task.identifier ?? task.id}`,
|
||
hasSavedDocuments: documents.length > 0, recordedOutputDecisions };
|
||
}
|
||
|
||
/** A completed onboarding parent still owes the result of its own child handoff.
|
||
* This permits a reporting turn without reopening Done or reviving cancellation. */
|
||
export async function isCompletedOnboardingHandoffWake(db: Connection, input: {
|
||
companyId: string; issueId: string; agentId: string; reason: string | null;
|
||
contextSnapshot: Record<string, unknown>;
|
||
}) {
|
||
if (input.reason !== "issue_children_completed" || typeof input.contextSnapshot.completedChildIssueId !== "string") return false;
|
||
const [source] = await db.select().from(issues).where(and(eq(issues.id, input.issueId), eq(issues.companyId, input.companyId)));
|
||
if (source?.originKind !== "onboarding_first_task" || source.status !== "done" || source.assigneeAgentId !== input.agentId) return false;
|
||
const children = await db.select({ id: issues.id, status: issues.status }).from(issues)
|
||
.where(and(eq(issues.companyId, input.companyId), eq(issues.parentId, source.id)));
|
||
return children.some(child => child.id === input.contextSnapshot.completedChildIssueId && child.status === "done") &&
|
||
children.every(child => ["done", "cancelled"].includes(child.status));
|
||
}
|
||
|
||
/** Records that the conversation agent read a handed-off task in its done state. The
|
||
* completion turn uses this to recognize a completion that an earlier reply reported. */
|
||
export async function recordCompletedHandoffRead(db: Db, input: {
|
||
companyId: string; agentId: string; runId: string;
|
||
task: { id: string; status: string; statusVersion?: number | null };
|
||
}) {
|
||
await recordCompletedHandoffReads(db, { ...input, tasks: [input.task] });
|
||
}
|
||
|
||
/** One lookup and one event append statement for a batch; payloads stay compatible with single reads. */
|
||
export async function recordCompletedHandoffReads(db: Db, input: {
|
||
companyId: string; agentId: string; runId: string;
|
||
tasks: ReadonlyArray<{ id: string; status: string; statusVersion?: number | null }>;
|
||
}) {
|
||
const completed = input.tasks.filter(task => task.status === "done" && typeof task.statusVersion === "number");
|
||
if (!completed.length) return;
|
||
const rows = await db.select({ taskId: handoffs.taskId }).from(handoffs)
|
||
.where(and(inArray(handoffs.taskId, completed.map(task => task.id)), eq(handoffs.companyId, input.companyId), eq(handoffs.agentId, input.agentId)));
|
||
const matching = new Set(rows.map(row => row.taskId));
|
||
await appendHeartbeatRunEventBatch(db, completed.filter(task => matching.has(task.id)).map(task => ({
|
||
companyId: input.companyId, runId: input.runId, agentId: input.agentId,
|
||
eventType: COMPLETED_HANDOFF_READ_EVENT, stream: "system", level: "info",
|
||
payload: { taskId: task.id, statusVersion: task.statusVersion },
|
||
})));
|
||
}
|
||
|
||
/** A reply in this conversation already reported this completion when a turn of the same
|
||
* agent read the task done for this completion and then linked the task in its reply. */
|
||
async function earlierCompletionReply(tx: Connection, row: NonNullable<Awaited<ReturnType<typeof loadAudience>>>, runId: string | null) {
|
||
const { task, handoff, delivery } = row;
|
||
if (!task.completedAt || !task.identifier) return null;
|
||
// A reopen supersedes this delivery and a new completion gets a higher status
|
||
// version, so any done read at or after this version belongs to this completion,
|
||
// including after an owner change that advanced the version while still done.
|
||
const readThisCompletion = sql`exists (select 1 from ${heartbeatRunEvents} handoff_read
|
||
where handoff_read.company_id = ${handoff.companyId} and handoff_read.run_id = ${issueComments.createdByRunId}
|
||
and handoff_read.event_type = ${COMPLETED_HANDOFF_READ_EVENT} and handoff_read.payload->>'taskId' = ${task.id}
|
||
and (handoff_read.payload->>'statusVersion')::bigint >= ${delivery.statusVersion}
|
||
and handoff_read.created_at <= ${issueComments.createdAt})`;
|
||
const replies = await tx.select({ id: issueComments.id, body: issueComments.body }).from(issueComments).where(and(
|
||
eq(issueComments.companyId, handoff.companyId), eq(issueComments.issueId, handoff.conversationId),
|
||
eq(issueComments.authorAgentId, handoff.agentId), isNull(issueComments.deletedAt),
|
||
isNotNull(issueComments.createdByRunId), runId ? ne(issueComments.createdByRunId, runId) : undefined,
|
||
gte(issueComments.createdAt, task.completedAt), readThisCompletion,
|
||
)).orderBy(asc(issueComments.createdAt), asc(issueComments.id)).limit(20);
|
||
const identifier = task.identifier.toUpperCase();
|
||
return replies.find(reply => extractIssueReferenceMatches(reply.body)
|
||
.some(match => match.identifier.toUpperCase() === identifier))?.id ?? null;
|
||
}
|
||
|
||
/** Marks one delivery delivered by the earlier reply that already reported it. */
|
||
async function deliverByEarlierReply(tx: Connection, row: NonNullable<Awaited<ReturnType<typeof loadAudience>>>, runId: string | null) {
|
||
const reportedBy = await earlierCompletionReply(tx, row, runId);
|
||
if (!reportedBy) return false;
|
||
// An earlier turn already told the person; repeating it would post the same news twice.
|
||
await tx.update(deliveries).set({ status: "delivered", responseCommentId: reportedBy, error: null })
|
||
.where(and(eq(deliveries.id, row.delivery.id), inArray(deliveries.status, [...pending])));
|
||
return true;
|
||
}
|
||
|
||
/** Before a completion wake becomes a run: settles every undispatched completion of this
|
||
* conversation that an earlier reply already reported. True only when at least one was
|
||
* reported and none is left for a completion turn, so the wake needs no run at all. */
|
||
export async function settleReportedChatCompletions(tx: Connection, input: { companyId: string; conversationId: string; agentId: string }) {
|
||
const [source] = await tx.select().from(issues).where(and(eq(issues.id, input.conversationId), eq(issues.companyId, input.companyId))).for("update");
|
||
if (!source || source.conversationAgentId !== input.agentId) return false;
|
||
const rows = await tx.select({ id: deliveries.id }).from(deliveries).innerJoin(handoffs, eq(handoffs.taskId, deliveries.taskId))
|
||
.where(and(eq(deliveries.companyId, input.companyId), eq(handoffs.conversationId, source.id), eq(handoffs.agentId, input.agentId),
|
||
eq(handoffs.sessionGeneration, source.conversationSessionGeneration), inArray(deliveries.status, [...pending]), isNull(deliveries.targetRunId)))
|
||
.orderBy(asc(deliveries.createdAt)).limit(21);
|
||
if (rows.length === 0 || rows.length > 20) return false;
|
||
let reported = 0, remaining = 0;
|
||
for (const { id } of rows) {
|
||
const row = await loadAudience(tx, id);
|
||
if (!row || !current(row)) {
|
||
await tx.update(deliveries).set({ status: "superseded" }).where(eq(deliveries.id, id));
|
||
continue;
|
||
}
|
||
if (await deliverByEarlierReply(tx, row, null)) reported++; else remaining++;
|
||
}
|
||
return reported > 0 && remaining === 0;
|
||
}
|
||
|
||
/** Freeze the input at turn start. New completions cannot be consumed by an already-running turn. */
|
||
export async function prepareChatCompletionTurn(db: Db, run: Run): Promise<Run> {
|
||
const issueId = run.contextSnapshot?.issueId;
|
||
if (typeof issueId !== "string") return run;
|
||
return db.transaction(async tx => {
|
||
const [source] = await tx.select().from(issues).where(and(eq(issues.id, issueId), eq(issues.companyId, run.companyId))).for("update");
|
||
if (!source) return run;
|
||
if (source.originKind === "onboarding_first_task" && run.contextSnapshot?.wakeReason === "issue_children_completed") {
|
||
const children = await tx.select().from(issues).where(and(eq(issues.companyId, run.companyId), eq(issues.parentId, source.id), eq(issues.status, "done"))).orderBy(issues.createdAt).limit(20);
|
||
const contextSnapshot = { ...run.contextSnapshot, onboardingCompletion: true,
|
||
chatCompletionUpdates: await Promise.all(children.map(child => taskResult(tx, child))) };
|
||
await tx.update(heartbeatRuns).set({ contextSnapshot }).where(eq(heartbeatRuns.id, run.id));
|
||
return { ...run, contextSnapshot };
|
||
}
|
||
const initialIds = ids(run);
|
||
// Only completion wakes absorb pending events; ordinary user turns retain their existing input.
|
||
if (initialIds.length === 0 && run.contextSnapshot?.wakeReason !== CHAT_COMPLETION_WAKE_REASON) return run;
|
||
if (run.contextSnapshot?.conversationSessionGeneration !== source.conversationSessionGeneration) throw new Error("chat_completion_superseded");
|
||
const rows = await tx.select({ delivery: deliveries }).from(deliveries).innerJoin(handoffs, eq(handoffs.taskId, deliveries.taskId))
|
||
.where(and(eq(deliveries.companyId, run.companyId), eq(handoffs.conversationId, issueId),
|
||
eq(handoffs.agentId, run.agentId), eq(handoffs.sessionGeneration, source.conversationSessionGeneration),
|
||
inArray(deliveries.status, [...pending]),
|
||
sql`(${deliveries.targetRunId} is null or ${deliveries.targetRunId} = ${run.id})`))
|
||
.orderBy(asc(deliveries.createdAt)).limit(20);
|
||
const updates: Awaited<ReturnType<typeof taskResult>>[] = [];
|
||
const accepted: string[] = [];
|
||
const reported: string[] = [];
|
||
for (const { delivery } of rows) {
|
||
const row = await loadAudience(tx, delivery.id);
|
||
if (!row || !current(row)) {
|
||
await tx.update(deliveries).set({ status: "superseded" }).where(eq(deliveries.id, delivery.id));
|
||
continue;
|
||
}
|
||
if (await deliverByEarlierReply(tx, row, run.id)) {
|
||
reported.push(delivery.id);
|
||
continue;
|
||
}
|
||
accepted.push(delivery.id); updates.push(await taskResult(tx, row.task));
|
||
await tx.update(deliveries).set({ status: "queued", targetRunId: run.id }).where(eq(deliveries.id, delivery.id));
|
||
}
|
||
if (accepted.length === 0 && reported.length > 0) {
|
||
const contextSnapshot = { ...run.contextSnapshot, chatCompletionDeliveryIds: [], chatCompletionAlreadyReported: reported };
|
||
await tx.update(heartbeatRuns).set({ contextSnapshot }).where(eq(heartbeatRuns.id, run.id));
|
||
return { ...run, contextSnapshot };
|
||
}
|
||
if (accepted.length === 0) throw new Error("chat_completion_superseded");
|
||
const contextSnapshot = { ...run.contextSnapshot, chatCompletionDeliveryIds: accepted, chatCompletionUpdates: updates };
|
||
await tx.update(heartbeatRuns).set({ contextSnapshot }).where(eq(heartbeatRuns.id, run.id));
|
||
return { ...run, contextSnapshot };
|
||
});
|
||
}
|
||
|
||
/** True when every completion this turn was woken for had already been reported. */
|
||
export function chatCompletionAlreadyReported(run: Run) {
|
||
const reported = run.contextSnapshot?.chatCompletionAlreadyReported;
|
||
return Array.isArray(reported) && reported.length > 0 && ids(run).length === 0;
|
||
}
|
||
|
||
export function chatCompletionInstruction(context: Record<string, unknown>) {
|
||
if (!Array.isArray(context.chatCompletionUpdates) || !context.chatCompletionUpdates.length) return "";
|
||
return `\n\nDelegated work has completed. Tell the user in this conversation what finished and provide access using the supplied task links. Report completion only for the tasks listed in this update; other tasks receive their own completion updates. Use the recorded status and result locations; do not repeat a promise to do work that is already Done. Do not start more work, change these tasks, or create duplicate review requests. Reuse and link the task’s existing canonical review/decision; completion updates are notifications, not fresh quality-review authority. recordedOutputDecisions are human verdicts on the exact versions shown. Link them instead of re-requesting review of those versions; do not treat approval as permission to publish. A separate explicit user request may authorize a new review. Keep opaque IDs and hashes out of your visible reply. The following JSON contains server-recorded lifecycle facts and result locations:\n${JSON.stringify(context.chatCompletionUpdates)}`;
|
||
}
|
||
|
||
/** Called under the comment transaction, before insertion. An event can publish only once. */
|
||
export async function existingChatCompletionReply(tx: Connection, runId: string, issueId: string) {
|
||
const [run] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||
if (!run || ids(run).length === 0) return null;
|
||
const rows = await tx.select().from(deliveries).where(and(eq(deliveries.companyId, run.companyId), inArray(deliveries.id, ids(run))));
|
||
for (const delivery of rows) {
|
||
// Match the status writer's task-first lock order, then reload the outbox
|
||
// row so a concurrent reopen cannot leave us with its pre-commit snapshot.
|
||
await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.id, delivery.taskId), eq(issues.companyId, run.companyId))).for("update");
|
||
const row = await loadAudience(tx, delivery.id);
|
||
if (!row || row.source?.id !== issueId || row.handoff.agentId !== run.agentId || !current(row)) throw new Error("chat_completion_superseded");
|
||
if (row.delivery.responseCommentId) {
|
||
const [comment] = await tx.select().from(issueComments).where(and(eq(issueComments.id, row.delivery.responseCommentId), eq(issueComments.issueId, issueId)));
|
||
if (comment) return comment;
|
||
}
|
||
if (row.delivery.status !== "queued" || row.delivery.targetRunId !== runId) throw new Error("chat_completion_superseded");
|
||
}
|
||
if (rows.length !== ids(run).length) throw new Error("chat_completion_superseded");
|
||
return null;
|
||
}
|
||
export async function acknowledgeChatCompletionReply(tx: Connection, runId: string, commentId: string) {
|
||
await tx.update(deliveries).set({ status: "delivered", responseCommentId: commentId, error: null })
|
||
.where(and(eq(deliveries.targetRunId, runId), eq(deliveries.status, "queued")));
|
||
}
|
||
|
||
export function chatCompletionDeliveryService(db: Db, heartbeat: { wakeup(agentId: string, options: {
|
||
source: "automation"; triggerDetail: "system"; reason: string; idempotencyKey: string; allowRunCoalescing: boolean;
|
||
requestedByActorType: "system"; requestedByActorId: string; payload: Record<string, unknown>; contextSnapshot: Record<string, unknown>;
|
||
}): Promise<{ id: string } | null> }) {
|
||
async function deliver(id: string): Promise<void> {
|
||
// Lease outbox work before leaving the transaction; recovery reuses the same wake key.
|
||
const claimed = await db.update(deliveries).set({ nextAttemptAt: new Date(Date.now() + LEASE_MS) })
|
||
.where(and(eq(deliveries.id, id), inArray(deliveries.status, [...pending]), lte(deliveries.nextAttemptAt, new Date())))
|
||
.returning().then(rows => rows[0]);
|
||
if (!claimed) return;
|
||
try {
|
||
const row = await loadAudience(db, id);
|
||
if (!row || !current(row)) {
|
||
await db.update(deliveries).set({ status: "superseded" }).where(eq(deliveries.id, id)); return;
|
||
}
|
||
const [agent] = await db.select().from(agents).where(and(eq(agents.id, row.handoff.agentId), eq(agents.companyId, claimed.companyId)));
|
||
if (!agent || ["paused", "terminated"].includes(agent.status)) return;
|
||
const key = `chat-completion:${id}:${claimed.attempts}`;
|
||
const wakes = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, claimed.companyId), eq(agentWakeupRequests.idempotencyKey, key)));
|
||
const runId = claimed.targetRunId ?? wakes.find(w => w.runId)?.runId;
|
||
if (runId) {
|
||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||
if (run && activeRuns.includes(run.status)) return;
|
||
|
||
const presentationDeadline = run?.runtimeMode === "native" && run.status === "succeeded" && run.finishedAt
|
||
? run.finishedAt.getTime() + MAX_ATTEMPTS * LEASE_MS : null;
|
||
if (run && presentationDeadline !== null && Date.now() < presentationDeadline) {
|
||
const scope = { companyId: claimed.companyId, issueId: row.handoff.conversationId,
|
||
runId: run.id, agentId: row.handoff.agentId };
|
||
const deferred = await db.transaction(async tx => {
|
||
|
||
|
||
const candidates = nativeResponsePresentationCandidates(scope).as("pending_native_completion_reply");
|
||
const [response] = await tx.select().from(candidates);
|
||
if (!response || Date.now() >= presentationDeadline) return false;
|
||
await notifyNativeResponsePresentation(tx, scope);
|
||
const updated = await tx.update(deliveries).set({ nextAttemptAt: new Date(Math.min(presentationDeadline, Math.max(
|
||
claimed.nextAttemptAt.getTime(), response.nextAttemptAt?.getTime() ?? 0,
|
||
))) }).where(and(eq(deliveries.id, id), eq(deliveries.companyId, claimed.companyId),
|
||
eq(deliveries.status, "queued"), eq(deliveries.targetRunId, run.id),
|
||
eq(deliveries.statusVersion, claimed.statusVersion), eq(deliveries.attempts, claimed.attempts),
|
||
eq(deliveries.nextAttemptAt, claimed.nextAttemptAt))).returning({ id: deliveries.id });
|
||
return updated.length > 0;
|
||
});
|
||
if (deferred) return;
|
||
}
|
||
const advanced = await db.transaction(async tx => {
|
||
|
||
|
||
const [task] = await tx.select({ status: issues.status }).from(issues)
|
||
.where(and(eq(issues.id, claimed.taskId), eq(issues.companyId, claimed.companyId))).for("update");
|
||
if (task?.status !== "done") return false;
|
||
const sameClaim = and(eq(deliveries.id, id), eq(deliveries.companyId, claimed.companyId),
|
||
inArray(deliveries.status, [...pending]), eq(deliveries.statusVersion, claimed.statusVersion),
|
||
eq(deliveries.attempts, claimed.attempts),
|
||
eq(deliveries.nextAttemptAt, claimed.nextAttemptAt),
|
||
claimed.targetRunId ? eq(deliveries.targetRunId, claimed.targetRunId) : isNull(deliveries.targetRunId));
|
||
const [latest] = await tx.select({ id: deliveries.id }).from(deliveries).where(sameClaim).for("update");
|
||
if (!latest) return false;
|
||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatCompletion);
|
||
const updated = await tx.update(deliveries).set({ status: claimed.attempts + 1 >= MAX_ATTEMPTS ? "exhausted" : "pending",
|
||
attempts: claimed.attempts + 1, targetRunId: null, nextAttemptAt: new Date(),
|
||
error: run?.error ?? "Completion turn ended without a reply" }).where(sameClaim).returning({ id: deliveries.id });
|
||
return updated.length > 0;
|
||
});
|
||
|
||
|
||
if (advanced && claimed.attempts + 1 < MAX_ATTEMPTS) await deliver(id);
|
||
return;
|
||
}
|
||
if (wakes.some(w => ["queued", "claimed", "deferred_issue_execution", "coalesced"].includes(w.status))) return;
|
||
if (wakes.length) {
|
||
// Terminal receipts must advance the key; reusing them collides forever.
|
||
const dailyCap = wakes.some(w => w.reason?.startsWith("heartbeat.daily_"));
|
||
const next = new Date(Date.now() + LEASE_MS);
|
||
if (dailyCap) next.setUTCHours(24, 0, 0, 0);
|
||
await db.update(deliveries).set({ attempts: claimed.attempts + 1, targetRunId: null,
|
||
status: !dailyCap && claimed.attempts + 1 >= MAX_ATTEMPTS ? "exhausted" : "pending",
|
||
nextAttemptAt: next, error: wakes[0].reason ?? "Completion wake did not start" })
|
||
.where(and(eq(deliveries.id, id), inArray(deliveries.status, [...pending])));
|
||
return;
|
||
}
|
||
// One undispatched head per conversation. Its turn absorbs the remaining
|
||
// events at admission; events arriving after admission need a later turn.
|
||
const siblings = await db.select({ delivery: deliveries, run: heartbeatRuns }).from(deliveries)
|
||
.innerJoin(handoffs, eq(handoffs.taskId, deliveries.taskId))
|
||
.leftJoin(heartbeatRuns, eq(heartbeatRuns.id, deliveries.targetRunId))
|
||
.where(and(eq(deliveries.companyId, claimed.companyId), eq(handoffs.conversationId, row.handoff.conversationId),
|
||
eq(handoffs.sessionGeneration, row.handoff.sessionGeneration), inArray(deliveries.status, [...pending])))
|
||
.orderBy(asc(deliveries.createdAt), asc(deliveries.id));
|
||
if (siblings.some(s => s.run && activeRuns.includes(s.run.status) &&
|
||
!Array.isArray(s.run.contextSnapshot?.chatCompletionUpdates))) return;
|
||
if (siblings.find(s => !s.delivery.targetRunId)?.delivery.id !== id) return;
|
||
// An earlier reply already reported this completion: no wake and no run.
|
||
// The next undispatched sibling becomes the head on the following pass.
|
||
if (await deliverByEarlierReply(db, row, null)) {
|
||
await notifyDeliveryWork(db, DELIVERY_QUEUES.chatCompletion);
|
||
return;
|
||
}
|
||
const run = await heartbeat.wakeup(row.handoff.agentId, {
|
||
source: "automation", triggerDetail: "system", reason: CHAT_COMPLETION_WAKE_REASON, idempotencyKey: key, allowRunCoalescing: false,
|
||
requestedByActorType: "system", requestedByActorId: "chat_completion_delivery",
|
||
payload: { issueId: row.handoff.conversationId, chatCompletionDeliveryIds: [id] },
|
||
contextSnapshot: { issueId: row.handoff.conversationId, taskId: row.handoff.conversationId,
|
||
wakeReason: CHAT_COMPLETION_WAKE_REASON, chatCompletionDeliveryIds: [id], conversationSessionGeneration: row.handoff.sessionGeneration },
|
||
});
|
||
if (!run) {
|
||
const receipts = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests)
|
||
.where(and(eq(agentWakeupRequests.companyId, claimed.companyId), eq(agentWakeupRequests.idempotencyKey, key)));
|
||
if (!receipts.length) throw new Error("Completion wake returned no run or durable receipt");
|
||
}
|
||
if (run) await db.update(deliveries).set({ targetRunId: run.id, status: "queued" })
|
||
.where(and(eq(deliveries.id, id), inArray(deliveries.status, [...pending]), sql`${deliveries.targetRunId} is null`));
|
||
} catch (error) {
|
||
const receipts = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests)
|
||
.where(and(eq(agentWakeupRequests.companyId, claimed.companyId), eq(agentWakeupRequests.idempotencyKey, `chat-completion:${id}:${claimed.attempts}`)));
|
||
await db.update(deliveries).set({ error: error instanceof Error ? error.message : String(error),
|
||
...(receipts.length ? {} : { attempts: claimed.attempts + 1,
|
||
status: claimed.attempts + 1 >= MAX_ATTEMPTS ? "exhausted" as const : "pending" as const }) })
|
||
.where(and(eq(deliveries.id, id), inArray(deliveries.status, [...pending])));
|
||
}
|
||
}
|
||
async function sweepPending(scope?: { companyId: string; taskId: string }) {
|
||
const due = await db.select({ id: deliveries.id }).from(deliveries)
|
||
.where(and(inArray(deliveries.status, [...pending]), lte(deliveries.nextAttemptAt, new Date()),
|
||
...(scope ? [eq(deliveries.companyId, scope.companyId), eq(deliveries.taskId, scope.taskId)] : [])))
|
||
.orderBy(asc(deliveries.nextAttemptAt)).limit(100);
|
||
for (const row of due) await deliver(row.id);
|
||
}
|
||
async function hasPending() {
|
||
return (await db.select({ id: deliveries.id }).from(deliveries)
|
||
.where(inArray(deliveries.status, [...pending])).limit(1)).length > 0;
|
||
}
|
||
return { deliver, sweepPending, hasPending };
|
||
}
|