feat(server): add scoped chat milestone recovery worker

Reconstruct issue-scoped projection work with the shared coordinator. Signal existing run-status, presentation, and question-receipt transactions; verify real cancellation and continuation producers. Keep production polling until remaining producer coverage is complete.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-25 06:55:23 -05:00
1 parent 362bf75b8f
commit 56ead750d3
9 files changed
+482 -40

No files matched your search

@@ -2,8 +2,8 @@
Baseline: Teams attempt recovery #17592 at f53ae3d6d2.
Branch: codex/sleep-28-chat-run-milestones.
Status: projection owners separated and lifecycle cursor regression repaired;
transaction producer coverage, worker registration and final acceptance pending.
Status: scoped worker and initial real producer signals implemented and locally
verified; complete producer coverage, production registration and final acceptance pending.
## Separate obligations discovered
@@ -145,3 +145,99 @@ Paperclip Review result. Neutral is not a completed positive review. #17590 at
queued at inspection. #17586 also has a neutral Paperclip Review. Final review
acceptance remains separate. Refreshed master is `2e47c42fad`; stack reconciliation
remains required. No stack PR was merged.
## Scoped worker and initial producer checkpoint
`chat-run-milestone-work.ts` now reuses `registerScopedDeadlineWork`. Its loader
shares the lifecycle candidate selector with the explicit projector, retaining
all existing SQL eligibility fences and timestamp precision. It reconstructs
one entry per company/issue, with a bounded 100-publication projection batch.
A full batch remains immediately due; a short batch has exhausted the selector
and disarms, including currently suppressed lineage. Real selection/write
failures retain the shared retry backoff. The scheduler owns coalescing,
company fairness, first writes during drain, concurrent changes, and shutdown.
No new persistent queue or independent timer is introduced.
`notifyChatRunMilestones` carries only company/issue identity through the existing
outer-transaction protocol. Issue-scoped invalidation includes descendants;
company scope is available for broader configuration or lifecycle changes.
Unrelated issues/companies are not reconstructed for an issue hint.
Initial production integrations now await intent before writing:
- `heartbeat/run-lifecycle.ts::writeRunStatus`: both shared status transition
callers signal the known company in their existing transaction. This is a
conservative company hint; it is not proof that all raw run writes are covered.
- `heartbeat/run-completion.ts`: the late presentation-decision write now owns
a short transaction and signals its issue (company fallback when unbound).
Provider execution and presentation generation remain outside that transaction.
- `question-response-delivery.ts::recordTerminal`: the existing transaction
signals the issue before recording the durable target run and outcome.
The production app still uses the combined periodic projector. The new worker
is exercised through actual coordinator registration in tests but is not yet
registered by `app.ts`; do not remove the sweep until the remaining producer
families in the table above are implemented. In particular: queue admission and
recovery writers, native finalization and semantic receipts, question claim/retry
transitions, interaction changes, publication settlement, conversation/endpoint
changes, and decision/wake ledger changes remain to be covered. The late legacy
presentation signal is implemented, but its own full completion integration
proof is still required. Current Cloud blockers and single-active-writer limits
remain unchanged.
### Worker and producer verification
15 real-PostgreSQL worker cases pass in
`/tmp/sleep28-worker-question-tests.log`. They cover configured idle/internal
runs/old history, startup and restart, first committed work, outer commit and
nested rollback, first wake during drain, issue/company isolation, suppressed
lineage reactivation, late presentation eligibility, 205 notices plus another
company, selection failure through drain, publication-write retry, a concurrent
commit during projection, shutdown, real cancellation and real question delivery.
Quiet assertions advance two minutes and observe zero DB selects/transactions.
The deadline test verifies no scans during the ten seconds before retry.
The two real producer proofs are causal, not just notification-spy assertions:
- Removing the actual `writeRunStatus` signal makes the cancellation test fail:
the queued notice exists but the terminal notice is absent. Restoring it passes.
Log: `/tmp/sleep28-cancellation-without-signal.log`.
- The question test creates and answers a real interaction, supplies a durable
published prompt, and invokes the real delivery service. Its wake callback
inserts the continuation without a milestone signal and proves no notice is
projected yet. The terminal receipt then activates the worker. Removing that
exact signal strands the notice and fails the assertion. Log:
`/tmp/sleep28-question-without-signal.log`.
The 31-case existing chat integration selection passes after the selector
refactor and producer additions, including presentation authority and native
progress. Log: `/tmp/sleep28-scoped-acceptance.log`. Expected contention/retry
warnings occurred, with all assertions passing. Server production typecheck
passes in `/tmp/sleep28-producer-typecheck.log`. No full-workspace or PR gate
completion is claimed yet.
Preserved fixture failures:
1. `/tmp/sleep28-worker-tests.log`: all 11 initial cases failed setup because the
fixture used provider name `slack` as connection transport. The schema requires
`chat_sdk`; no production transport change was made.
2. `/tmp/sleep28-worker-fixed-tests.log`: ten passed; the same-company isolation
fixture violated the application-name uniqueness constraint. Fixture names now
include their identity.
3. `/tmp/sleep28-worker-producer-tests.log`: eleven passed; cancellation required
a responsible-user membership, and teardown omitted its new run-event rows.
Both are now represented/cleaned by the fixture.
4. `/tmp/sleep28-worker-producer-fixed-tests.log`: assertions reached cancellation
projection, but default follow-on recovery admitted another execution and
teardown deadlocked against it. The test now uses cancellation's existing
`suppressImmediateRecovery` option to exercise Stop without starting successor
execution. This does not change the default production recovery policy.
5. All 12 corrected cases passed in `/tmp/sleep28-worker-final-tests.log`; two
concurrency/retry cases made 14 pass in `/tmp/sleep28-worker-expanded-tests.log`;
the real question case brings the latest total to 15. These counts overlap.
This checkpoint is progress on slice 28, not a new independently complete reason
or a claim that milestone polling is retired. The original full PR-stack scope,
older unresolved CI/review findings, final master reconciliation and each PR's
latest-head acceptance remain required.
@@ -0,0 +1,261 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { activityLog, agents, authUsers, companyMemberships, chatConversations, chatEndpoints, chatMessageLinks, chatPublications, companies, createDb,
heartbeatRunEvents, heartbeatRuns, issueComments, issueQuestionResponseDeliveries, issueThreadInteractions, issues, toolApplications, toolConnections } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
import { issueThreadInteractionService } from "../services/issue-thread-interactions.js";
import { questionResponseDeliveryService } from "../services/question-response-delivery.js";
import { heartbeatService } from "../services/heartbeat.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MILESTONE_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("chat lifecycle milestone worker", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: ReturnType<typeof createDb>;
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
let enabled = true;
const errors: unknown[] = [];
const fixtureCompanies: string[] = [];
const fixtureUsers: string[] = [];
type Fixture = Awaited<ReturnType<typeof fixture>>;
let f: Fixture;
beforeAll(async () => {
if (!external) temporary = await startEmbeddedPostgresTestDatabase("chat-milestones-");
db = createDb(external ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
}, 60_000);
async function fixture(companyId = randomUUID()) {
const agentId = randomUUID(), issueId = randomUUID(), endpointId = randomUUID(), conversationId = randomUUID(), commentId = randomUUID(), userId = randomUUID();
if (!fixtureCompanies.includes(companyId)) {
fixtureCompanies.push(companyId);
await db.insert(companies).values({ id: companyId, name: "Milestones", issuePrefix: `M${companyId.slice(0, 7)}` });
}
fixtureUsers.push(userId);
await db.insert(authUsers).values({ id: userId, name: "Owner", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "owner" });
await db.insert(agents).values({ id: agentId, companyId, name: "Maya", adapterType: "paperclip_runner" });
await db.insert(issues).values({ id: issueId, companyId, title: "Chat task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: userId });
const applicationId = randomUUID(), connectionId = randomUUID();
await db.insert(toolApplications).values({ id: applicationId, companyId, name: `Chat ${applicationId}`, type: "chat" });
await db.insert(toolConnections).values({ id: connectionId, companyId, applicationId, name: "Chat", uid: connectionId, transport: "chat_sdk", connectionPurpose: "channel", enabled: true, status: "active" });
await db.insert(chatEndpoints).values({ id: endpointId, companyId, connectionId, assignedAgentId: agentId, publicId: endpointId, provider: "slack", status: "active" });
await db.insert(chatConversations).values({ id: conversationId, companyId, endpointId, issueId, externalConversationId: conversationId, externalThreadId: conversationId, externalLabel: "Chat" });
await db.insert(issueComments).values({ id: commentId, companyId, issueId, authorType: "user", body: "Start work" });
await db.insert(chatMessageLinks).values({ companyId, endpointId, conversationId, commentId, providerMessageId: commentId, direction: "inbound" });
return { companyId, agentId, issueId, endpointId, conversationId, commentId, userId };
}
beforeEach(async () => {
enabled = true; errors.length = 0; f = await fixture();
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
});
afterEach(async () => {
await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks();
for (const companyId of fixtureCompanies.splice(0)) {
await db.delete(activityLog).where(eq(activityLog.companyId, companyId));
await db.delete(chatMessageLinks).where(eq(chatMessageLinks.companyId, companyId));
await db.delete(chatPublications).where(eq(chatPublications.companyId, companyId));
await db.delete(chatConversations).where(eq(chatConversations.companyId, companyId));
await db.delete(chatEndpoints).where(eq(chatEndpoints.companyId, companyId));
await db.delete(issueQuestionResponseDeliveries).where(eq(issueQuestionResponseDeliveries.companyId, companyId));
await db.delete(issueThreadInteractions).where(eq(issueThreadInteractions.companyId, companyId));
await db.delete(issueComments).where(eq(issueComments.companyId, companyId));
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, companyId));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
await db.delete(issues).where(eq(issues.companyId, companyId));
await db.delete(agents).where(eq(agents.companyId, companyId));
await db.delete(companyMemberships).where(eq(companyMemberships.companyId, companyId));
await db.delete(companies).where(eq(companies.id, companyId));
}
for (const userId of fixtureUsers.splice(0)) await db.delete(authUsers).where(eq(authUsers.id, userId));
expect(errors).toEqual([]);
});
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
function runValues(scope = f, patch: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
return { id: randomUUID(), companyId: scope.companyId, agentId: scope.agentId, responsibleUserId: scope.userId, status: "running", updatedAt: new Date(),
contextSnapshot: { issueId: scope.issueId, source: "chat:slack", endpointId: scope.endpointId, wakeCommentId: scope.commentId }, ...patch };
}
async function run(scope = f, patch: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
const values = runValues(scope, patch);
await db.transaction(async tx => { await notifyChatRunMilestones(tx, scope); await tx.insert(heartbeatRuns).values(values); });
return values;
}
async function start() {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => enabled, canReconcile: () => true, onError: error => errors.push(error) });
await chatRunMilestoneWork(db).register(coordinator, () => enabled).ready;
}
async function flush() {
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
}
async function quiet(ms = 120_000) {
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
try { await vi.advanceTimersByTimeAsync(ms); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
finally { reads.mockRestore(); writes.mockRestore(); }
}
const publications = (scope = f) => db.select().from(chatPublications).where(eq(chatPublications.issueId, scope.issueId));
it("leaves configured idle, internal runs and old history quiet", async () => {
await run(f, { contextSnapshot: { issueId: f.issueId, source: "issue.comment" } });
await run(f, { updatedAt: new Date(Date.now() - 2 * 86_400_000) });
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
});
it("reconstructs missing notices at startup and leaves published history quiet across restart", async () => {
const row = await run(); await start(); await flush();
expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${row.id}:working:${f.endpointId}` })]);
await quiet(); await coordinator!.stop(); await start(); await flush(); await quiet();
expect(await publications()).toHaveLength(1);
});
it("wakes on the first committed row after becoming quiet", async () => {
await start(); await flush(); await quiet(); await run(); await flush();
expect(await publications()).toHaveLength(1); await quiet();
});
it("wakes from the real run cancellation transaction without a manual milestone signal", async () => {
const row = await run(f, { status: "queued", runtimeMode: "legacy" });
await start(); await flush(); await quiet();
// This fixture exercises the Stop producer, not successor execution.
await heartbeatService(db).cancelRun(row.id, "Operator stopped queued work", { suppressImmediateRecovery: true });
await flush();
expect((await publications()).map(publication => publication.idempotencyKey).sort()).toEqual([
`run:${row.id}:failed:${f.endpointId}`, `run:${row.id}:queued:${f.endpointId}`,
].sort());
await quiet();
});
it("projects a continuation only after the real question delivery receipt commits", async () => {
const source = await run(f, { status: "succeeded" });
const interactions = issueThreadInteractionService(db);
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
options: [{ id: "yes", label: "Yes" }],
}] },
}, { agentId: f.agentId, runId: source.id });
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
// A provider acknowledgement is durable input to this projection test.
await db.insert(chatPublications).values({ companyId: f.companyId, endpointId: f.endpointId, conversationId: f.conversationId,
issueId: f.issueId, idempotencyKey: `interaction:${interaction.id}:${f.endpointId}`,
payload: { text: "Continue?", interactionId: interaction.id }, state: "published", providerMessageId: "question-1",
}).onConflictDoNothing();
await db.update(chatPublications).set({ state: "published", providerMessageId: "question-1" }).where(eq(chatPublications.issueId, f.issueId));
await start(); await flush(); await quiet();
let targetId!: string;
const delivery = questionResponseDeliveryService(db, { heartbeat: {
cancelRun: async () => { throw new Error("The source is already terminal"); },
wakeup: async (_agentId, options) => {
const values = runValues(f, { status: "queued", contextSnapshot: options.contextSnapshot });
const [target] = await db.insert(heartbeatRuns).values(values).returning();
targetId = target.id;
await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${target.id}:`))).toBe(false);
return target;
},
} });
await expect(delivery.deliver(interaction.id)).resolves.toMatchObject({ status: "fallback_queued" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${targetId}:queued:${f.endpointId}`);
await quiet();
});
it("waits for outer commit and ignores rolled-back nested writes", async () => {
await start(); await flush();
await expect(db.transaction(async tx => {
await tx.transaction(async nested => { await notifyChatRunMilestones(nested, f); await nested.insert(heartbeatRuns).values(runValues()); });
throw new Error("rollback fixture");
})).rejects.toThrow("rollback fixture");
await flush(); expect(await publications()).toEqual([]); await quiet();
let release!: () => void, staged!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const stagedPromise = new Promise<void>(resolve => { staged = resolve; });
const writing = db.transaction(async tx => {
await tx.transaction(async nested => { await notifyChatRunMilestones(nested, f); await nested.insert(heartbeatRuns).values(runValues()); });
staged(); await gate;
});
try { await stagedPromise; await vi.advanceTimersByTimeAsync(1000); expect(await publications()).toEqual([]); }
finally { release(); await writing; }
await flush(); expect(await publications()).toHaveLength(1); await quiet();
});
it("retains the first committed wake during drain without periodic SQL", async () => {
await start(); await flush(); enabled = false;
await run(); await vi.advanceTimersByTimeAsync(1000);
expect(await publications()).toEqual([]); await quiet();
enabled = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
expect(await publications()).toHaveLength(1); await quiet();
});
it("scopes mutation wakes to their issue and company", async () => {
const sibling = await fixture(f.companyId), foreign = await fixture();
await start(); await flush();
// Unsupported raw writers deliberately do not wake: they provide a sentinel
// proving a supported issue mutation does not perform a global scan.
await db.insert(heartbeatRuns).values([runValues(sibling), runValues(foreign)]);
await run(); await flush();
expect(await publications()).toHaveLength(1); expect(await publications(sibling)).toEqual([]); expect(await publications(foreign)).toEqual([]);
await db.transaction(tx => notifyChatRunMilestones(tx, { companyId: f.companyId }));
await flush(); expect(await publications(sibling)).toHaveLength(1); expect(await publications(foreign)).toEqual([]);
await quiet();
});
it("disarms suppressed lineage and reactivates after its binding changes", async () => {
const row = await run(f, { contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: randomUUID(), wakeCommentId: f.commentId } });
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, f);
await tx.update(heartbeatRuns).set({ contextSnapshot: runValues().contextSnapshot }).where(eq(heartbeatRuns.id, row.id));
});
await flush(); expect(await publications()).toHaveLength(1); await quiet();
});
it("waits quietly for a late presentation decision", async () => {
const row = await run(f, { status: "succeeded" }); await start(); await flush();
expect(await publications()).toEqual([]); await quiet();
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, f);
await tx.update(heartbeatRuns).set({ resultJson: { presentationDecision: { source: "none" } } }).where(eq(heartbeatRuns.id, row.id));
});
await flush(); expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${row.id}:completed:${f.endpointId}` })]); await quiet();
});
it("drains more than one batch without duplicate notices or starving another company", async () => {
const other = await fixture();
await db.insert(heartbeatRuns).values([...Array.from({ length: 205 }, () => runValues()), runValues(other)]);
await start(); await flush(); expect(await publications()).toHaveLength(205); expect(await publications(other)).toHaveLength(1); await quiet();
});
it("retains a failed selection for retry through drain", async () => {
await run();
const select = vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("temporary read failure"); });
await start(); select.mockRestore(); expect(errors.splice(0)).toHaveLength(1);
enabled = false; await quiet(); enabled = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
expect(await publications()).toHaveLength(1); await quiet();
});
it("retries a failed publication write without idle scans before its deadline", async () => {
await run();
const transaction = vi.spyOn(db, "transaction").mockRejectedValueOnce(new Error("temporary publication failure"));
await start(); transaction.mockRestore(); expect(errors.splice(0)).toHaveLength(1);
expect(await publications()).toEqual([]); await quiet(10_000);
await vi.advanceTimersByTimeAsync(30_000); await flush(); expect(await publications()).toHaveLength(1); await quiet();
});
it("preserves a committed change arriving while a projection batch is running", async () => {
await run();
const transaction = db.transaction.bind(db);
let release!: () => void, entered!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const staged = new Promise<void>(resolve => { entered = resolve; });
const held = vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => {
entered(); await gate; return transaction(callback, config);
});
const starting = start();
try {
await staged;
await transaction(async tx => { await notifyChatRunMilestones(tx, f); await tx.insert(heartbeatRuns).values(runValues()); });
} finally { release(); await starting; held.mockRestore(); }
await flush(); expect(await publications()).toHaveLength(2); await quiet();
});
it("stops future wakes and reconstructs writes made after shutdown", async () => {
await start(); await flush(); await coordinator!.stop(); await run(); await quiet();
expect(await publications()).toEqual([]); await start(); await flush(); expect(await publications()).toHaveLength(1);
});
});
@@ -0,0 +1,9 @@
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
/** Lineage changes can authorize descendants, so invalidate the issue rather
* than just the run whose row changed. Configuration may invalidate a company. */
export type ChatRunMilestoneScope = { companyId: string; issueId?: string };
export function notifyChatRunMilestones(tx: object, scope: ChatRunMilestoneScope) {
return notifyDeliveryWork(tx, DELIVERY_QUEUES.chatRunMilestones, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }));
}
@@ -0,0 +1,38 @@
import type { Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import { registerScopedDeadlineWork } from "./scoped-deadline-work.js";
import type { ChatRunMilestoneScope } from "./chat-run-milestone-notifications.js";
import { enqueueChatRunLifecycleMilestones, loadChatRunLifecycleMilestoneScopes } from "./chat-run-publications.js";
type Entry = { id: string; companyId: string; issueId: string; dueAt: number };
const batchSize = 100;
/** No periodic discovery: reconstruct on startup/committed changes, drain
* current notices, and retain a retry only for failure or an unfinished batch. */
export function chatRunMilestoneWork(db: Db, publicBaseUrl?: string | null) {
async function load(scope?: ChatRunMilestoneScope, signal?: AbortSignal): Promise<Entry[]> {
return (await loadChatRunLifecycleMilestoneScopes(db, scope, signal)).map(row => ({
...row, id: `${row.companyId}:${row.issueId}`, dueAt: 0,
}));
}
async function check(entry: Entry, signal?: AbortSignal): Promise<Entry | null> {
const inserted = await enqueueChatRunLifecycleMilestones(db, {
scope: { companyId: entry.companyId, issueId: entry.issueId },
publicBaseUrl, limit: batchSize, signal,
});
// A short batch exhausted the selector, including currently suppressed
// candidates. Their dependencies must signal a later eligibility change.
return inserted === batchSize ? { ...entry, dueAt: Date.now() } : null;
}
function register(coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
return registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.chatRunMilestones,
decodeScope: (hint: string): ChatRunMilestoneScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (entry: Entry, scope?: ChatRunMilestoneScope) => !scope ||
(entry.companyId === scope.companyId && (!scope.issueId || entry.issueId === scope.issueId)),
load, check,
}, canRun);
}
return { load, check, register };
}
+65 -33
View File
@@ -1,3 +1,4 @@
import type { ChatRunMilestoneScope } from "./chat-run-milestone-notifications.js";
import { notifyChatPublicationWork } from "./chat-work-notifications.js";
import {
and,
@@ -654,21 +655,19 @@ export async function enqueueSafeNativeChatProgress(
return inserted;
}
/**
* Project only coarse run lifecycle into bound external conversations. Raw
* output, errors, tool events, and reasoning stay in Paperclip. Idempotency is
* keyed by run, milestone, and endpoint so polling and restarts are harmless.
*/
export async function enqueueChatRunLifecycleMilestones(
db: Db,
input: ChatRunProjectionInput = {},
): Promise<number> {
const { since, limit } = chatRunProjectionWindow(input);
const issueIdFromContext = boundRunIssueIdSql();
const explicitlyAuthoredCommentReason = sql<boolean>`(
type ChatRunMilestoneCursor = {
updatedAt: string;
runId: string;
conversationId: string;
};
const explicitlyAuthoredCommentReason = sql<boolean>`(
${issueComments.metadata} ->> 'authorizationReason' = 'paperclip_runner_protocol'
or left(coalesce(${issueComments.metadata} ->> 'authorizationReason', ''), 6) = 'allow_'
)`;
function lifecycleMilestonePages(db: Db, since: Date, scope?: ChatRunMilestoneScope) {
const issueIdFromContext = boundRunIssueIdSql();
const milestoneFromStatus = sql<string>`case
when ${heartbeatRuns.status} = 'queued' then 'queued'
when ${heartbeatRuns.status} = 'running'
@@ -757,21 +756,8 @@ export async function enqueueChatRunLifecycleMilestones(
and continuation_wake.payload->>'issueId' = ${issueIdFromContext}
)
)`;
let inserted = 0;
let cursor: {
updatedAt: string;
runId: string;
conversationId: string;
} | null = null;
const bindingsCache = new Map<
string,
Awaited<ReturnType<typeof resolveChatOriginPublicationBindings>>
>();
const providerInteractionCache = new Map<string, boolean>();
while (inserted < limit) {
const pageSize = Math.max(25, Math.min(200, limit - inserted));
const pageCursor: typeof cursor = cursor;
const rows: ChatRunMilestoneCandidate[] = await db
return (pageSize: number, pageCursor: ChatRunMilestoneCursor | null): Promise<ChatRunMilestoneCandidate[]> => {
return db
.select({
runId: heartbeatRuns.id,
runStatus: heartbeatRuns.status,
@@ -934,6 +920,8 @@ export async function enqueueChatRunLifecycleMilestones(
and target_delivery.status in ('delivered', 'fallback_queued')
)`,
),
scope ? eq(heartbeatRuns.companyId, scope.companyId) : undefined,
scope?.issueId ? eq(chatConversations.issueId, scope.issueId) : undefined,
gte(heartbeatRuns.updatedAt, since),
isNull(chatPublications.id),
pageCursor
@@ -948,15 +936,59 @@ export async function enqueueChatRunLifecycleMilestones(
asc(chatConversations.id),
)
.limit(pageSize);
};
}
function milestoneCursor(row: ChatRunMilestoneCandidate): ChatRunMilestoneCursor {
return { updatedAt: row.runUpdatedAt, runId: row.runId, conversationId: row.conversationId };
}
/** Reconstruct issue-scoped projection work. Candidate rows are only hints;
* projection still applies current lineage and provider authority. */
export async function loadChatRunLifecycleMilestoneScopes(
db: Db,
scope?: ChatRunMilestoneScope,
signal?: AbortSignal,
): Promise<Array<{ companyId: string; issueId: string }>> {
const pages = lifecycleMilestonePages(db, chatRunProjectionWindow({}).since, scope);
const scopes = new Map<string, { companyId: string; issueId: string }>();
let cursor: ChatRunMilestoneCursor | null = null;
for (;;) {
signal?.throwIfAborted();
const rows = await pages(100, cursor);
for (const row of rows) scopes.set(`${row.companyId}:${row.issueId}`, { companyId: row.companyId, issueId: row.issueId });
if (rows.length < 100) return [...scopes.values()];
cursor = milestoneCursor(rows[rows.length - 1]);
}
}
/**
* Project only coarse run lifecycle into bound external conversations. Raw
* output, errors, tool events, and reasoning stay in Paperclip. Idempotency is
* keyed by run, milestone, and endpoint so polling and restarts are harmless.
*/
export async function enqueueChatRunLifecycleMilestones(
db: Db,
input: ChatRunProjectionInput & { scope?: ChatRunMilestoneScope; signal?: AbortSignal } = {},
): Promise<number> {
const { since, limit } = chatRunProjectionWindow(input);
let inserted = 0;
const pages = lifecycleMilestonePages(db, since, input.scope);
let cursor: ChatRunMilestoneCursor | null = null;
const bindingsCache = new Map<
string,
Awaited<ReturnType<typeof resolveChatOriginPublicationBindings>>
>();
const providerInteractionCache = new Map<string, boolean>();
while (inserted < limit) {
const pageSize = Math.max(25, Math.min(200, limit - inserted));
input.signal?.throwIfAborted();
const rows = await pages(pageSize, cursor);
if (rows.length === 0) break;
const lastRow = rows.at(-1)!;
cursor = {
updatedAt: lastRow.runUpdatedAt,
runId: lastRow.runId,
conversationId: lastRow.conversationId,
};
cursor = milestoneCursor(rows[rows.length - 1]);
for (const row of rows) {
input.signal?.throwIfAborted();
if (inserted >= limit) break;
const milestone = milestoneForStatus(row.runStatus, row.runErrorCode);
if (!milestone) continue;
@@ -54,6 +54,7 @@ export const DELIVERY_QUEUES = {
chatVerification: "chat-verification",
chatInbound: "chat-inbound",
chatPublications: "chat-publications",
chatRunMilestones: "chat-run-milestones",
chatReceipts: "chat-slack-receipts",
browser: "browser-use",
email: "email-channels",
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { measureRunPhase } from "../execution-preparation-timing.js";
import { pendingCaseHandoffForRun } from "./case-handoff-lifecycle.js";
import { NON_RETRYABLE_PREFLIGHT_FAILURE_CODES, isHeartbeatRunTerminalStatus } from "./run-lifecycle.js";
@@ -889,13 +890,13 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
};
}
await db
.update(heartbeatRuns)
.set({
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: livenessRun.companyId, issueId: issueId ?? undefined });
await tx.update(heartbeatRuns).set({
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ presentationDecision })}::jsonb`,
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, livenessRun.id));
}).where(eq(heartbeatRuns.id, livenessRun.id));
});
await appendRunEvent(livenessRun, {
eventType: "run.presentation.resolved",
stream: "system",
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import os from "node:os";
import {
@@ -1018,6 +1019,7 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
const tx = transaction as unknown as Db;
if (isHeartbeatRunTerminalStatus(status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, companyId);
await notifyExecutionRunWork(tx);
if (companyId) await notifyChatRunMilestones(tx, { companyId });
return write(tx);
});
}
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { and, asc, desc, eq, inArray, isNull, lte, or, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
@@ -504,6 +505,7 @@ export function questionResponseDeliveryService(
}): Promise<QuestionResponseDeliveryOutcome> {
const at = now();
const updated = await db.transaction(async (tx) => {
await notifyChatRunMilestones(tx, { companyId: input.delivery.companyId, issueId: input.delivery.issueId });
const row = await tx
.update(issueQuestionResponseDeliveries)
.set({