Files
PaperClipAI/server/src/__tests__/chat-run-milestone-work.test.ts
T
DottaandPaperclip 5f0da562db fix(server): wake chat lifecycle milestones from committed changes (#17606)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected chats show safe notices when a run starts, waits,
finishes, or fails.
> - A recurring scan projected these notices even when no eligible work
existed.
> - Eligibility depends on committed run state, conversation lineage,
and current authority.
> - Existing database rows can reconstruct that work after restart.
> - This PR wakes a scoped worker from those writes and removes
lifecycle polling.
> - Native progress events keep their separate existing owner until the
next migration.

## Linked Issues or Issue Description

**What happened?**

The chat maintenance timer repeatedly scanned run lifecycle state. Empty
and settled instances still queried the database. A run-status event
alone cannot replace the scan: prompt receipts, continuation receipts,
identity changes, and restored authority can make a run eligible later.

**What did you expect?**

Committed changes wake projection. Startup reconstructs eligible work. A
completed pass stops until another relevant change arrives. Failed work
retains recovery without scanning settled history forever.

**Steps to reproduce**

Configure an automatic chat endpoint. Finish its work, or leave it
unused. Observe recurring lifecycle projection queries from the chat
maintenance callback.

Stacked on #17592. Related foundation: #15625. This PR owns lifecycle
milestones only.

## What Changed

- Preserve upstream coalesced Cloud membership writes together with
locked membership restoration and its committed chat notification.

- Signal milestone eligibility from the real Slack account-link
transaction.

- Register the lifecycle worker with the existing delivery coordinator.
Remove lifecycle projection from the periodic chat callback. Keep native
progress in a separately named lane and preserve the explicit combined
API.
- Reconstruct company/issue scopes from the existing selector. Drain
bounded batches with company fairness. Retry failures, retain writes
through drain/resume, and disarm settled work.
- Signal from supported run, presentation, interaction, receipt,
lineage, conversation, recovery, and authority writers. Preserve their
transaction ownership, including nested savepoints and uncertain
settlement. No new table is added.
- Preserve provider authorization, authored-final suppression,
responsible-user continuity, deduplication, the 24-hour selection
window, and raw-output restrictions. Keep provider interpretation in its
owner.
- Preserve PostgreSQL timestamp precision in lifecycle pagination.
Rejected submillisecond candidates must not repeat the same page and
starve later notices.
- Document producer coverage and alternate-writer dispositions in
`doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`.

## Verification

Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting
master after #17592 merged at
`7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates
the actual landed parent. The entire 77-path milestone patch is
binary-identical to the locally verified preparation across that
ancestry reconciliation; both unrelated master paths are preserved. The
four earlier producer/test integration conflicts retain scoped milestone
and voice notices, current onboarding completion admission, and expanded
identity query/read-scope coverage.

- **330 distinct selected cases pass on the verified unchanged patch:**
305 across 14 complete suites (all five milestone worker suites, run
identity, board claim, Cloud provisioning, auth session routes, Cloud
actor authentication, stale-lock recovery, GitHub credentials, project
Git authorization, and app admission wiring), plus 25 selected real chat
producer cases. The remaining 1,590 chat integration cases were skipped,
not passed.
- The real producer cases cover saved milestone discovery, GitHub and
Slack identity changes, account-link rollback/reconfirmation, ordinary
ingress reactivation, post-close action/delivery receipt settlement, all
ten authority-restoration paths including Cloud synchronization, and
changed native review metadata.
- Worker coverage verifies outer commit, rollback, uncertain-write
recovery, restart, bounded batching/fairness, timestamp precision,
retries, drain/resume, and settled SQL quiet. Authorization and existing
publication deduplication remain with their owning services.
- Server production typechecking, build, and module-boundary checks
pass. The tests above were executed separately. Production typechecking
excludes `src/__tests__` and does not replace those runs. Each database
owner uses its own newly created and app-migrated PostgreSQL fixture.
- [Current-parent acceptance and preserved failure
ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28)
records commands, counts, producer scope, conflict dispositions, and
limits. Earlier combined-stack counts and failures keep their original
revision scope; local acceptance does not erase old CI failures.
- No new migration, queue table, dependency, workflow, or shared
scheduler contract is introduced. Parent migration 0397, bounded consent
recovery, voice cleanup/retention repairs, and runner checkpoint
restoration remain present.
- **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No
Ponytail review is present, which is an explicitly authorized
nonblocker; no failing review is waived. Older reviews and parent checks
are not current passes. Actual failing Ponytail findings must be fixed;
an absent/stopped/paused turn without a verdict is an authorized
nonblocker. Active unfinished Ponytail uses its original request/start
timer, with a 25-minute monitor cutoff respecting the user's 30-minute
maximum.
- No complete local repository matrix, live provider tenant, external
alarm, physical host/database suspension, or fleet savings claim is
made. Native progress remains a separate following slice.

## Risks

- A future eligibility writer must notify the existing projection
boundary. Independent SQL writers or another active server process are
outside this process-local signal contract.
- Authority and lineage remain sensitive. The worker uses the existing
selectors and publication policy; it does not grant execution or
disclosure authority.
- Native progress and other publication maintenance still have recurring
work. This PR removes one reason, not the entire chat maintenance timer.
- Worker quiet does not prove database suspension, host sleep, or
external alarm coverage. Cloud sleep blockers remain unchanged.
- Merge eligibility requires passing current-head CI and Greptile 5/5
after this actual-master reconciliation. Subsequent stack slices retain
their own gates.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window size are not exposed. Used repository inspection,
reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub
tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details available in this session)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the documented focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 11:07:54 -05:00

988 lines
67 KiB
TypeScript

import { documentService } from "../services/documents.js";
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { agents, agentWakeupRequests, caseHandoffs, cases, chatConversations, chatEndpoints, chatPublications, completionContracts, createDb, nativeRunFinalizations, nativeRunResults, statusDecisions, statusDecisionEffects, workAssessments, subscribeDatabaseWork, heartbeatRuns, issueComments, issueQuestionResponseDeliveries, issueThreadInteractions, issues } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
import { issueThreadInteractionService } from "../services/issue-thread-interactions.js";
import { questionResponseDeliveryService } from "../services/question-response-delivery.js";
import { createQueuedCommentIssueLockWriter } from "../modules/wake-queue/adapters/queued-comment-postgres.js";
import { createDiscardQueuedComment, createReorderQueuedComments } from "../modules/wake-queue/application/queued-comment-use-cases.js";
import { createRunDispatch } from "../modules/run-dispatch/index.js";
import { createAdmissionTransactionScope, createPostgresWakeQueueAdapter, createWakeAdmissionWriter } from "../modules/wake-queue/adapters/postgres.js";
import { instanceSettingsService } from "../services/instance-settings.js";
import { createHeartbeatQueue } from "../services/heartbeat/queue.js";
import { createHeartbeatQueueTestDependencies } from "./helpers/heartbeat-queue.js";
import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY, CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE } from "../services/chat-control-recovery-stop.js";
import { heartbeatService } from "../services/heartbeat.js";
import { commitNativeStatusDecision, dispatchPendingNativeStatusEffects } from "../services/native-runtime/status-decision-committer.js";
import { claimQueuedNativeReviewRun } from "../services/native-runtime/native-review-dispatch.js";
import { createChatPublicationResolution } from "../services/chat-channels/publications/resolution.js";
import { createChatPublicationWorker } from "../services/chat-channels/publications/worker.js";
import { CAPABILITIES } from "../services/chat-channels/provider-catalog.js";
import { createChatPublicationFailures } from "../services/chat-channels/publications/failures.js";
import { PaperclipRunnerToolAuthority } from "../services/native-runtime/paperclip-runner-tool-authority.js";
import { finalizeNativeRun } from "../services/native-runtime/native-run-finalizer.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MILESTONE_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("chat lifecycle milestone worker", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: ReturnType<typeof createDb>;
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
let enabled = true;
const errors: unknown[] = [];
let fixtures: ReturnType<typeof createChatMilestoneFixtures>;
type Fixture = Awaited<ReturnType<typeof fixture>>;
let f: Fixture;
beforeAll(async () => {
if (!external) temporary = await startEmbeddedPostgresTestDatabase("chat-milestones-");
db = createDb(external ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
fixtures = createChatMilestoneFixtures(db);
}, 60_000);
const fixture = (companyId?: string) => fixtures.create(companyId);
beforeEach(async () => {
enabled = true; errors.length = 0; f = await fixture();
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
});
afterEach(async () => {
await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks();
await fixtures.cleanup();
expect(errors).toEqual([]);
});
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
function runValues(scope = f, patch: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
return { id: randomUUID(), companyId: scope.companyId, agentId: scope.agentId, responsibleUserId: scope.userId, status: "running", updatedAt: new Date(),
contextSnapshot: { issueId: scope.issueId, source: "chat:slack", endpointId: scope.endpointId, wakeCommentId: scope.commentId }, ...patch };
}
async function run(scope = f, patch: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
const values = runValues(scope, patch);
await db.transaction(async tx => { await notifyChatRunMilestones(tx, scope); await tx.insert(heartbeatRuns).values(values); });
return values;
}
async function start() {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => enabled, canReconcile: () => true, onError: error => errors.push(error) });
await chatRunMilestoneWork(db).register(coordinator, () => enabled).ready;
}
async function flush() {
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
}
async function quiet(ms = 120_000) {
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
try { await vi.advanceTimersByTimeAsync(ms); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
finally { reads.mockRestore(); writes.mockRestore(); }
}
const publications = (scope = f) => db.select().from(chatPublications).where(eq(chatPublications.issueId, scope.issueId));
it("leaves configured idle, internal runs and old history quiet", async () => {
await run(f, { contextSnapshot: { issueId: f.issueId, source: "issue.comment" } });
await run(f, { updatedAt: new Date(Date.now() - 2 * 86_400_000) });
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
});
it("reconstructs missing notices at startup and leaves published history quiet across restart", async () => {
const row = await run(); await start(); await flush();
expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${row.id}:working:${f.endpointId}` })]);
await quiet(); await coordinator!.stop(); await start(); await flush(); await quiet();
expect(await publications()).toHaveLength(1);
});
it("wakes on the first committed row after becoming quiet", async () => {
await start(); await flush(); await quiet(); await run(); await flush();
expect(await publications()).toHaveLength(1); await quiet();
});
it("wakes from the real run cancellation transaction without a manual milestone signal", async () => {
const row = await run(f, { status: "queued", runtimeMode: "legacy" });
await start(); await flush(); await quiet();
// This fixture exercises the Stop producer, not successor execution.
await heartbeatService(db).cancelRun(row.id, "Operator stopped queued work", { suppressImmediateRecovery: true });
await flush();
expect((await publications()).map(publication => publication.idempotencyKey).sort()).toEqual([
`run:${row.id}:failed:${f.endpointId}`, `run:${row.id}:queued:${f.endpointId}`,
].sort());
await quiet();
});
it.each(["promote", "suppress"] as const)("wakes when real retry dispatch chooses to %s", async action => {
const row = await run(f, { status: "scheduled_retry", runtimeMode: "legacy", scheduledRetryAt: new Date(Date.now() - 1000), scheduledRetryReason: "transient_error" });
if (action === "suppress") await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId));
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
const dispatch = createRunDispatch(db);
const result = await dispatch.promoteScheduledRetry({ companyId: f.companyId, runId: row.id });
expect(result.outcome).toBe(action === "promote" ? "promoted" : "gate_suppressed");
await flush();
expect((await publications()).map(publication => publication.idempotencyKey)).toContain(
`run:${row.id}:${action === "promote" ? "queued" : "failed"}:${f.endpointId}`,
);
await quiet();
expect((await dispatch.promoteScheduledRetry({ companyId: f.companyId, runId: row.id })).outcome).toBe("not_promoted");
await quiet();
});
it("wakes when the real staleness gate cancels queued work", async () => {
const row = await run(f, { status: "queued", runtimeMode: "legacy", scheduledRetryReason: "max_turns_continuation" });
await start(); await flush(); await quiet();
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId));
const dispatch = createRunDispatch(db);
await dispatch.cancelStaleQueuedRun({ companyId: f.companyId, runId: row.id, expectedStatus: "queued" });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, row.id)))[0].status).toBe("cancelled");
await flush();
expect((await publications()).map(publication => publication.idempotencyKey)).toContain(`run:${row.id}:failed:${f.endpointId}`);
await quiet();
await dispatch.cancelStaleQueuedRun({ companyId: f.companyId, runId: row.id, expectedStatus: "queued" });
await quiet();
});
it.each(["deferred", "immediate"] as const)("wakes when real %s recovery admits chat work", async kind => {
const finishing = await run(f, { status: "succeeded", runtimeMode: "legacy", finishedAt: new Date(),
contextSnapshot: { issueId: f.issueId, source: "issue_assigned" },
});
const wakeId = randomUUID();
if (kind === "deferred") await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: f.agentId,
source: "on_demand", status: "deferred_issue_execution", payload: { issueId: f.issueId },
});
const adapter = createPostgresWakeQueueAdapter(db, {
resolveResponsibleUserId: async () => f.userId,
getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }),
resolveSessionBeforeForWakeup: async () => null,
});
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
let admittedId: string | undefined;
await adapter.withIssueExecutionLock({ companyId: f.companyId, runId: finishing.id, now: new Date() }, async (locked, ports) => {
const input = { companyId: f.companyId, issue: locked.primaryIssue, finishingRun: locked.run,
contextSnapshot: runValues().contextSnapshot, responsibleUserId: f.userId, sessionBefore: null, now: new Date(),
};
const agent = { id: f.agentId, companyId: f.companyId, name: "Maya", invokable: true };
if (kind === "deferred") {
expect(await ports.transaction.claimDeferredWakeForPromotion({ companyId: f.companyId, wakeId, now: new Date() })).toBe(true);
admittedId = (await ports.transaction.finalizePromotedWake({ ...input, wakeId, deferredAgent: agent,
reason: "issue_execution_promoted", source: "on_demand", triggerDetail: null, payload: {},
})).id;
} else {
admittedId = (await ports.transaction.queueImmediateRecoveryRun({ ...input, recoveryAgent: agent, reason: "issue_execution_recovery" })).id;
}
expect(await publications()).toEqual([]);
return { outcome: { kind: "released" }, postCommitEffects: [] };
});
expect(admittedId).toBeDefined();
await flush();
expect((await publications()).map(publication => publication.idempotencyKey)).toContain(`run:${admittedId}:queued:${f.endpointId}`);
await quiet();
});
async function queue() {
const dependencies = createHeartbeatQueueTestDependencies(db);
const [agent] = await db.select().from(agents).where(eq(agents.id, f.agentId));
dependencies.getAgent.mockResolvedValue(agent);
dependencies.instanceSettings.getExperimental.mockImplementation(() => instanceSettingsService(db).getExperimental());
return { service: createHeartbeatQueue(db, dependencies), dependencies };
}
it("wakes on real queue admission while execution is held by drain", async () => {
const { service, dependencies } = await queue();
dependencies.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" });
await start(); await flush(); await quiet();
const admitted = await service.enqueueWakeup(f.agentId, { source: "on_demand", reason: "issue_commented",
contextSnapshot: runValues().contextSnapshot, requestedByActorType: "user", requestedByActorId: f.userId,
});
expect(admitted).toMatchObject({ status: "queued", companyId: f.companyId });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${admitted!.id}:queued:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled();
await quiet();
});
it.each(["ordinary", "legacy_comments", "queued_comments"] as const)("wakes on a real %s queue claim", async kind => {
const wakeId = randomUUID();
const context = runValues().contextSnapshot;
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: f.agentId,
source: "on_demand", status: "queued",
payload: kind === "queued_comments" ? { _paperclipWakeContext: { wakeCommentIds: [f.commentId] } } : {},
});
const seeded = await run(f, { status: "queued", runtimeMode: "legacy", wakeupRequestId: wakeId,
contextSnapshot: kind === "ordinary" ? context : { ...context, wakeCommentIds: [f.commentId] },
});
await db.update(agentWakeupRequests).set({ runId: seeded.id }).where(eq(agentWakeupRequests.id, wakeId));
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id));
const { service, dependencies } = await queue();
await start(); await flush(); await quiet();
expect(await service.claimQueuedRun(before)).toMatchObject({ id: seeded.id, status: "running" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:working:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled();
await quiet();
expect(await service.claimQueuedRun(before)).toBeNull();
await quiet();
});
it("wakes when claim rejects a superseded case handoff and leaves repeated claims quiet", async () => {
const wakeId = randomUUID();
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: f.agentId,
source: "assignment", status: "queued", payload: { issueId: f.issueId },
});
const seeded = await run(f, { status: "queued", runtimeMode: "legacy", wakeupRequestId: wakeId });
const [record] = await db.insert(cases).values({ companyId: f.companyId, caseNumber: 1, identifier: "M-CASE-1",
caseType: "work", title: "Chat task", workIssueId: f.issueId, handoffVersion: 2 }).returning();
await db.insert(caseHandoffs).values({ companyId: f.companyId, caseId: record.id, issueId: f.issueId,
version: 1, toAgentId: f.agentId, phase: "implementation", instructions: "Build the change", actorType: "system",
actorId: "test", idempotencyKey: "superseded", inputFingerprint: "fixture", wakeupRequestId: wakeId,
expectedIssueStatusVersion: 0, appliedIssueStatusVersion: 0, state: "delivered" });
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id));
const { service, dependencies } = await queue();
await start(); await flush(); await quiet();
expect(await service.claimQueuedRun(before)).toBeNull();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id)))[0])
.toMatchObject({ status: "cancelled", errorCode: "case_handoff_superseded" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:failed:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled();
await quiet();
expect(await service.claimQueuedRun(before)).toBeNull();
await quiet();
});
it("wakes when dispatch fails an unverifiable chat recovery and leaves repeated gates quiet", async () => {
const seeded = await run(f, { status: "running", runtimeMode: "legacy", invocationSource: "automation",
runnerProfileJson: { [CHAT_CONTROL_RECOVERY_ADMISSION_KEY]: {} },
});
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id));
const { service, dependencies } = await queue();
const dispatch = vi.fn(async () => null);
await start(); await flush(); await quiet();
await service.withChatControlRecoveryGate(before, "dispatch", dispatch);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id)))[0])
.toMatchObject({ status: "failed", errorCode: CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE });
expect(dispatch).not.toHaveBeenCalled(); expect(dependencies.executeRun).not.toHaveBeenCalled();
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:failed:${f.endpointId}`);
await quiet();
await service.withChatControlRecoveryGate(before, "dispatch", dispatch);
await quiet();
});
it.each(["queued", "scheduled_retry"] as const)("publishes cancellation when admission replaces a reassigned %s owner", async status => {
const old = await run(f, { status, runtimeMode: "legacy", scheduledRetryAt: status === "scheduled_retry" ? new Date(Date.now() + 60_000) : null });
const [replacement] = await db.insert(agents).values({ companyId: f.companyId, name: "Replacement", adapterType: "paperclip_runner" }).returning();
await db.update(issues).set({ assigneeAgentId: replacement.id, executionRunId: old.id }).where(eq(issues.id, f.issueId));
const { service, dependencies } = await queue();
dependencies.getAgent.mockResolvedValue(replacement);
dependencies.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" });
await start(); await flush(); await quiet();
const admitted = await service.enqueueWakeup(replacement.id, { source: "on_demand", reason: "issue_commented",
contextSnapshot: runValues().contextSnapshot, requestedByActorType: "user", requestedByActorId: f.userId,
});
expect(admitted).toMatchObject({ agentId: replacement.id, status: "queued" });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, old.id)))[0])
.toMatchObject({ status: "cancelled", errorCode: status === "queued" ? "lock_released_on_reassignment" : "issue_reassigned" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${old.id}:failed:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled(); await quiet();
});
async function editableQueue(staleContext = false) {
const wakeId = randomUUID();
await db.update(issueComments).set({ authorUserId: f.userId }).where(eq(issueComments.id, f.commentId));
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: f.agentId,
source: "on_demand", status: "queued", payload: { issueId: f.issueId, _paperclipWakeContext: { wakeCommentIds: [f.commentId] } },
});
const seeded = await run(f, { status: "queued", runtimeMode: "legacy", wakeupRequestId: wakeId,
contextSnapshot: { ...runValues().contextSnapshot, wakeCommentId: staleContext ? randomUUID() : f.commentId },
});
await db.update(agentWakeupRequests).set({ runId: seeded.id }).where(eq(agentWakeupRequests.id, wakeId));
const issue = { id: f.issueId, companyId: f.companyId, assigneeAgentId: f.agentId, executionRunId: seeded.id };
const actor = { actorType: "user" as const, actorId: f.userId, agentId: null, runId: null, agentApiKeyId: null };
const issueLock = createQueuedCommentIssueLockWriter(db, {
syncCommentReferences: async () => {}, deleteCommentReferenceSource: async () => {}, syncCommentExternalObjectsSafely: async () => {},
});
const snapshot = await issueLock.withLockedQueue({ issue, actor, queueId: wakeId }, async locked => locked.queue);
return { seeded, issueLock, input: { issue, actor, queueId: wakeId, revision: snapshot.revision, now: new Date() } };
}
it("wakes when queued-comment mutation reconstructs run lineage from its authoritative queue", async () => {
const { seeded, issueLock, input } = await editableQueue(true);
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
await createReorderQueuedComments({ issueLock })({ ...input, orderedCommentIds: [f.commentId] });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:queued:${f.endpointId}`);
await quiet();
});
it("settles queued-comment discard intent only after the owning transaction commits", async () => {
const { seeded, input } = await editableQueue();
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, "chat-run-milestones", event => events.push(event));
try {
await db.transaction(async tx => {
const writer = createQueuedCommentIssueLockWriter(tx as unknown as typeof db, {
syncCommentReferences: async () => {}, deleteCommentReferenceSource: async () => {}, syncCommentExternalObjectsSafely: async () => {},
});
await createDiscardQueuedComment({ issueLock: writer })({ ...input, commentId: f.commentId });
expect(events).toContain("intent"); expect(events).not.toContain("settled");
});
expect(events).toContain("settled");
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id)))[0])
.toMatchObject({ status: "cancelled", errorCode: "queued_comment_discarded" });
} finally { unsubscribe(); }
});
it("wakes when the real coalescing writer gives an existing run chat lineage", async () => {
const seeded = await run(f, { contextSnapshot: { issueId: f.issueId, source: "issue_assigned" } });
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
await db.transaction(async tx => {
await createWakeAdmissionWriter().coalesceIntoActiveExecutionRun(
createAdmissionTransactionScope(f.companyId, tx as unknown as typeof db), {
companyId: f.companyId, activeExecutionRunId: seeded.id, mergedContextSnapshot: runValues().contextSnapshot,
agentId: f.agentId, source: "on_demand", triggerDetail: null, payload: { issueId: f.issueId },
requestedByActorType: "user", requestedByActorId: f.userId, idempotencyKey: null,
});
expect(await publications()).toEqual([]);
});
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:working:${f.endpointId}`);
await quiet();
});
it("wakes on a claim released for drain and leaves repeated release quiet", async () => {
const seeded = await run();
const { service } = await queue();
await start(); await flush(); await quiet();
await service.releaseRunClaimedJustBeforeSuppression(seeded.id);
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:queued:${f.endpointId}`);
await quiet();
await service.releaseRunClaimedJustBeforeSuppression(seeded.id);
await quiet();
});
it("wakes an admitted monitor through its previously committed source receipt", async () => {
const source = await run(f, { runtimeMode: "native", nativeIssueId: f.issueId });
await db.update(issues).set({ executionRunId: source.id }).where(eq(issues.id, f.issueId));
await start(); await flush(); await quiet();
const nextCheckAt = new Date(Date.now() + 60_000).toISOString();
const authority = new PaperclipRunnerToolAuthority(db, { ...f, runId: source.id });
const request = { tool: "set_task_monitor", callId: randomUUID(), arguments: {
idempotencyKey: "milestone-monitor", monitor: { nextCheckAt, notes: "Check the saved task" },
} };
await expect(authority.execute(request)).resolves.toMatchObject({ taskId: f.issueId, replayed: false });
// The monitor and source receipt commit together, before due admission. This
// fixture finishes the source without executing a harness; admission below
// uses the real queue, with execution held by task drain.
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, source.id));
await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, f.issueId));
await quiet();
const { service, dependencies } = await queue();
dependencies.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" });
const admitted = await service.enqueueWakeup(f.agentId, { source: "automation", reason: "issue_monitor_due",
idempotencyKey: `issue-monitor:${f.issueId}:${nextCheckAt}`,
payload: { issueId: f.issueId, nextCheckAt },
contextSnapshot: { issueId: f.issueId, source: "issue.monitor" },
requestedByActorType: "user", requestedByActorId: f.userId,
});
expect(admitted).toMatchObject({ status: "queued" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${admitted!.id}:queued:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled();
await quiet();
});
async function nativeFixture(phase = "committed") {
const contractId = randomUUID(), resultId = randomUUID();
await db.insert(completionContracts).values({
id: contractId, companyId: f.companyId, issueId: f.issueId, revision: 1,
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "phase6-v1", risk: "standard",
completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
contractJson: { objective: "Respond to chat" }, canonicalSha256: "milestone-contract",
createdByActorType: "system", createdByActorId: "test",
});
// Persisted audit-only routing can commit without a status decision. Recovery
// still owes the terminal run projection after the controller restarts.
const row = await run(f, { runtimeMode: "native", nativeIssueId: f.issueId, completionContractId: contractId,
resultJson: { nativeAttentionRouting: [{ decisionId: null, reasonCode: "attention_duplicate_suppressed",
materializedTargets: [{ kind: "existing_attention" }] }] },
});
await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId,
runId: row.id, completionContractId: contractId, serverFingerprint: resultId, schemaStatus: "accepted",
resultJson: { terminal: { runTerminalState: "cancelled" } }, canonicalSha256: "milestone-result",
});
await db.insert(nativeRunFinalizations).values({ runId: row.id, companyId: f.companyId, issueId: f.issueId,
phase, resultId,
});
return { row, contractId, resultId };
}
async function continuationDecisionFixture() {
const native = await nativeFixture("assessing");
const assessmentId = randomUUID();
await db.insert(workAssessments).values({ id: assessmentId, companyId: f.companyId, issueId: f.issueId,
runId: native.row.id, contractId: native.contractId, resultId: native.resultId, triggerKind: "run_result",
triggerActorCompanyId: f.companyId, priorIssueStatus: "in_progress", priorStatusVersion: 0,
policyVersion: "phase6-v1", assessmentJson: {}, inputDigest: assessmentId,
});
return { companyId: f.companyId, issueId: f.issueId, runId: native.row.id, assessmentId,
priorStatus: "in_progress", priorStatusVersion: 0, priorDecisionId: null,
decision: { policyVersion: "phase6-v1", statusAction: "preserve" as const, toStatus: "in_progress" as const,
reasonCode: "live_continuation_registered", unblockDescriptor: null,
effects: [{ kind: "enqueue_continuation" as const, agentId: f.agentId, continuationKind: "same_agent" as const,
summary: "Continue responding", idempotencyKey: `continuation:${native.row.id}` }],
},
};
}
it("wakes from a governed native reviewer claim and leaves repeated claims quiet", async () => {
const input = await continuationDecisionFixture();
const { decision } = await commitNativeStatusDecision({ db, ...input });
const [reviewer] = await db.insert(agents).values({ companyId: f.companyId, name: "Reviewer", adapterType: "paperclip_runner" }).returning();
await db.update(statusDecisions).set({ decisionJson: { ...decision.decisionJson, projectedStatusVersion: 1 } }).where(eq(statusDecisions.id, decision.id));
await db.update(issues).set({ status: "in_review", statusVersion: 1, lastStatusDecisionId: decision.id, executionRunId: null }).where(eq(issues.id, f.issueId));
await db.update(chatEndpoints).set({ assignedAgentId: reviewer.id }).where(eq(chatEndpoints.id, f.endpointId));
const [interaction] = await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId,
kind: "request_confirmation", status: "pending", sourceRunId: input.runId, addresseeAgentId: reviewer.id,
effectiveResolverPolicy: "anyone", resolverPolicyProvenance: "inherited",
payload: { version: 1, prompt: "Review the work", target: { type: "custom", key: "native_completion_review", revisionId: decision.id } },
}).returning();
const wakeId = randomUUID();
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: reviewer.id, source: "automation", status: "queued" });
const seeded = await run(f, { agentId: reviewer.id, status: "queued", runtimeMode: "native", nativeIssueId: f.issueId, wakeupRequestId: wakeId,
contextSnapshot: { ...runValues().contextSnapshot, nativeReviewInteractionId: interaction.id, nativeReviewDecisionId: decision.id },
});
await db.update(agentWakeupRequests).set({ runId: seeded.id }).where(eq(agentWakeupRequests.id, wakeId));
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id));
const claim = { run: before, agentNameKey: "reviewer", claimedAt: new Date(), claimValues: {} };
await start(); await flush(); await quiet();
expect(await claimQueuedNativeReviewRun(db, claim)).toMatchObject({ id: seeded.id, status: "running" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:working:${f.endpointId}`);
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(seeded.id);
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)))[0].status).toBe("claimed");
await quiet();
expect(await claimQueuedNativeReviewRun(db, claim)).toBeNull(); await quiet();
});
it("wakes from native committed-result projection and leaves repeated finalization quiet", async () => {
const { row } = await nativeFixture();
await start(); await flush(); await quiet();
await finalizeNativeRun({ db, runId: row.id, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
await flush();
expect((await publications()).map(publication => publication.idempotencyKey).sort()).toEqual([
`run:${row.id}:failed:${f.endpointId}`, `run:${row.id}:working:${f.endpointId}`,
].sort());
await quiet();
await finalizeNativeRun({ db, runId: row.id, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
// A no-op repair must not enqueue another scan. This also catches deferred
// coordinator callbacks that start after the finalizer returns.
await quiet();
expect(await publications()).toHaveLength(2);
});
it("signals a new native decision at outer commit but not an already committed replay", async () => {
const input = await continuationDecisionFixture();
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, "chat-run-milestones", event => events.push(event));
try {
await db.transaction(async tx => {
await commitNativeStatusDecision({ db: tx as unknown as typeof db, ...input });
expect(events).toEqual(["intent"]);
});
expect(events).toEqual(["intent", "settled"]);
events.length = 0;
await expect(commitNativeStatusDecision({ db, ...input })).resolves.toMatchObject({ replayed: true });
expect(events).toEqual([]);
} finally { unsubscribe(); }
});
it("wakes a suppressed continuation when its native effect delivery receipt commits", async () => {
const input = await continuationDecisionFixture();
const { decision } = await commitNativeStatusDecision({ db, ...input });
const [intent] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
// Reconstruct an interrupted delivery with its already admitted successor.
// The target has no direct chat identity: only the delivered ledger can
// authorize inheriting the source run's conversation.
await db.update(statusDecisionEffects).set({ deliveryState: "pending", deliveredAt: null })
.where(eq(statusDecisionEffects.decisionId, decision.id));
const target = await run(f, { status: "queued", contextSnapshot: {
issueId: f.issueId, source: "native_status_decision", nativeStatusWakeIntentId: intent.id,
} });
await db.update(agentWakeupRequests).set({ status: "coalesced", runId: target.id }).where(eq(agentWakeupRequests.id, intent.id));
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
reason: "issue_status_changed", requestedByActorType: "system", requestedByActorId: `native-status-wake-dispatch:${intent.id}`,
runId: target.id, status: "queued", idempotencyKey: intent.idempotencyKey, payload: { issueId: f.issueId },
});
await start(); await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${target.id}:`))).toBe(false);
await quiet();
await dispatchPendingNativeStatusEffects({ db, ...input, decisionId: decision.id });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${target.id}:queued:${f.endpointId}`);
await quiet();
await expect(dispatchPendingNativeStatusEffects({ db, ...input, decisionId: decision.id })).resolves.toEqual([]);
await quiet();
});
it("wakes an admitted continuation when native wake dispatch repairs its original receipt", async () => {
const input = await continuationDecisionFixture();
await commitNativeStatusDecision({ db, ...input });
const [intent] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
const target = await run(f, { status: "queued", contextSnapshot: {
issueId: f.issueId, source: "native_status_decision", nativeStatusWakeIntentId: intent.id,
} });
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
reason: "issue_status_changed", requestedByActorType: "system", requestedByActorId: `native-status-wake-dispatch:${intent.id}`,
runId: target.id, status: "queued", idempotencyKey: intent.idempotencyKey, payload: { issueId: f.issueId },
});
await start(); await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${target.id}:`))).toBe(false);
await quiet();
const { service, dependencies } = await queue();
await expect(service.dispatchPendingNativeStatusWakeups({ companyId: f.companyId })).resolves.toMatchObject({ recovered: 1, dispatched: 0 });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${target.id}:queued:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled();
await quiet();
await expect(service.dispatchPendingNativeStatusWakeups({ companyId: f.companyId })).resolves.toMatchObject({ scanned: 0 });
await quiet();
});
it("rechecks a fresh native continuation after admission finishes its wake receipt", async () => {
const input = await continuationDecisionFixture();
await commitNativeStatusDecision({ db, ...input });
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, input.runId));
await start(); await flush(); await quiet();
const { service, dependencies } = await queue();
let admittedId: string | undefined;
dependencies.getSchedulingSuppression.mockImplementation(async () => {
const rows = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
const admitted = rows.find(row => row.id !== input.runId);
if (admitted && !admittedId) {
admittedId = admitted.id;
// Admission already committed and signalled, but the original wake
// receipt is still claimed. Let that first projection finish now.
await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${admittedId}:`))).toBe(false);
}
return { suppressed: true, reason: "task_drain" };
});
await expect(service.dispatchPendingNativeStatusWakeups({ companyId: f.companyId })).resolves.toMatchObject({ dispatched: 1 });
expect(admittedId).toBeDefined();
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${admittedId}:queued:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled();
await quiet();
});
it("projects a continuation only after the real question delivery receipt commits", async () => {
const source = await run(f, { status: "succeeded" });
const interactions = issueThreadInteractionService(db);
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
options: [{ id: "yes", label: "Yes" }],
}] },
}, { agentId: f.agentId, runId: source.id });
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
// A provider acknowledgement is durable input to this projection test.
await db.insert(chatPublications).values({ companyId: f.companyId, endpointId: f.endpointId, conversationId: f.conversationId,
issueId: f.issueId, idempotencyKey: `interaction:${interaction.id}:${f.endpointId}`,
payload: { text: "Continue?", interactionId: interaction.id }, state: "published", providerMessageId: "question-1",
}).onConflictDoNothing();
await db.update(chatPublications).set({ state: "published", providerMessageId: "question-1" }).where(eq(chatPublications.issueId, f.issueId));
await start(); await flush(); await quiet();
let targetId!: string;
const delivery = questionResponseDeliveryService(db, { heartbeat: {
cancelRun: async () => { throw new Error("The source is already terminal"); },
wakeup: async (_agentId, options) => {
const values = runValues(f, { status: "queued", contextSnapshot: options.contextSnapshot });
const [target] = await db.insert(heartbeatRuns).values(values).returning();
targetId = target.id;
await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${target.id}:`))).toBe(false);
return target;
},
} });
await expect(delivery.deliver(interaction.id)).resolves.toMatchObject({ status: "fallback_queued" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${targetId}:queued:${f.endpointId}`);
await quiet();
});
it.each([
["question", "answer"], ["question", "skip"], ["question", "cancel"],
["question", "withdraw"], ["question", "expire"], ["question", "historical answer"],
["confirmation", "accept"], ["confirmation", "reject"], ["confirmation", "skip"],
["confirmation", "withdraw"], ["confirmation", "expire"],
["question", "comment"], ["question", "historical comment"],
["confirmation", "comment"], ["confirmation", "historical comment"],
["question", "replacement"], ["confirmation", "replacement"], ["confirmation", "startup supersession"],
["question", "addressee deletion"], ["confirmation", "addressee deletion"],
] as const)("rechecks an unsent %s after %s", async (kind, action) => {
const source = await run(f, { status: "failed", errorCode: "test_run_failed" });
const interactions = issueThreadInteractionService(db);
const issue = { id: f.issueId, companyId: f.companyId, status: "in_progress" };
const addressee = action === "addressee deletion"
? (await db.insert(agents).values({ companyId: f.companyId, name: "Reviewer", adapterType: "paperclip_runner" }).returning())[0] : null;
const createInteraction = (sourceRunId: string, supersedePendingSiblingInteractions = true) => interactions.create(issue, kind === "confirmation" ? {
kind: "request_confirmation", continuationPolicy: "wake_assignee", sourceRunId, addresseeAgentId: addressee?.id,
title: "Confirm", payload: { version: 1, prompt: "Continue?", supersedeOnUserComment: true },
} : {
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId, addresseeAgentId: addressee?.id,
title: "Choose", payload: { version: 1, title: "Choose", supersedeOnUserComment: true, questions: [{
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
options: [{ id: "yes", label: "Yes" }],
}] },
}, { agentId: f.agentId, runId: sourceRunId }, { supersedePendingSiblingInteractions });
const interaction = await createInteraction(source.id);
const replacement = action === "replacement" || action === "startup supersession" ? await run() : null;
if (action === "startup supersession") {
await createInteraction(replacement!.id, false);
await db.update(issueThreadInteractions).set({ createdAt: new Date(Date.now() - 1000) }).where(eq(issueThreadInteractions.id, interaction.id));
}
if (action === "historical answer") {
await interactions.dismissInteraction(issue, interaction.id, { userId: f.userId });
await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId));
}
await start(); await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${source.id}:`))).toBe(false);
await quiet();
if (action === "addressee deletion") {
expect((await interactions.cancelPendingForDeletedAddressee(f.companyId, addressee!.id)).map(row => row.id)).toContain(interaction.id);
} else if (action === "startup supersession") {
expect(await interactions.sweepSupersededPendingRequestConfirmations()).toMatchObject({ expired: 1 });
} else if (action === "replacement") {
await createInteraction(replacement!.id);
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0].status).toBe("expired");
} else if (action === "comment" || action === "historical comment") {
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
authorType: "user", authorUserId: f.userId, origin: "comment", body: "Ignore the earlier request" }).returning();
const expired = action === "comment"
? await interactions.expireRequestConfirmationsSupersededByComment(issue, comment, { userId: f.userId })
: await interactions.expireRequestConfirmationsSupersededByHistoricalComments(issue);
expect(expired.map(row => row.id)).toContain(interaction.id);
} else if (action === "accept") {
await interactions.acceptInteraction({ ...issue, projectId: null, goalId: null }, interaction.id, {}, { userId: f.userId });
} else if (action === "reject") {
await interactions.rejectInteraction(issue, interaction.id, { reason: "No longer needed" }, { userId: f.userId });
} else if (action === "skip") {
await interactions.skipInteraction(issue, interaction.id, { reason: "No longer needed" }, { userId: f.userId });
} else if (action === "cancel") {
await interactions.cancelQuestions(issue, interaction.id, { reason: "No longer needed" }, { userId: f.userId });
} else if (action === "withdraw") {
await interactions.withdrawInteraction(issue, interaction.id, { reason: "No longer needed" }, { userId: f.userId });
} else if (action === "expire") {
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId));
await interactions.expirePendingInteractionsForTerminalIssue({ ...issue, status: "cancelled" }, { userId: f.userId });
} else {
await interactions.answerQuestions(issue, interaction.id,
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
}
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${source.id}:failed:${f.endpointId}`);
if (action === "historical answer") {
expect(await db.select().from(issueQuestionResponseDeliveries).where(eq(issueQuestionResponseDeliveries.interactionId, interaction.id))).toEqual([]);
}
await quiet();
});
it.each(["change", "delete", "accept", "reject"] as const)("rechecks a document confirmation after %s detects a stale target", async mode => {
const source = await run(f, { status: "failed", errorCode: "test_run_failed" });
const interactions = issueThreadInteractionService(db), docs = documentService(db);
const issue = { id: f.issueId, companyId: f.companyId, status: "in_progress", projectId: null, goalId: null };
const input = { issueId: f.issueId, key: "plan", format: "markdown", body: "First plan", createdByUserId: f.userId };
const { document } = await docs.upsertIssueDocument(input);
const interaction = await interactions.create(issue, {
kind: "request_confirmation", continuationPolicy: "wake_assignee", sourceRunId: source.id,
payload: { version: 1, prompt: "Apply this plan?", target: { type: "issue_document", issueId: f.issueId,
documentId: document.id, key: document.key, revisionId: document.latestRevisionId!, revisionNumber: document.latestRevisionNumber } },
}, { agentId: f.agentId, runId: source.id });
await start(); await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${source.id}:`))).toBe(false);
await quiet();
const changed = mode === "delete"
? { ...(await docs.deleteIssueDocument(f.issueId, document.key))!, latestRevisionId: null, latestRevisionNumber: null }
: (await docs.upsertIssueDocument({ ...input, body: "Revised plan", baseRevisionId: document.latestRevisionId })).document;
await flush(); await quiet();
if (mode === "accept" || mode === "reject") {
await expect(mode === "accept"
? interactions.acceptInteraction(issue, interaction.id, {}, { userId: f.userId })
: interactions.rejectInteraction(issue, interaction.id, {}, { userId: f.userId }))
.rejects.toMatchObject({ status: 409, details: { code: "interaction_stale_target" } });
} else {
expect((await interactions.expireStaleRequestConfirmationsForIssueDocument(issue, changed, { userId: f.userId })).map(row => row.id)).toContain(interaction.id);
}
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0].status).toBe("expired");
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${source.id}:failed:${f.endpointId}`);
await quiet();
});
it("wakes a question continuation after its in-flight prompt receipt commits", async () => {
await db.update(chatEndpoints).set({ provider: "discord" }).where(eq(chatEndpoints.id, f.endpointId));
const source = await run(f, { status: "failed", contextSnapshot: { ...runValues().contextSnapshot, source: "chat:discord" } });
const interactions = issueThreadInteractionService(db);
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
options: [{ id: "yes", label: "Yes" }],
}] },
}, { agentId: f.agentId, runId: source.id });
const prompt = (await publications()).find(row => row.idempotencyKey === `interaction:${interaction.id}:${f.endpointId}`)!;
const [endpoint] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, f.endpointId));
const [conversation] = await db.select().from(chatConversations).where(eq(chatConversations.id, f.conversationId));
const targetId = randomUUID();
let beforeReceiptKeys: string[] | undefined;
await start(); await flush(); await quiet();
const deliver = questionResponseDeliveryService(db, { heartbeat: {
wakeup: async () => {
const [target] = await db.insert(heartbeatRuns).values(runValues(f, {
id: targetId, status: "queued", contextSnapshot: { issueId: f.issueId },
})).returning();
return target;
},
} });
const post = vi.fn(async () => {
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
await deliver.deliver(interaction.id);
await flush();
beforeReceiptKeys = (await publications()).map(row => row.idempotencyKey);
return { id: "published-question" };
});
// Controlled transport/authorization ports; actual attempt lease, receipt
// transaction, question delivery receipt and milestone worker use PostgreSQL.
const worker = createChatPublicationWorker(db, {
CAPABILITIES, settleGitHubAutomaticPublication: async () => false,
ensureDurablePublicationTransport: async publication => publication,
tryProcessTeamsFilePublication: async () => false,
endpointRecord: async () => ({ endpoint, assignedAgentName: "Maya", connectionName: "Chat", applicationId: "test", credentialSecretRefs: [] }),
runtimeContextForRecord: () => ({ generation: 0, localEpoch: 0, credentialFingerprint: "test", version: "test" }),
hasCommittedTaskControlCompletion: async () => false, nonDirectDestinationAllowed: () => true,
runOwnershipMilestoneSupersessionReason: async () => null,
progressSupersededByFastResponse: async () => false, runProgressSupersededByPublishedInteraction: async () => false,
currentTaskControlPayload: async publication => publication.payload,
withCredentialMutationLease: async (_endpoint, operation) => operation({ assertOwned: async () => {} }),
closeProgressToReplace: async () => null, interactionResolutionPublicationToReplace: async () => null,
interactionPromptPublicationToReplace: async () => null, runPublicationToReplace: async () => null,
inboundWakePublicationToReplace: async () => null, taskStatusPublicationToReplace: async () => null,
claimPublicationTransportAuthorization: async () => ({ endpoint, conversation, authorizationActionId: null, preparedText: null }),
postSafePublication: post, commitTaskControlCompletion: async () => {}, receiptReactionCompletionRunId: async () => null,
finalizePublicationFailure: async (_publication, error) => { errors.push(error); return false; },
redactError: String,
} as Parameters<typeof createChatPublicationWorker>[1]);
await worker.processSelectedPublication(prompt);
expect(post).toHaveBeenCalledTimes(1);
expect((await db.select().from(chatPublications).where(eq(chatPublications.id, prompt.id)))[0]).toMatchObject({ state: "published", providerMessageId: "published-question" });
expect(beforeReceiptKeys).toBeDefined();
expect(beforeReceiptKeys).not.toContain(`run:${targetId}:queued:${f.endpointId}`);
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${targetId}:queued:${f.endpointId}`);
await quiet();
});
it.each([
["retry", null], ["failed", null], ["delivery_unknown", "cancel"], ["delivery_unknown", "mark_delivered"],
] as const)("rechecks resolved prompt ownership after delivery %s (%s)", async (state, resolution) => {
const source = await run(f, { status: "failed", errorCode: "test_run_failed" });
const interactions = issueThreadInteractionService(db);
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
options: [{ id: "yes", label: "Yes" }],
}] },
}, { agentId: f.agentId, runId: source.id });
// Model a send already claimed when the user answers through the UI.
const [prompt] = await db.update(chatPublications).set({ state: "streaming", attempts: 1 })
.where(eq(chatPublications.idempotencyKey, `interaction:${interaction.id}:${f.endpointId}`)).returning();
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
expect(prompt).toBeDefined();
await start(); await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${source.id}:`))).toBe(false);
await quiet();
const failures = createChatPublicationFailures(db, {
FailedChatRetryPublicationReadError: class extends Error {},
MAX_ERROR_TEXT: 1000, getTaskBaseUrl: () => null,
invalidateRuntime: async () => false, runtimeCallbackEndpoint: async () => null,
});
const failure = Object.assign(new Error("provider rejected the send"), { status: state === "retry" ? 429 : 400 });
// The finalizer receives the selection from before the attempt increment.
await expect(failures.finalizePublicationFailure({ ...prompt, attempts: 0 }, failure, state === "delivery_unknown")).resolves.toBe(true);
expect((await db.select().from(chatPublications).where(eq(chatPublications.id, prompt.id)))[0].state).toBe(state);
await flush();
expect((await publications()).some(row => row.idempotencyKey === `run:${source.id}:failed:${f.endpointId}`)).toBe(state === "failed");
await quiet();
if (resolution) {
const targetId = randomUUID();
await questionResponseDeliveryService(db, { heartbeat: { wakeup: async () => {
const [target] = await db.insert(heartbeatRuns).values(runValues(f, {
id: targetId, status: "queued", contextSnapshot: { issueId: f.issueId },
})).returning();
return target;
} } }).deliver(interaction.id);
await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${targetId}:`))).toBe(false);
await quiet();
const [endpoint] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, f.endpointId));
const resolver = createChatPublicationResolution(db, {
endpointRecord: async () => ({ endpoint, assignedAgentName: "Maya", connectionName: "Chat", applicationId: "test", credentialSecretRefs: [] }),
withCredentialMutationLease: async (_endpoint, operation) => operation({ assertOwned: async () => {} }),
commitTaskControlCompletion: async () => {}, processPendingPublications: async () => {},
} as Parameters<typeof createChatPublicationResolution>[1]);
await resolver.resolvePublication(f.endpointId, prompt.id, resolution, f.userId);
await flush();
const keys = (await publications()).map(row => row.idempotencyKey);
expect(keys.includes(`run:${source.id}:failed:${f.endpointId}`)).toBe(resolution === "cancel");
expect(keys.includes(`run:${targetId}:queued:${f.endpointId}`)).toBe(resolution === "mark_delivered");
await quiet();
}
});
it.each(["retry", "stale claim", "live claim", "rolled-back reset"] as const)("rechecks question prompt binding after %s", async (mode) => {
const source = await run();
const interactions = issueThreadInteractionService(db);
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
options: [{ id: "yes", label: "Yes" }],
}] },
}, { agentId: f.agentId, runId: source.id });
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
const other = await fixture(f.companyId);
await db.update(chatEndpoints).set({ assignedAgentId: f.agentId }).where(eq(chatEndpoints.id, other.endpointId));
await db.update(chatConversations).set({ issueId: f.issueId }).where(eq(chatConversations.id, other.conversationId));
await db.insert(chatPublications).values({ companyId: f.companyId, endpointId: other.endpointId, conversationId: other.conversationId,
issueId: f.issueId, idempotencyKey: `interaction:${interaction.id}:${other.endpointId}`,
payload: { text: "Continue?", interactionId: interaction.id }, state: "published", providerMessageId: "other-question",
});
const events: string[] = [];
let claimEvents: string[] = [];
let duringClaimNotices: string[] | undefined;
const unsubscribe = subscribeDatabaseWork(db, "chat-run-milestones", event => events.push(event));
const delivery = questionResponseDeliveryService(db, { heartbeat: {
cancelRun: async () => {
claimEvents = [...events];
await start(); await flush();
duringClaimNotices = (await publications()).map(row => row.idempotencyKey);
throw new Error("temporary cancellation failure");
},
wakeup: async () => { throw new Error("Must retain the still-active source"); },
} });
try {
if (mode === "retry") {
await expect(delivery.deliver(interaction.id)).resolves.toBeNull();
} else {
await db.update(issueQuestionResponseDeliveries).set({ status: "delivering", attemptCount: 1,
lastAttemptAt: new Date(Date.now() - (mode === "live claim" ? 0 : 60_000)),
}).where(eq(issueQuestionResponseDeliveries.interactionId, interaction.id));
await start(); await flush();
duringClaimNotices = (await publications()).map(row => row.idempotencyKey);
if (mode === "rolled-back reset") {
await expect(db.transaction(async tx => {
await questionResponseDeliveryService(tx as unknown as typeof db, { heartbeat: {} as never }).sweepPending(0);
expect(events).toEqual(["intent"]);
throw new Error("rollback reset");
})).rejects.toThrow("rollback reset");
} else {
await delivery.sweepPending(0);
}
}
expect(duringClaimNotices).toBeDefined();
expect(duringClaimNotices!.some(key => key.startsWith(`run:${source.id}:`))).toBe(false);
expect((await db.select().from(issueQuestionResponseDeliveries).where(eq(issueQuestionResponseDeliveries.interactionId, interaction.id)))[0].status).toBe(mode === "live claim" || mode === "rolled-back reset" ? "delivering" : "pending");
await flush();
const noticeExists = (await publications()).some(row => row.idempotencyKey === `run:${source.id}:working:${f.endpointId}`);
expect(noticeExists).toBe(mode === "retry" || mode === "stale claim");
if (mode === "retry") expect(claimEvents).toEqual(["intent", "settled"]);
else if (mode === "live claim") expect(events).toEqual([]);
else {
expect(events[0]).toBe("intent");
expect(events.slice(1)).toContain("settled");
expect(events.filter(event => event === "intent")).toHaveLength(1);
}
await quiet();
} finally { unsubscribe(); }
});
it("waits for outer commit and ignores rolled-back nested writes", async () => {
await start(); await flush();
await expect(db.transaction(async tx => {
await tx.transaction(async nested => { await notifyChatRunMilestones(nested, f); await nested.insert(heartbeatRuns).values(runValues()); });
throw new Error("rollback fixture");
})).rejects.toThrow("rollback fixture");
await flush(); expect(await publications()).toEqual([]); await quiet();
let release!: () => void, staged!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const stagedPromise = new Promise<void>(resolve => { staged = resolve; });
const writing = db.transaction(async tx => {
await tx.transaction(async nested => { await notifyChatRunMilestones(nested, f); await nested.insert(heartbeatRuns).values(runValues()); });
staged(); await gate;
});
try { await stagedPromise; await vi.advanceTimersByTimeAsync(1000); expect(await publications()).toEqual([]); }
finally { release(); await writing; }
await flush(); expect(await publications()).toHaveLength(1); await quiet();
});
it("retains the first committed wake during drain without periodic SQL", async () => {
await start(); await flush(); enabled = false;
await run(); await vi.advanceTimersByTimeAsync(1000);
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
const drained = idleWorkSnapshot();
expect(await publications()).toEqual([]); await quiet(); expect(idleWorkSnapshot()).toEqual(drained);
enabled = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
expect(await publications()).toHaveLength(1); await quiet();
});
it("scopes mutation wakes to their issue and company", async () => {
const sibling = await fixture(f.companyId), foreign = await fixture();
await start(); await flush();
// Unsupported raw writers deliberately do not wake: they provide a sentinel
// proving a supported issue mutation does not perform a global scan.
await db.insert(heartbeatRuns).values([runValues(sibling), runValues(foreign)]);
await run(); await flush();
expect(await publications()).toHaveLength(1); expect(await publications(sibling)).toEqual([]); expect(await publications(foreign)).toEqual([]);
await db.transaction(tx => notifyChatRunMilestones(tx, { companyId: f.companyId }));
await flush(); expect(await publications(sibling)).toHaveLength(1); expect(await publications(foreign)).toEqual([]);
await quiet();
});
it("disarms suppressed lineage and reactivates after its binding changes", async () => {
const row = await run(f, { contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: randomUUID(), wakeCommentId: f.commentId } });
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, f);
await tx.update(heartbeatRuns).set({ contextSnapshot: runValues().contextSnapshot }).where(eq(heartbeatRuns.id, row.id));
});
await flush(); expect(await publications()).toHaveLength(1); await quiet();
});
it("waits quietly for a late presentation decision", async () => {
const row = await run(f, { status: "succeeded" }); await start(); await flush();
expect(await publications()).toEqual([]); await quiet();
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, f);
await tx.update(heartbeatRuns).set({ resultJson: { presentationDecision: { source: "none" } } }).where(eq(heartbeatRuns.id, row.id));
});
await flush(); expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${row.id}:completed:${f.endpointId}` })]); await quiet();
});
it("drains more than one batch without duplicate notices or starving another company", async () => {
const other = await fixture();
await db.insert(heartbeatRuns).values([...Array.from({ length: 205 }, () => runValues()), runValues(other)]);
await start(); await flush(); expect(await publications()).toHaveLength(205); expect(await publications(other)).toHaveLength(1); await quiet();
});
it("retains a failed selection for retry through drain", async () => {
await run();
const select = vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("temporary read failure"); });
await start(); select.mockRestore(); expect(errors.splice(0)).toHaveLength(1);
enabled = false; await vi.advanceTimersByTimeAsync(30_000);
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
const drained = idleWorkSnapshot();
await quiet(); expect(idleWorkSnapshot()).toEqual(drained);
enabled = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
expect(await publications()).toHaveLength(1); await quiet();
});
it("retries a failed publication write without idle scans before its deadline", async () => {
await run();
const transaction = vi.spyOn(db, "transaction").mockRejectedValueOnce(new Error("temporary publication failure"));
await start(); transaction.mockRestore(); expect(errors.splice(0)).toHaveLength(1);
expect(await publications()).toEqual([]); await quiet(10_000);
await vi.advanceTimersByTimeAsync(30_000); await flush(); expect(await publications()).toHaveLength(1); await quiet();
});
it("preserves a committed change arriving while a projection batch is running", async () => {
await run();
const transaction = db.transaction.bind(db);
let release!: () => void, entered!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const staged = new Promise<void>(resolve => { entered = resolve; });
const held = vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => {
entered(); await gate; return transaction(callback, config);
});
const starting = start();
try {
await staged;
await transaction(async tx => { await notifyChatRunMilestones(tx, f); await tx.insert(heartbeatRuns).values(runValues()); });
} finally { release(); await starting; held.mockRestore(); }
await flush(); expect(await publications()).toHaveLength(2); await quiet();
});
it("stops future wakes and reconstructs writes made after shutdown", async () => {
await start(); await flush(); await coordinator!.stop(); await run(); await quiet();
expect(await publications()).toEqual([]); await start(); await flush(); expect(await publications()).toHaveLength(1);
});
});