mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 20:21:50 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People also ask agents for work in their existing chat tools. > - Each external conversation needs one task and a current authorized source. > - Retries, Stop, and provider failures must not duplicate work or expose private data. > - The first chat PR establishes the opt-in provider and data contracts. > - This PR adds experimental channel integration and its durable control plane. > - Users can request work from connected channels and inspect delivery in Paperclip. ## Linked Issues or Issue Description Refs #13100 and #13092. This is the second of exactly two chat PRs. Foundation #13100 is merged and changed 143 files. Runner prerequisite #13092 is also merged. This PR changes 400 files against master, below the 500-file review limit. It contains no wireframe images or HTML galleries. ## What Changed - Add native Slack, GitHub, Microsoft Teams, Telegram, and Discord chat connections. Keep chat disabled unless the operator enables experimental chat connectors. Preserve the production GitHub tool connection and its normal setup path. - Bind each provider bot identity to one immutable Paperclip agent. Bind each admitted external conversation to one task. Paperclip owns tasks, runs, permissions, and audit records. - Add durable admission, per-conversation queues, questions, task controls, progress, final replies, images, files, and delivery receipts. Board comments remain internal unless explicitly sent to the channel. - Check current identity, provider reach, resource access, credentials, runtime generation, and exact source before provider effects. Keep private responses private. Never send raw reasoning, private logs, credentials, or tool arguments. - Hold uncertain sends for explicit audited resolution. Make Board Send-to-channel atomic and idempotent. Keep reconnect and setup credentials in Paperclip secret storage. - Preserve current native-runner authority across retries, lost acknowledgements, and recovery. Keep immutable input and completion contracts separate from newer user input. Receipt reconciliation cannot launch a provider. - Reconcile chat close/new ordering and provider-effect lock order. Audit resource access changes in the same transaction. Submit only the selected resource from each UI toggle so stale pages cannot undo unrelated access changes. - Drain Codex stdout before certifying process exit. Bound the drain with the existing shutdown grace. Preserve observed terminal authority without treating an undrained process as successful or reusable. - Incorporate master `018ca5da` with its ACP Stop, mobile task layout, runner packaging, and official lock changes. Preserve dedicated chat-answer continuations in both directions when ordinary queued comments are adopted after Stop. - Fence late adapter readiness behind an earlier Stop for the same run. Preserve verified cleanup for registered adapters. Handle single Stop, agent pause, duplicate Stops, and failure release without creating a false cancellation receipt. - Incorporate master's `6dd48cad4` wake-queue extraction. Preserve exact failed-chat retry authorization and lineage, retired question-source suppression, and the block on generic recovery that would discard the admitted source. Fresh deferred input retains its separate promotion path. - Incorporate master `2a05b5ed3` and its queue-admission extraction, simplified transaction ports, and separate runner CI job. Preserve exact durable receipts, actor separation, and dedicated-answer isolation through the new module. A failed receipt insert rolls back the accompanying deferred-wake merge. ## Verification Current head: `afe19299d06253cb628eb398e91d1200ea9f412a`, incorporating master `2a05b5ed3457ea33efd6895520447d1d97fe98d8`. The conflicts are resolved. This successor fixes two test-harness boundaries exposed by CI: per-case route-module preparation and actual durable-save completion before intentional runner termination. Production code and all existing test/turn deadlines are unchanged. [Exact-head Greptile review](https://github.com/paperclipai/paperclip/pull/13038#issuecomment-5587250594) is **5/5**, completed September 10 at 13:20:55 UTC, with no actionable findings or open review threads. [Fresh exact-head CI](https://github.com/paperclipai/paperclip/actions/runs/34481724341) passes **all 24 jobs**, including Build and both required aggregates. Normal exact-head guarded merge was attempted and rejected by the remaining branch approval policy: CODEOWNER review is required and no human approval is present. Normal **squash auto-merge is enabled** as of September 10 at 13:36:26 UTC. Requested CODEOWNERS have been notified; no approval bypass or self-approval was used. Earlier-head results below remain historical evidence, not qualification of this successor. - Final exact-head Linux evidence: 995/995 chat integration cases; 36/36 agent-skills routes; 35/35 runner live-session cases, including real process kill/resume; 1948 runner Vitest cases with three existing benchmark/platform guards; 870/870 API-authority cases; and 104 browser cases with four existing optional skips. Rust, conformance/replay, full repository build, typecheck, canary, all server/workspace shards, and both required aggregates pass with normal CI concurrency. Earlier failed attempts remain recorded below. - Latest test-only qualification: 141/141 route/permissions/authentication cases pass in separate cold forks, with plain server types and independent review clear. The real-runner suite passes 35/35, with plain runner types and independent review clear. A controlled premature-save acknowledgement fails as expected; matching ownership/effect/process evidence, rejected saves, real turn outcome, test abort, and pre-kill liveness are covered. No local reproduction of the original CI scheduling failure is claimed. The preceding [CI run](https://github.com/paperclipai/paperclip/actions/runs/34479680858) passes 21/24 jobs, including all 995 Linux chat cases and browser aggregate (104 passed, four existing optional skips); only Build, the skills serialized shard, and the required verification aggregate fail. Its exact-head Greptile review was 5/5. Both failed job logs are retained. - Final fixture qualification: all eight focused Discord cases and all 995 chat integration cases pass. The exact modal statement/PID is observed before taking the real connection lock; the test then proves its actual blocking relationship before mutation. Original SQL execution, provider behavior, negative assertions, and 1s/15s timeouts remain unchanged. Independent review is clear and test/production hashes remain frozen. The preceding [CI attempt](https://github.com/paperclipai/paperclip/actions/runs/34477184777) passed 22 jobs, including Build/runner, typecheck, canary, all other test shards, and browser aggregate (104 passed, four existing optional skips); the two fixture failures and failed verification aggregate remain recorded, not relabeled as a pass. - Current queue-module composition: 308/308 recovery/batching/queue/Stop tests; 995/995 full chat integration; 89/89 module tests, including real PostgreSQL receipt-insert rollback; 24/24 workflow/module-boundary tests; plain server and UI types. All four actual local process/ACP browser paths pass in 1.4 minutes. Fresh databases, no skips or retries, stable reviewed source hashes. The initial boundary failure is retained; its no-op service wrapper was removed without changing recovery context or weakening the check. An exploratory standalone test-directory typecheck fails because its new upstream transformation config is not a standalone typechecking project; standard CI/build does not invoke it, and no configuration was weakened to suppress those diagnostics. - The preceding head `e02a63d462ce5d47433b0aeb632bb6fd20aab1ba` passed [all 24 CI jobs](https://github.com/paperclipai/paperclip/actions/runs/34436462958) and exact-head Greptile review at 5/5. Required CODEOWNER review prevented its normal merge before master advanced again. - Final extracted-module composition: 307/307 recovery, batching, queue and Stop-control tests; 995/995 full chat integration; 49/49 module tests including eight PostgreSQL adapter cases; and 19/19 issue-update tests. Plain server types pass. All four actual local process/ACP browser paths pass in 1.3 minutes. Fresh databases, no skips or retries in these cohorts, frozen source hashes, and independent review clear. - The preceding head `3e4e1c1c` passes [all PR CI jobs](https://github.com/paperclipai/paperclip/actions/runs/34415826820), including Build and required `ci / verify` and `ci / e2e`. Both the original Rust failure and the previously load-sensitive lineage fixture pass with unchanged Linux concurrency. Master advanced afterward and required this reconciliation. - Final master composition: 448/448 focused UI tests, 186/186 adapter tests, 24/24 queue/control tests, and 11/11 packaging tests. Plain UI, server, shared, and adapter types pass. Token gates and diff checks pass. Independent server and UI reviews are clear. - Stop-registration regression: both real-service cases fail against exact `a95` source and pass with the fix. The full corrected recovery/control suite passes 265/265. Duplicate-owner and failed-Stop controls also pass. Plain server types pass. The readiness barrier prevents provider startup without adding an acknowledgment to an already terminal run. - Final qualification strengthens terminal-field equality and repeats both affected cases successfully on a fresh database. All four actual local process/ACP browser paths pass again in 1.3 minutes, without skips or retries. The final screenshot shows Cancelled, a paused subtree, retained input, and no error toast. - Two new actual-service regressions fail before the merge fix. They prove that queued-comment adoption could consume a dedicated chat answer or add unrelated input to that answer. The fixed four-case cohort passes, including ordinary upstream continuation and adapter Stop controls. Full recovery passes 257/257. All four actual local process/ACP Stop browser flows pass in 1.4 minutes, without skips or retries, on a fresh database. - The unchanged runner artifact was qualified with 171/171 transport tests, 870/870 API-authority tests, conformance 1/1, and replay 11/11. Six controlled reader tests prove the exit/drain repair. Its local serial Rust workspace passed 546 top-level cases plus two invoked helpers; the later passing Linux CI supplies default-concurrency evidence. - Prior exact-source full chat integration passes 995/995. Settings regressions cover concurrent stale pages, 501 destinations, pending state, rejected updates, and explicit retry. These deterministic tests do not prove live provider behavior. - Retained failed attempts and their causes are in the [qualification log](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-chat-queue-and-webhook-repair.md). The first merge adapter run timed out while macOS slept for 290 seconds. Its unchanged repeat passed with a temporary sleep guard. No assertion, deadline, or CI gate was weakened. Review commands include `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-process-recovery.test.ts src/__tests__/issue-queued-comments-routes.test.ts` and `pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/acp-stop-continuation.spec.ts`. Database suites require fresh disposable databases. See the [browser runbook](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-04-chat-adapters-browser-e2e-runbook.md) for provider setup and separate live acceptance steps. ## Risks - This remains experimental. Deterministic tests and bounded live evidence do not establish every provider feature, tenant, permission layout, or media shape. Teams work-tenant qualification is still open. - Failed and uncertain provider effects remain visible and can require operator action. A transport receipt does not prove recipient visibility. - Native controller and runner artifacts must remain compatible. Preserve lease ownership, terminal authority, source binding, and quarantine during future changes. - Access and audit rows commit together, but activity notifications remain best-effort. This is not a new durable event outbox. - The PR operation does not deploy a live server, replace its runner, or change provider permissions. Remaining live qualification is documented in the [temporary handoff](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-open-qualification-followups.md). ## Model Used OpenAI Codex assisted with implementation, tool execution, testing, and review. The work records `gpt-6-astra` assistance. The environment does not report a context-window size. No private reasoning traces are included. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1304 lines
41 KiB
TypeScript
1304 lines
41 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { eq } from "drizzle-orm";
|
|
import {
|
|
afterAll,
|
|
afterEach,
|
|
beforeAll,
|
|
describe,
|
|
expect,
|
|
it,
|
|
vi,
|
|
} from "vitest";
|
|
import {
|
|
activityLog,
|
|
agentWakeupRequests,
|
|
agents,
|
|
companies,
|
|
createDb,
|
|
goals,
|
|
heartbeatRuns,
|
|
issueComments,
|
|
issueQuestionResponseDeliveries,
|
|
issueThreadInteractions,
|
|
issues,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
getEmbeddedPostgresTestSupport,
|
|
startEmbeddedPostgresTestDatabase,
|
|
} from "./helpers/embedded-postgres.js";
|
|
import { issueThreadInteractionService } from "../services/issue-thread-interactions.js";
|
|
import {
|
|
buildQuestionResponseDeliveryEnvelope,
|
|
formatQuestionResponseSteeringMessage,
|
|
questionResponseDeliveryService,
|
|
} from "../services/question-response-delivery.js";
|
|
|
|
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
|
const describeEmbeddedPostgres = embeddedPostgresSupport.supported
|
|
? describe
|
|
: describe.skip;
|
|
|
|
const DIRECT_ADAPTER_TYPES = [
|
|
"acpx_local",
|
|
"claude_local",
|
|
"codex_local",
|
|
"cursor_cloud",
|
|
"cursor",
|
|
"gemini_local",
|
|
"grok_local",
|
|
"hermes_gateway",
|
|
"hermes_local",
|
|
"kimi_local",
|
|
"openclaw_gateway",
|
|
"opencode_local",
|
|
"pi_local",
|
|
"process",
|
|
"http",
|
|
"external_test_adapter",
|
|
] as const;
|
|
|
|
describeEmbeddedPostgres("question response delivery", () => {
|
|
let db!: ReturnType<typeof createDb>;
|
|
let tempDb: Awaited<
|
|
ReturnType<typeof startEmbeddedPostgresTestDatabase>
|
|
> | null = null;
|
|
|
|
beforeAll(async () => {
|
|
tempDb = await startEmbeddedPostgresTestDatabase(
|
|
"paperclip-question-delivery-",
|
|
);
|
|
db = createDb(tempDb.connectionString);
|
|
}, 20_000);
|
|
|
|
afterEach(async () => {
|
|
await db.delete(issueQuestionResponseDeliveries);
|
|
await db.delete(issueThreadInteractions);
|
|
await db.delete(issueComments);
|
|
await db.delete(activityLog);
|
|
await db.delete(heartbeatRuns);
|
|
await db.delete(agentWakeupRequests);
|
|
await db.delete(issues);
|
|
await db.delete(goals);
|
|
await db.delete(agents);
|
|
await db.delete(companies);
|
|
});
|
|
|
|
afterAll(async () => {
|
|
await tempDb?.cleanup();
|
|
});
|
|
|
|
async function seed(
|
|
args: {
|
|
adapterType?: string;
|
|
runtimeMode?: "legacy" | "native";
|
|
sourceStatus?: string;
|
|
successorStatus?: "queued" | "running";
|
|
sourceCommentBody?: string;
|
|
attachSourceCommentToInteraction?: boolean;
|
|
} = {},
|
|
) {
|
|
const companyId = randomUUID();
|
|
const agentId = randomUUID();
|
|
const goalId = randomUUID();
|
|
const issueId = randomUUID();
|
|
const sourceRunId = randomUUID();
|
|
const successorRunId = args.successorStatus ? randomUUID() : null;
|
|
await db.insert(companies).values({
|
|
id: companyId,
|
|
name: "Question delivery",
|
|
issuePrefix: `Q${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
|
requireBoardApprovalForNewAgents: false,
|
|
});
|
|
await db.insert(agents).values({
|
|
id: agentId,
|
|
companyId,
|
|
name: "Runner",
|
|
role: "engineer",
|
|
status: "active",
|
|
adapterType: args.adapterType ?? "codex_local",
|
|
adapterConfig: {},
|
|
runtimeConfig: {},
|
|
permissions: {},
|
|
});
|
|
await db.insert(goals).values({
|
|
id: goalId,
|
|
companyId,
|
|
title: "Test",
|
|
level: "task",
|
|
status: "active",
|
|
});
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
goalId,
|
|
title: "Deliver answers",
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: sourceRunId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "manual",
|
|
status: args.sourceStatus ?? "succeeded",
|
|
runtimeMode: args.runtimeMode ?? "native",
|
|
driverKind: "codex",
|
|
contextSnapshot: { issueId },
|
|
...(args.sourceStatus === "running"
|
|
? { startedAt: new Date() }
|
|
: { finishedAt: new Date() }),
|
|
});
|
|
if (successorRunId && args.successorStatus) {
|
|
await db.insert(heartbeatRuns).values({
|
|
id: successorRunId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "manual",
|
|
status: args.successorStatus,
|
|
runtimeMode: args.runtimeMode ?? "native",
|
|
driverKind: "codex",
|
|
contextSnapshot: { issueId },
|
|
...(args.successorStatus === "running"
|
|
? { startedAt: new Date() }
|
|
: {}),
|
|
});
|
|
}
|
|
|
|
const sourceCommentId = args.sourceCommentBody ? randomUUID() : null;
|
|
if (sourceCommentId) {
|
|
await db.insert(issueComments).values({
|
|
id: sourceCommentId,
|
|
companyId,
|
|
issueId,
|
|
authorUserId: "external-user",
|
|
body: args.sourceCommentBody!,
|
|
});
|
|
}
|
|
|
|
const interactionSvc = issueThreadInteractionService(db);
|
|
const interaction = await interactionSvc.create(
|
|
{ id: issueId, companyId },
|
|
{
|
|
kind: "ask_user_questions",
|
|
continuationPolicy: "wake_assignee",
|
|
sourceCommentId:
|
|
args.attachSourceCommentToInteraction === false
|
|
? null
|
|
: sourceCommentId,
|
|
sourceRunId,
|
|
payload: {
|
|
version: 1,
|
|
title: "Server choices",
|
|
questions: [
|
|
{
|
|
id: "purpose",
|
|
prompt: "What is it for?",
|
|
selectionMode: "single",
|
|
required: true,
|
|
options: [
|
|
{ id: "custom", label: "Write an answer", freeText: true },
|
|
],
|
|
},
|
|
{
|
|
id: "runtime",
|
|
prompt: "Which runtime?",
|
|
selectionMode: "single",
|
|
required: true,
|
|
options: [
|
|
{ id: "node", label: "Node.js" },
|
|
{ id: "bun", label: "Bun" },
|
|
],
|
|
},
|
|
{
|
|
id: "features",
|
|
prompt: "Which features?",
|
|
selectionMode: "multi",
|
|
options: [
|
|
{ id: "health", label: "Health check" },
|
|
{ id: "logs", label: "Request logs" },
|
|
],
|
|
},
|
|
],
|
|
questionSet: {
|
|
schema: "paperclip.question_set.v1",
|
|
title: "Server choices",
|
|
questions: [
|
|
{
|
|
id: "purpose",
|
|
header: "Purpose",
|
|
prompt: "What is it for?",
|
|
required: true,
|
|
answerMode: "text",
|
|
},
|
|
{
|
|
id: "runtime",
|
|
header: "Runtime",
|
|
prompt: "Which runtime?",
|
|
required: true,
|
|
answerMode: "single_select",
|
|
options: [
|
|
{ id: "node", label: "Node.js" },
|
|
{ id: "bun", label: "Bun" },
|
|
],
|
|
},
|
|
{
|
|
id: "features",
|
|
header: "Features",
|
|
prompt: "Which features?",
|
|
required: false,
|
|
answerMode: "multi_select",
|
|
options: [
|
|
{ id: "health", label: "Health check" },
|
|
{ id: "logs", label: "Request logs" },
|
|
],
|
|
customAnswer: { enabled: true, label: "Other" },
|
|
},
|
|
],
|
|
},
|
|
},
|
|
},
|
|
{ agentId, runId: sourceRunId },
|
|
);
|
|
const answered = await interactionSvc.answerQuestions(
|
|
{ id: issueId, companyId, status: "in_progress" },
|
|
interaction.id,
|
|
{
|
|
answers: [
|
|
{ questionId: "purpose", optionIds: [], otherText: "Internal API" },
|
|
{ questionId: "runtime", optionIds: ["node"] },
|
|
{
|
|
questionId: "features",
|
|
optionIds: ["health", "logs"],
|
|
otherText: "Metrics",
|
|
},
|
|
],
|
|
},
|
|
{ userId: "board-user" },
|
|
);
|
|
return {
|
|
companyId,
|
|
agentId,
|
|
issueId,
|
|
sourceRunId,
|
|
sourceCommentId,
|
|
successorRunId,
|
|
interaction: answered,
|
|
};
|
|
}
|
|
|
|
it("persists the receipt atomically and steers exactly once into a running successor", async () => {
|
|
const seeded = await seed({ successorStatus: "running" });
|
|
const newerRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: newerRunId,
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
invocationSource: "manual",
|
|
status: "running",
|
|
runtimeMode: "native",
|
|
driverKind: "codex",
|
|
contextSnapshot: { issueId: seeded.issueId },
|
|
startedAt: new Date(),
|
|
});
|
|
await db
|
|
.update(issues)
|
|
.set({ executionRunId: seeded.successorRunId })
|
|
.where(eq(issues.id, seeded.issueId));
|
|
const persistedBeforeDelivery = await db
|
|
.select()
|
|
.from(issueQuestionResponseDeliveries)
|
|
.where(
|
|
eq(
|
|
issueQuestionResponseDeliveries.interactionId,
|
|
seeded.interaction.id,
|
|
),
|
|
)
|
|
.then((rows) => rows[0]);
|
|
expect(persistedBeforeDelivery).toMatchObject({
|
|
status: "pending",
|
|
correlationId: `question-response:${seeded.interaction.id}`,
|
|
sourceRunId: seeded.sourceRunId,
|
|
});
|
|
|
|
const steer = vi.fn().mockResolvedValue({ turnId: "turn-successor" });
|
|
const wakeup = vi.fn();
|
|
const service = questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
steer,
|
|
});
|
|
const first = await service.deliver(seeded.interaction.id);
|
|
const second = await service.deliver(seeded.interaction.id);
|
|
|
|
expect(first).toMatchObject({
|
|
status: "delivered",
|
|
mode: "steered",
|
|
targetRunId: seeded.successorRunId,
|
|
targetTurnId: "turn-successor",
|
|
duplicate: false,
|
|
});
|
|
expect(second).toMatchObject({ mode: "steered", duplicate: true });
|
|
expect(steer).toHaveBeenCalledTimes(1);
|
|
expect(steer).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
runId: seeded.successorRunId,
|
|
correlationId: `question-response:${seeded.interaction.id}`,
|
|
message: expect.stringContaining("- Runtime — Which runtime?: Node.js"),
|
|
}),
|
|
);
|
|
expect(wakeup).not.toHaveBeenCalled();
|
|
|
|
const [delivery] = await db.select().from(issueQuestionResponseDeliveries);
|
|
expect(delivery).toMatchObject({
|
|
status: "delivered",
|
|
deliveryMode: "steered",
|
|
targetRunId: seeded.successorRunId,
|
|
targetTurnId: "turn-successor",
|
|
attemptCount: 1,
|
|
});
|
|
const deliveryEvents = await db
|
|
.select()
|
|
.from(activityLog)
|
|
.where(eq(activityLog.action, "issue.question_response_delivered"));
|
|
expect(deliveryEvents).toHaveLength(1);
|
|
expect(JSON.stringify(deliveryEvents[0]?.details)).not.toContain(
|
|
"Internal API",
|
|
);
|
|
expect(JSON.stringify(deliveryEvents[0]?.details)).not.toContain("Node.js");
|
|
});
|
|
|
|
it.each(["native", "legacy"] as const)(
|
|
"never mixes an external-chat question response into another live chat run in %s mode",
|
|
async (runtimeMode) => {
|
|
const seeded = await seed({
|
|
runtimeMode,
|
|
sourceStatus: "running",
|
|
successorStatus: "running",
|
|
sourceCommentBody:
|
|
"External request whose complete trailing instructions must survive the response continuation.",
|
|
attachSourceCommentToInteraction: false,
|
|
});
|
|
if (!seeded.sourceCommentId) {
|
|
throw new Error("Expected a source comment for the external-chat run");
|
|
}
|
|
expect(seeded.interaction.sourceCommentId).toBeNull();
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
contextSnapshot: {
|
|
issueId: seeded.issueId,
|
|
source: "chat:slack",
|
|
commentId: seeded.sourceCommentId,
|
|
wakeCommentId: seeded.sourceCommentId,
|
|
wakeCommentIds: [seeded.sourceCommentId],
|
|
},
|
|
})
|
|
.where(eq(heartbeatRuns.id, seeded.sourceRunId));
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
contextSnapshot: {
|
|
issueId: seeded.issueId,
|
|
source: "chat:telegram",
|
|
wakeCommentId: randomUUID(),
|
|
},
|
|
})
|
|
.where(eq(heartbeatRuns.id, seeded.successorRunId!));
|
|
await db
|
|
.update(issues)
|
|
.set({ executionRunId: seeded.successorRunId })
|
|
.where(eq(issues.id, seeded.issueId));
|
|
|
|
const [dedicatedRun] = await db
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
id: randomUUID(),
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
invocationSource: "automation",
|
|
status: "queued",
|
|
runtimeMode: "native",
|
|
driverKind: "codex",
|
|
contextSnapshot: {
|
|
issueId: seeded.issueId,
|
|
interactionId: seeded.interaction.id,
|
|
sourceRunId: seeded.sourceRunId,
|
|
source: "issue.interaction.respond",
|
|
},
|
|
})
|
|
.returning();
|
|
if (!dedicatedRun) throw new Error("Expected dedicated continuation run");
|
|
const wakeup = vi.fn().mockResolvedValue(dedicatedRun);
|
|
const cancelRun = vi.fn().mockResolvedValue({
|
|
id: seeded.sourceRunId,
|
|
status: "cancelled",
|
|
});
|
|
const steer = vi.fn();
|
|
const resolveNativeQuestion = vi
|
|
.fn()
|
|
.mockResolvedValue("queued" as const);
|
|
const outcome = await questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup, cancelRun } as never,
|
|
steer,
|
|
resolveNativeQuestion,
|
|
}).deliver(seeded.interaction.id);
|
|
|
|
expect(resolveNativeQuestion).not.toHaveBeenCalled();
|
|
expect(steer).not.toHaveBeenCalled();
|
|
expect(cancelRun).toHaveBeenCalledWith(
|
|
seeded.sourceRunId,
|
|
"Superseded by a dedicated external-chat answer continuation",
|
|
expect.objectContaining({
|
|
errorCode: "external_chat_continuation",
|
|
resultJson: expect.objectContaining({
|
|
interactionId: seeded.interaction.id,
|
|
externalChatContinuation: true,
|
|
}),
|
|
terminationGraceMs: 2_000,
|
|
}),
|
|
);
|
|
expect(cancelRun.mock.invocationCallOrder[0]).toBeLessThan(
|
|
wakeup.mock.invocationCallOrder[0]!,
|
|
);
|
|
expect(wakeup).toHaveBeenCalledWith(
|
|
seeded.agentId,
|
|
expect.objectContaining({
|
|
allowRunCoalescing: false,
|
|
idempotencyKey: `question-response:${seeded.interaction.id}`,
|
|
payload: expect.objectContaining({
|
|
externalChatContinuation: true,
|
|
sourceCommentId: seeded.sourceCommentId,
|
|
wakeCommentId: seeded.sourceCommentId,
|
|
wakeCommentIds: [seeded.sourceCommentId],
|
|
}),
|
|
contextSnapshot: expect.objectContaining({
|
|
externalChatContinuation: true,
|
|
sourceCommentId: seeded.sourceCommentId,
|
|
wakeCommentId: seeded.sourceCommentId,
|
|
wakeCommentIds: [seeded.sourceCommentId],
|
|
}),
|
|
}),
|
|
);
|
|
expect(outcome).toMatchObject({
|
|
status: "fallback_queued",
|
|
mode: "wake_fallback",
|
|
targetRunId: dedicatedRun.id,
|
|
});
|
|
expect(dedicatedRun.id).not.toBe(seeded.sourceRunId);
|
|
const [delivery] = await db
|
|
.select()
|
|
.from(issueQuestionResponseDeliveries);
|
|
expect(delivery).toMatchObject({
|
|
lastErrorCode: "steering_external_chat_context_incompatible",
|
|
targetRunId: dedicatedRun.id,
|
|
});
|
|
},
|
|
);
|
|
|
|
it("does not promote a source-run wake comment from another issue", async () => {
|
|
const seeded = await seed();
|
|
const goalId = await db
|
|
.select({ goalId: issues.goalId })
|
|
.from(issues)
|
|
.where(eq(issues.id, seeded.issueId))
|
|
.then((rows) => rows[0]?.goalId);
|
|
if (!goalId) throw new Error("Expected source issue goal");
|
|
const otherIssueId = randomUUID();
|
|
const otherCommentId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: otherIssueId,
|
|
companyId: seeded.companyId,
|
|
goalId,
|
|
title: "Unrelated issue",
|
|
status: "todo",
|
|
priority: "medium",
|
|
});
|
|
await db.insert(issueComments).values({
|
|
id: otherCommentId,
|
|
companyId: seeded.companyId,
|
|
issueId: otherIssueId,
|
|
authorUserId: "external-user",
|
|
body: "This other issue must remain outside the continuation.",
|
|
});
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
contextSnapshot: {
|
|
issueId: seeded.issueId,
|
|
source: "chat:slack",
|
|
wakeCommentId: otherCommentId,
|
|
wakeCommentIds: [otherCommentId],
|
|
},
|
|
})
|
|
.where(eq(heartbeatRuns.id, seeded.sourceRunId));
|
|
|
|
const fallbackRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: fallbackRunId,
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
invocationSource: "automation",
|
|
status: "queued",
|
|
runtimeMode: "native",
|
|
driverKind: "codex",
|
|
contextSnapshot: {
|
|
issueId: seeded.issueId,
|
|
interactionId: seeded.interaction.id,
|
|
source: "issue.interaction.respond",
|
|
},
|
|
});
|
|
const wakeup = vi.fn().mockResolvedValue({
|
|
id: fallbackRunId,
|
|
driverKind: "codex",
|
|
});
|
|
const resolveNativeQuestion = vi.fn().mockResolvedValue("queued" as const);
|
|
const outcome = await questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
resolveNativeQuestion,
|
|
}).deliver(seeded.interaction.id);
|
|
|
|
expect(resolveNativeQuestion).not.toHaveBeenCalled();
|
|
expect(outcome).toMatchObject({
|
|
mode: "wake_fallback",
|
|
targetRunId: fallbackRunId,
|
|
});
|
|
const wakeOptions = wakeup.mock.calls[0]?.[1];
|
|
expect(wakeOptions?.payload).toMatchObject({ sourceCommentId: null });
|
|
expect(wakeOptions?.payload).not.toHaveProperty("wakeCommentId");
|
|
expect(wakeOptions?.payload).not.toHaveProperty("wakeCommentIds");
|
|
expect(wakeOptions?.contextSnapshot).toMatchObject({
|
|
sourceCommentId: null,
|
|
});
|
|
expect(wakeOptions?.contextSnapshot).not.toHaveProperty("wakeCommentId");
|
|
expect(wakeOptions?.contextSnapshot).not.toHaveProperty("wakeCommentIds");
|
|
});
|
|
|
|
it("resolves an in-flight native input request before creating a continuation", async () => {
|
|
const seeded = await seed({
|
|
adapterType: "paperclip_runner",
|
|
runtimeMode: "native",
|
|
sourceStatus: "running",
|
|
});
|
|
const wakeup = vi.fn();
|
|
const resolveNativeQuestion = vi.fn().mockResolvedValue("queued" as const);
|
|
|
|
const outcome = await questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
resolveNativeQuestion,
|
|
}).deliver(seeded.interaction.id);
|
|
|
|
expect(outcome).toMatchObject({
|
|
status: "delivered",
|
|
mode: "steered",
|
|
targetRunId: seeded.sourceRunId,
|
|
});
|
|
expect(resolveNativeQuestion).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
id: seeded.interaction.id,
|
|
status: "answered",
|
|
}),
|
|
);
|
|
expect(wakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("keeps native input delivery pending while its PRP session is unavailable", async () => {
|
|
const seeded = await seed({
|
|
adapterType: "paperclip_runner",
|
|
runtimeMode: "native",
|
|
sourceStatus: "running",
|
|
});
|
|
const wakeup = vi.fn();
|
|
|
|
const outcome = await questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
resolveNativeQuestion: vi.fn().mockResolvedValue("pending" as const),
|
|
}).deliver(seeded.interaction.id);
|
|
|
|
expect(outcome).toBeNull();
|
|
expect(wakeup).not.toHaveBeenCalled();
|
|
const [delivery] = await db.select().from(issueQuestionResponseDeliveries);
|
|
expect(delivery).toMatchObject({
|
|
status: "pending",
|
|
attemptCount: 1,
|
|
errorCount: 0,
|
|
lastErrorCode: "native_question_session_unavailable",
|
|
});
|
|
});
|
|
|
|
it("coalesces into a queued successor without creating another wake", async () => {
|
|
const seeded = await seed({ successorStatus: "queued" });
|
|
const successor = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, seeded.successorRunId!))
|
|
.then((rows) => rows[0]!);
|
|
const wakeup = vi.fn().mockResolvedValue(successor);
|
|
const steer = vi.fn();
|
|
const outcome = await questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
steer,
|
|
}).deliver(seeded.interaction.id);
|
|
|
|
expect(outcome).toMatchObject({
|
|
status: "delivered",
|
|
mode: "coalesced",
|
|
targetRunId: successor.id,
|
|
});
|
|
expect(steer).not.toHaveBeenCalled();
|
|
expect(wakeup).toHaveBeenCalledTimes(1);
|
|
expect(wakeup.mock.calls[0]?.[1]).toMatchObject({
|
|
idempotencyKey: `question-response:${seeded.interaction.id}`,
|
|
contextSnapshot: {
|
|
interactionId: seeded.interaction.id,
|
|
interactionStatus: "answered",
|
|
},
|
|
});
|
|
});
|
|
|
|
it("never steers into the source run and keeps a skipped wake retryable", async () => {
|
|
const seeded = await seed({ sourceStatus: "running" });
|
|
const wakeup = vi.fn().mockResolvedValue(null);
|
|
const steer = vi.fn();
|
|
const outcome = await questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
steer,
|
|
}).deliver(seeded.interaction.id);
|
|
|
|
expect(outcome).toBeNull();
|
|
expect(steer).not.toHaveBeenCalled();
|
|
expect(wakeup).toHaveBeenCalledTimes(1);
|
|
expect(JSON.stringify(wakeup.mock.calls[0])).not.toContain("Internal API");
|
|
expect(JSON.stringify(wakeup.mock.calls[0])).not.toContain("Node.js");
|
|
const [delivery] = await db.select().from(issueQuestionResponseDeliveries);
|
|
expect(delivery).toMatchObject({
|
|
status: "pending",
|
|
attemptCount: 1,
|
|
lastErrorCode: "question_response_wake_skipped",
|
|
});
|
|
});
|
|
|
|
it("delivers after wake suppression outlasts the bounded error retry limit", async () => {
|
|
const seeded = await seed({ sourceStatus: "running" });
|
|
const fallbackRunId = randomUUID();
|
|
let wakeAttempts = 0;
|
|
const wakeup = vi.fn().mockImplementation(async () => {
|
|
wakeAttempts += 1;
|
|
if (wakeAttempts <= 5) return null;
|
|
return db
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
id: fallbackRunId,
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
invocationSource: "automation",
|
|
status: "queued",
|
|
runtimeMode: "legacy",
|
|
driverKind: "codex",
|
|
contextSnapshot: { issueId: seeded.issueId },
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]!);
|
|
});
|
|
const service = questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
steer: vi.fn(),
|
|
});
|
|
|
|
for (let attempt = 0; attempt < 5; attempt += 1) {
|
|
await expect(service.deliver(seeded.interaction.id)).resolves.toBeNull();
|
|
}
|
|
const delivered = await service.deliver(seeded.interaction.id);
|
|
|
|
expect(delivered).toMatchObject({
|
|
status: "fallback_queued",
|
|
mode: "wake_fallback",
|
|
targetRunId: fallbackRunId,
|
|
});
|
|
const [delivery] = await db.select().from(issueQuestionResponseDeliveries);
|
|
expect(delivery).toMatchObject({
|
|
status: "fallback_queued",
|
|
attemptCount: 6,
|
|
errorCount: 0,
|
|
targetRunId: fallbackRunId,
|
|
});
|
|
});
|
|
|
|
it("preserves the actual-error retry budget after prolonged wake suppression", async () => {
|
|
const seeded = await seed({ sourceStatus: "running" });
|
|
const fallbackRunId = randomUUID();
|
|
let wakeAttempts = 0;
|
|
const wakeup = vi.fn().mockImplementation(async () => {
|
|
wakeAttempts += 1;
|
|
if (wakeAttempts <= 5) return null;
|
|
if (wakeAttempts === 6)
|
|
throw new Error("scheduler temporarily unavailable");
|
|
return db
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
id: fallbackRunId,
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
invocationSource: "automation",
|
|
status: "queued",
|
|
runtimeMode: "legacy",
|
|
driverKind: "codex",
|
|
contextSnapshot: { issueId: seeded.issueId },
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]!);
|
|
});
|
|
const service = questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
steer: vi.fn(),
|
|
});
|
|
|
|
for (let attempt = 0; attempt < 5; attempt += 1) {
|
|
await expect(service.deliver(seeded.interaction.id)).resolves.toBeNull();
|
|
}
|
|
await expect(service.deliver(seeded.interaction.id)).resolves.toBeNull();
|
|
const [afterError] = await db
|
|
.select()
|
|
.from(issueQuestionResponseDeliveries);
|
|
expect(afterError).toMatchObject({
|
|
status: "pending",
|
|
attemptCount: 6,
|
|
errorCount: 1,
|
|
lastErrorCode: "scheduler temporarily unavailable",
|
|
});
|
|
|
|
await expect(service.deliver(seeded.interaction.id)).resolves.toMatchObject(
|
|
{
|
|
status: "fallback_queued",
|
|
targetRunId: fallbackRunId,
|
|
},
|
|
);
|
|
const [delivered] = await db.select().from(issueQuestionResponseDeliveries);
|
|
expect(delivered).toMatchObject({
|
|
status: "fallback_queued",
|
|
attemptCount: 7,
|
|
errorCount: 1,
|
|
targetRunId: fallbackRunId,
|
|
});
|
|
});
|
|
|
|
it("enforces one durable wake per question-response idempotency key", async () => {
|
|
const seeded = await seed({ sourceStatus: "running" });
|
|
const idempotencyKey = `question-response:${seeded.interaction.id}`;
|
|
const request = {
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
reason: "issue_commented",
|
|
idempotencyKey,
|
|
} as const;
|
|
|
|
await db
|
|
.insert(agentWakeupRequests)
|
|
.values({ ...request, status: "queued" });
|
|
await expect(
|
|
db.insert(agentWakeupRequests).values({
|
|
...request,
|
|
status: "coalesced",
|
|
}),
|
|
).rejects.toMatchObject({ cause: { code: "23505" } });
|
|
|
|
// Suppression receipts are intentionally outside the fence so the outbox
|
|
// can retry after scheduling is enabled again.
|
|
await expect(
|
|
db.insert(agentWakeupRequests).values({
|
|
...request,
|
|
status: "skipped",
|
|
finishedAt: new Date(),
|
|
}),
|
|
).resolves.toBeDefined();
|
|
});
|
|
|
|
it("reuses the winning wake when a concurrent insert hits the idempotency fence", async () => {
|
|
const seeded = await seed({ sourceStatus: "running" });
|
|
const fallbackRunId = randomUUID();
|
|
const wakeup = vi
|
|
.fn()
|
|
.mockImplementation(
|
|
async (
|
|
_agentId: string,
|
|
options: { idempotencyKey?: string | null },
|
|
) => {
|
|
const request = {
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
reason: "issue_commented",
|
|
idempotencyKey: options.idempotencyKey,
|
|
} as const;
|
|
const [winner] = await db
|
|
.insert(agentWakeupRequests)
|
|
.values({
|
|
...request,
|
|
status: "queued",
|
|
})
|
|
.returning();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: fallbackRunId,
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
invocationSource: "automation",
|
|
status: "queued",
|
|
runtimeMode: "legacy",
|
|
driverKind: "codex",
|
|
wakeupRequestId: winner!.id,
|
|
contextSnapshot: { issueId: seeded.issueId },
|
|
});
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId: fallbackRunId })
|
|
.where(eq(agentWakeupRequests.id, winner!.id));
|
|
|
|
// Model the losing claimant reaching the same transactional insert after
|
|
// the winner commits. The service must recover the winner's receipt.
|
|
await db.insert(agentWakeupRequests).values({
|
|
...request,
|
|
status: "coalesced",
|
|
});
|
|
throw new Error("unreachable");
|
|
},
|
|
);
|
|
|
|
const outcome = await questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
steer: vi.fn(),
|
|
}).deliver(seeded.interaction.id);
|
|
|
|
expect(outcome).toMatchObject({
|
|
status: "fallback_queued",
|
|
mode: "wake_fallback",
|
|
targetRunId: fallbackRunId,
|
|
});
|
|
expect(wakeup).toHaveBeenCalledTimes(1);
|
|
const [delivery] = await db.select().from(issueQuestionResponseDeliveries);
|
|
expect(delivery).toMatchObject({
|
|
status: "fallback_queued",
|
|
attemptCount: 1,
|
|
errorCount: 0,
|
|
targetRunId: fallbackRunId,
|
|
});
|
|
});
|
|
|
|
it("settles against a canonical wake from the previous assignee without retrying forever", async () => {
|
|
const seeded = await seed({ sourceStatus: "running" });
|
|
const replacementAgentId = randomUUID();
|
|
await db.insert(agents).values({
|
|
id: replacementAgentId,
|
|
companyId: seeded.companyId,
|
|
name: "Replacement runner",
|
|
role: "engineer",
|
|
status: "active",
|
|
adapterType: "codex_local",
|
|
adapterConfig: {},
|
|
runtimeConfig: {},
|
|
permissions: {},
|
|
});
|
|
await db
|
|
.update(issues)
|
|
.set({ assigneeAgentId: replacementAgentId })
|
|
.where(eq(issues.id, seeded.issueId));
|
|
await db.insert(agentWakeupRequests).values({
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
reason: "issue_commented",
|
|
status: "queued",
|
|
idempotencyKey: `question-response:${seeded.interaction.id}`,
|
|
});
|
|
const wakeup = vi.fn();
|
|
|
|
const outcome = await questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
steer: vi.fn(),
|
|
}).deliver(seeded.interaction.id);
|
|
|
|
expect(wakeup).not.toHaveBeenCalled();
|
|
expect(outcome).toMatchObject({
|
|
status: "fallback_queued",
|
|
mode: "wake_fallback",
|
|
targetRunId: null,
|
|
});
|
|
const [delivery] = await db.select().from(issueQuestionResponseDeliveries);
|
|
expect(delivery).toMatchObject({
|
|
status: "fallback_queued",
|
|
attemptCount: 1,
|
|
errorCount: 0,
|
|
});
|
|
});
|
|
|
|
it("reuses a durable wake receipt instead of issuing a duplicate continuation", async () => {
|
|
const seeded = await seed({ sourceStatus: "running" });
|
|
const [wakeRequest] = await db
|
|
.insert(agentWakeupRequests)
|
|
.values({
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
reason: "issue_commented",
|
|
status: "queued",
|
|
idempotencyKey: `question-response:${seeded.interaction.id}`,
|
|
})
|
|
.returning();
|
|
const [wakeRun] = await db
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
invocationSource: "automation",
|
|
status: "queued",
|
|
runtimeMode: "legacy",
|
|
wakeupRequestId: wakeRequest!.id,
|
|
contextSnapshot: { issueId: seeded.issueId },
|
|
})
|
|
.returning();
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId: wakeRun!.id })
|
|
.where(eq(agentWakeupRequests.id, wakeRequest!.id));
|
|
const wakeup = vi.fn();
|
|
|
|
const outcome = await questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
steer: vi.fn(),
|
|
}).deliver(seeded.interaction.id);
|
|
|
|
expect(outcome).toMatchObject({
|
|
status: "delivered",
|
|
mode: "coalesced",
|
|
targetRunId: wakeRun!.id,
|
|
});
|
|
expect(wakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("recovers a completed wake when receipt finalization was interrupted", async () => {
|
|
const seeded = await seed({ sourceStatus: "running" });
|
|
const [wakeRequest] = await db
|
|
.insert(agentWakeupRequests)
|
|
.values({
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
reason: "issue_commented",
|
|
status: "completed",
|
|
idempotencyKey: `question-response:${seeded.interaction.id}`,
|
|
finishedAt: new Date(),
|
|
})
|
|
.returning();
|
|
const [wakeRun] = await db
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
invocationSource: "automation",
|
|
status: "succeeded",
|
|
runtimeMode: "legacy",
|
|
wakeupRequestId: wakeRequest!.id,
|
|
contextSnapshot: { issueId: seeded.issueId },
|
|
finishedAt: new Date(),
|
|
})
|
|
.returning();
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId: wakeRun!.id })
|
|
.where(eq(agentWakeupRequests.id, wakeRequest!.id));
|
|
const wakeup = vi.fn();
|
|
|
|
const outcome = await questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
steer: vi.fn(),
|
|
}).deliver(seeded.interaction.id);
|
|
|
|
expect(outcome).toMatchObject({
|
|
status: "fallback_queued",
|
|
mode: "wake_fallback",
|
|
targetRunId: wakeRun!.id,
|
|
});
|
|
expect(wakeup).not.toHaveBeenCalled();
|
|
const [delivery] = await db.select().from(issueQuestionResponseDeliveries);
|
|
expect(delivery).toMatchObject({
|
|
status: "fallback_queued",
|
|
errorCount: 0,
|
|
targetRunId: wakeRun!.id,
|
|
});
|
|
});
|
|
|
|
it("keeps a long wake claim leased while the side effect is active", async () => {
|
|
const seeded = await seed({ sourceStatus: "running" });
|
|
// Each call gets its own resolver. A second, unexpected call fails at
|
|
// once instead of sharing one resolver with the first call and hanging.
|
|
const wakeResolvers: Array<(value: null) => void> = [];
|
|
const wakeup = vi.fn(() => {
|
|
if (wakeResolvers.length > 0) {
|
|
throw new Error(
|
|
"wakeup was invoked a second time for the same claim; a re-entrant delivery must fail at once, not hang",
|
|
);
|
|
}
|
|
return new Promise<null>((resolve) => {
|
|
wakeResolvers.push(resolve);
|
|
});
|
|
});
|
|
|
|
// The renewal timer and the sweep both read time from this injected
|
|
// clock. The test moves the clock by hand, so the assertions below do
|
|
// not depend on the speed of a database round trip or a timer callback.
|
|
let clock = new Date("2026-01-01T00:00:00.000Z");
|
|
const service = questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
steer: vi.fn(),
|
|
now: () => clock,
|
|
claimStaleMs: 40,
|
|
claimRefreshMs: 5,
|
|
});
|
|
|
|
const deliveryPromise = service.deliver(seeded.interaction.id);
|
|
await vi.waitFor(() => expect(wakeup).toHaveBeenCalledTimes(1));
|
|
const [claimedRow] = await db
|
|
.select()
|
|
.from(issueQuestionResponseDeliveries)
|
|
.where(
|
|
eq(
|
|
issueQuestionResponseDeliveries.interactionId,
|
|
seeded.interaction.id,
|
|
),
|
|
);
|
|
const claimedAt = claimedRow!.lastAttemptAt!.getTime();
|
|
|
|
// Move the clock past claimStaleMs, then wait for a real renewal tick to
|
|
// pick it up. This proves the renewal ran. It does not only assert that
|
|
// a fixed real-time sleep was long enough.
|
|
clock = new Date(clock.getTime() + 1000);
|
|
await vi.waitFor(async () => {
|
|
const [row] = await db
|
|
.select()
|
|
.from(issueQuestionResponseDeliveries)
|
|
.where(
|
|
eq(
|
|
issueQuestionResponseDeliveries.interactionId,
|
|
seeded.interaction.id,
|
|
),
|
|
);
|
|
expect(row?.lastAttemptAt?.getTime()).toBeGreaterThan(claimedAt);
|
|
});
|
|
|
|
await expect(service.sweepPending()).resolves.toMatchObject({ scanned: 0 });
|
|
wakeResolvers[0]!(null);
|
|
await expect(deliveryPromise).resolves.toBeNull();
|
|
|
|
expect(wakeup).toHaveBeenCalledTimes(1);
|
|
const [delivery] = await db
|
|
.select()
|
|
.from(issueQuestionResponseDeliveries)
|
|
.where(
|
|
eq(
|
|
issueQuestionResponseDeliveries.interactionId,
|
|
seeded.interaction.id,
|
|
),
|
|
);
|
|
expect(delivery).toMatchObject({ status: "pending", attemptCount: 1 });
|
|
});
|
|
|
|
it("fences a stale worker after a newer claim generation takes ownership", async () => {
|
|
const seeded = await seed({ sourceStatus: "running" });
|
|
let releaseFirstWake!: (value: { id: string; driverKind: string }) => void;
|
|
const firstWakeup = vi.fn(
|
|
() =>
|
|
new Promise<{ id: string; driverKind: string }>((resolve) => {
|
|
releaseFirstWake = resolve;
|
|
}),
|
|
);
|
|
const firstService = questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup: firstWakeup } as never,
|
|
steer: vi.fn(),
|
|
claimStaleMs: 40,
|
|
claimRefreshMs: 5,
|
|
});
|
|
|
|
const firstDelivery = firstService.deliver(seeded.interaction.id);
|
|
await vi.waitFor(() => expect(firstWakeup).toHaveBeenCalledTimes(1));
|
|
|
|
// Simulate recovery after the first worker stopped renewing. The next
|
|
// claim increments attemptCount, which is the fencing generation.
|
|
await db
|
|
.update(issueQuestionResponseDeliveries)
|
|
.set({
|
|
status: "pending",
|
|
lastAttemptAt: new Date(0),
|
|
})
|
|
.where(
|
|
eq(
|
|
issueQuestionResponseDeliveries.interactionId,
|
|
seeded.interaction.id,
|
|
),
|
|
);
|
|
|
|
const secondRunId = randomUUID();
|
|
const secondWakeup = vi.fn().mockImplementation(async () =>
|
|
db
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
id: secondRunId,
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
invocationSource: "automation",
|
|
status: "queued",
|
|
runtimeMode: "legacy",
|
|
driverKind: "codex",
|
|
contextSnapshot: { issueId: seeded.issueId },
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]!),
|
|
);
|
|
const secondOutcome = await questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup: secondWakeup } as never,
|
|
steer: vi.fn(),
|
|
}).deliver(seeded.interaction.id);
|
|
|
|
expect(secondOutcome).toMatchObject({
|
|
status: "fallback_queued",
|
|
targetRunId: secondRunId,
|
|
duplicate: false,
|
|
});
|
|
releaseFirstWake({ id: randomUUID(), driverKind: "codex" });
|
|
await expect(firstDelivery).resolves.toMatchObject({
|
|
status: "fallback_queued",
|
|
targetRunId: secondRunId,
|
|
duplicate: true,
|
|
});
|
|
|
|
const [delivery] = await db.select().from(issueQuestionResponseDeliveries);
|
|
expect(delivery).toMatchObject({
|
|
status: "fallback_queued",
|
|
targetRunId: secondRunId,
|
|
attemptCount: 2,
|
|
});
|
|
const deliveryEvents = await db
|
|
.select()
|
|
.from(activityLog)
|
|
.where(eq(activityLog.action, "issue.question_response_delivered"));
|
|
expect(deliveryEvents).toHaveLength(1);
|
|
});
|
|
|
|
it.each(DIRECT_ADAPTER_TYPES)(
|
|
"keeps %s on the existing wake path without invoking native steering",
|
|
async (adapterType) => {
|
|
const seeded = await seed({
|
|
adapterType,
|
|
runtimeMode: "legacy",
|
|
successorStatus: "running",
|
|
});
|
|
const fallbackRunId = randomUUID();
|
|
const wakeup = vi.fn().mockImplementation(async () =>
|
|
db
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
id: fallbackRunId,
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
invocationSource: "automation",
|
|
status: "queued",
|
|
runtimeMode: "legacy",
|
|
contextSnapshot: { issueId: seeded.issueId },
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]!),
|
|
);
|
|
const steer = vi.fn();
|
|
|
|
const outcome = await questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
steer,
|
|
}).deliver(seeded.interaction.id);
|
|
|
|
expect(outcome).toMatchObject({
|
|
status: "fallback_queued",
|
|
mode: "wake_fallback",
|
|
targetRunId: fallbackRunId,
|
|
});
|
|
expect(steer).not.toHaveBeenCalled();
|
|
expect(wakeup).toHaveBeenCalledTimes(1);
|
|
expect(wakeup.mock.calls[0]?.[1]).toMatchObject({
|
|
idempotencyKey: `question-response:${seeded.interaction.id}`,
|
|
contextSnapshot: {
|
|
issueId: seeded.issueId,
|
|
interactionId: seeded.interaction.id,
|
|
interactionStatus: "answered",
|
|
},
|
|
});
|
|
},
|
|
);
|
|
|
|
it("falls back once when successor steering is unsupported", async () => {
|
|
const seeded = await seed({ successorStatus: "running" });
|
|
const fallbackRunId = randomUUID();
|
|
const steer = vi.fn().mockRejectedValue(
|
|
Object.assign(new Error("unsupported"), {
|
|
code: "steering_unsupported",
|
|
}),
|
|
);
|
|
const wakeup = vi.fn().mockImplementation(async () =>
|
|
db
|
|
.insert(heartbeatRuns)
|
|
.values({
|
|
id: fallbackRunId,
|
|
companyId: seeded.companyId,
|
|
agentId: seeded.agentId,
|
|
invocationSource: "automation",
|
|
status: "queued",
|
|
driverKind: "codex",
|
|
contextSnapshot: { issueId: seeded.issueId },
|
|
})
|
|
.returning()
|
|
.then((rows) => rows[0]!),
|
|
);
|
|
const service = questionResponseDeliveryService(db, {
|
|
heartbeat: { wakeup } as never,
|
|
steer,
|
|
});
|
|
const first = await service.deliver(seeded.interaction.id);
|
|
const second = await service.deliver(seeded.interaction.id);
|
|
|
|
expect(first).toMatchObject({
|
|
status: "fallback_queued",
|
|
mode: "wake_fallback",
|
|
targetRunId: fallbackRunId,
|
|
});
|
|
expect(second?.duplicate).toBe(true);
|
|
expect(steer).toHaveBeenCalledTimes(1);
|
|
expect(wakeup).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("formats text, select labels, multi-select, and custom answers in order", async () => {
|
|
const seeded = await seed();
|
|
const envelope = buildQuestionResponseDeliveryEnvelope(seeded.interaction);
|
|
expect(envelope.response).toEqual({
|
|
schema: "paperclip.question_response.v1",
|
|
answers: {
|
|
purpose: { text: "Internal API" },
|
|
runtime: { selectedOptionIds: ["node"] },
|
|
features: {
|
|
selectedOptionIds: ["health", "logs"],
|
|
customText: "Metrics",
|
|
},
|
|
},
|
|
});
|
|
expect(formatQuestionResponseSteeringMessage(envelope)).toBe(
|
|
[
|
|
"Answered questions",
|
|
"",
|
|
"- Purpose — What is it for?: Internal API",
|
|
"- Runtime — Which runtime?: Node.js",
|
|
"- Features — Which features?: Health check, Request logs, Metrics",
|
|
].join("\n"),
|
|
);
|
|
});
|
|
});
|