mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 08:53:19 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
842 lines
58 KiB
TypeScript
842 lines
58 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { agents, authUsers, companies, completionContracts, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issues, nativeRunFinalizations, nativeRunResults, issueThreadInteractions, chatTaskHandoffs, chatCompletionDeliveries, agentWakeupRequests, statusDecisions, type Db, approvals, issueApprovals } from "@paperclipai/db";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
|
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
|
import { NativeRunCoordinatorStore } from "./native-run-coordinator-store.js";
|
|
import { finalizeNativeRun, recoverCommittedNativeChatResponse } from "./native-run-finalizer.js";
|
|
import { nativeResponsePresentationWork } from "./native-response-presentation-work.js";
|
|
import { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
|
import { idleWorkSnapshot } from "../task-admission.js";
|
|
import { chatCompletionDeliveryService, existingChatCompletionReply } from "../chat-completion-delivery.js";
|
|
import { instanceSettingsService } from "../instance-settings.js";
|
|
import { issueService } from "../issues.js";
|
|
import * as nativeResponseNotifications from "../native-response-presentation-notifications.js";
|
|
import { compactRunLogChunk } from "../heartbeat/run-log.js";
|
|
|
|
const runLogReads = vi.hoisted(() => [] as unknown[]);
|
|
vi.mock("../run-log-store.js", async (importOriginal) => {
|
|
const actual = await importOriginal<typeof import("../run-log-store.js")>();
|
|
return { ...actual, getRunLogStore: () => {
|
|
const store = actual.getRunLogStore();
|
|
return { ...store, read: (...args: Parameters<typeof store.read>) => { runLogReads.push(args[0]); return store.read(...args); } };
|
|
} };
|
|
});
|
|
|
|
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_AGENT_CHAT_RESPONSE_TEST_DATABASE_URL");
|
|
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
|
describe.skipIf(!support.supported)("committed Agent Chat reply recovery", () => {
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
|
let db: Db;
|
|
beforeAll(async () => {
|
|
if (!external) database = await startEmbeddedPostgresTestDatabase("agent-chat-response-");
|
|
db = createDb(external ?? database!.connectionString);
|
|
}, 60_000);
|
|
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
|
|
|
|
async function fixture(conversation = true, options: { succeed?: boolean; disposition?: "done" | "needs_review" | "yielded"; channel?: "final" | "unknown";
|
|
beforeFinalize?: (ids: { companyId: string; issueId: string; runId: string; agentId: string }) => Promise<void>;
|
|
reply?: string; logRef?: string } = {}) {
|
|
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
|
|
const sessionId = randomUUID(), runnerId = randomUUID(), userId = `fixture-${randomUUID()}`, wakeId = randomUUID();
|
|
await db.insert(authUsers).values({ id: userId, name: "Conversation owner", email: `${userId}@example.test`,
|
|
createdAt: new Date("2020-01-01T00:00:00Z"), updatedAt: new Date("2020-01-01T00:00:00Z") });
|
|
await db.insert(companies).values({ id: companyId, name: "Reply recovery fixture", issuePrefix: companyId.slice(0, 8) });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Conversation agent", adapterType: "paperclip_runner" });
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Conversation reply", status: "in_progress", assigneeAgentId: agentId,
|
|
...(conversation ? { conversationAgentId: agentId, conversationUserId: userId, conversationState: "active", conversationSessionGeneration: 0 } : {}) });
|
|
await db.insert(issueComments).values({ id: wakeId, companyId, issueId, authorUserId: userId, body: "Hire one teammate and tell me when ready.",
|
|
createdAt: new Date("2020-01-01T00:00:00Z") });
|
|
const [contract] = await db.insert(completionContracts).values({ companyId, issueId, revision: 1,
|
|
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard", completionAuthority: "server_arbiter",
|
|
incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: { revision: "test", objective: "Answer the user", criteria: [] },
|
|
canonicalSha256: "fixture-conversation-contract", createdByActorType: "system", createdByActorId: "test" }).returning();
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, nativeSessionId: sessionId,
|
|
runnerInstanceId: runnerId, completionContractId: contract.id, completionContractSha256: contract.canonicalSha256,
|
|
status: "running", runtimeMode: "native", responsibleUserId: userId,
|
|
...(options.logRef ? { logStore: "local_file", logRef: options.logRef } : {}),
|
|
contextSnapshot: { issueId, source: "issue.comment", wakeCommentId: wakeId, ...(conversation ? { conversationSessionGeneration: 0 } : {}) } });
|
|
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
|
const port = new NativeRunCoordinatorStore(db, { companyId, issueId, runId, agentId, normalizedSessionId: sessionId,
|
|
runnerSourceInstanceId: runnerId, completionContractId: contract.id, completionContractSha256: contract.canonicalSha256,
|
|
completionContractRevision: "test", completionContractCriterionIds: [] });
|
|
const reply = options.reply ?? "The teammate is hired and ready for your next instruction.";
|
|
const prpEvent = { runId, normalizedSessionId: sessionId, turnId: "hire-turn", sourceEventId: "fixture-final-answer",
|
|
payload: { kind: "agentMessage", channel: "final", providerPhase: "final", text: reply } };
|
|
if (options.channel) prpEvent.payload.channel = options.channel;
|
|
await db.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: 170, eventType: "item.completed", payload: { prpEvent } });
|
|
const disposition = options.disposition ?? "done";
|
|
await options.beforeFinalize?.({ companyId, issueId, runId, agentId });
|
|
await port.completeRun({ result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: disposition,
|
|
summary: "A completed hire, with its final message preserved separately.",
|
|
...(disposition === "yielded" ? { continuation: { kind: "response_wake" as const, summary: "Wait for the next human message", idempotencyKey: "conversation-next-message" } } : {}),
|
|
completionClaim: { contractRevision: "test",
|
|
objectiveSatisfied: true, criteria: [], remainingWork: [] }, evidence: [], verification: [],
|
|
attentionRequests: disposition === "needs_review" ? [{ kind: "review", ownerClass: "human", summary: "Review the completed work." }] : [], artifacts: [] },
|
|
terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed", reportedWorkDisposition: disposition }, turnId: "hire-turn" });
|
|
await finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: false });
|
|
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0].phase).toBe("committed");
|
|
if (options.succeed !== false) await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date("2020-01-01T00:02:00Z") }).where(eq(heartbeatRuns.id, runId));
|
|
const input = { companyId, issueId, runId };
|
|
return { ...input, input, agentId, userId, wakeId, prpEvent, reply };
|
|
}
|
|
it("settles the exact claimed wake when recovery projects a committed native run", async () => {
|
|
// The provider and result are already committed. Recovery owns terminal
|
|
// projection because the live executor will not return to completeRun.
|
|
const f = await fixture(false, { succeed: false });
|
|
const sourceWakeId = randomUUID();
|
|
await db.insert(agentWakeupRequests).values({ id: sourceWakeId, companyId: f.companyId,
|
|
agentId: f.agentId, source: "issue", reason: "issue_assigned", status: "claimed",
|
|
runId: f.runId, claimedAt: new Date(), payload: { issueId: f.issueId } });
|
|
await db.update(heartbeatRuns).set({ wakeupRequestId: sourceWakeId }).where(eq(heartbeatRuns.id, f.runId));
|
|
|
|
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
|
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
|
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, sourceWakeId));
|
|
expect(run).toMatchObject({ status: "succeeded", nativePhase: "committed" });
|
|
expect(run.finishedAt).not.toBeNull();
|
|
expect(wake).toMatchObject({ status: "completed", runId: f.runId });
|
|
expect(wake.finishedAt).not.toBeNull();
|
|
});
|
|
|
|
const comments = (runId: string) => db.select().from(issueComments).where(eq(issueComments.createdByRunId, runId));
|
|
const issue = async (issueId: string) => (await db.select().from(issues).where(eq(issues.id, issueId)))[0];
|
|
|
|
async function completion(f: Awaited<ReturnType<typeof fixture>>) {
|
|
const taskId = randomUUID(), deliveryId = randomUUID();
|
|
await instanceSettingsService(db).updateExperimental({ enableAgentChat: true });
|
|
await db.insert(issues).values({ id: taskId, companyId: f.companyId, title: "Completed review", status: "done", statusVersion: 2 });
|
|
await db.insert(chatTaskHandoffs).values({ companyId: f.companyId, taskId, conversationId: f.issueId,
|
|
agentId: f.agentId, sessionGeneration: 0 });
|
|
await db.insert(chatCompletionDeliveries).values({ id: deliveryId, companyId: f.companyId, taskId,
|
|
statusVersion: 2, status: "queued", targetRunId: f.runId, nextAttemptAt: new Date(0) });
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date() }).where(and(eq(heartbeatRuns.id, f.runId), eq(heartbeatRuns.status, "succeeded")));
|
|
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, source: "automation", wakeReason: "chat_task_completed",
|
|
conversationMode: true, conversationSessionGeneration: 0, chatCompletionDeliveryIds: [deliveryId], chatCompletionUpdates: [{ taskId }] } })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
const wakeup = vi.fn(async (_agentId: string, options: { idempotencyKey: string }) => {
|
|
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
|
status: "skipped", reason: "heartbeat.daily_run_limit", idempotencyKey: options.idempotencyKey });
|
|
return null;
|
|
});
|
|
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
|
status: "completed", reason: "chat_task_completed", idempotencyKey: `chat-completion:${deliveryId}:0`, runId: f.runId });
|
|
const delivery = async () => (await db.select().from(chatCompletionDeliveries).where(eq(chatCompletionDeliveries.id, deliveryId)))[0];
|
|
return { taskId, deliveryId, wakeup, delivery, service: chatCompletionDeliveryService(db, { wakeup }) };
|
|
}
|
|
|
|
it("preserves accepted native callback presentation when outbox retry runs first at the daily cap", async () => {
|
|
const f = await fixture(), outbox = await completion(f);
|
|
const work = nativeResponsePresentationWork(db);
|
|
const [entry] = await work.load(f.input);
|
|
expect(entry).toMatchObject({ id: f.runId });
|
|
|
|
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
await expect(work.check(entry)).resolves.toBeNull();
|
|
const [comment] = await comments(f.runId);
|
|
expect(comment).toMatchObject({ body: f.reply, authorAgentId: f.agentId, createdByRunId: f.runId });
|
|
expect(await comments(f.runId)).toHaveLength(1);
|
|
expect(await issue(f.issueId)).toMatchObject({ status: "in_review", conversationState: "waiting" });
|
|
expect(await outbox.delivery()).toMatchObject({ status: "delivered", targetRunId: f.runId, attempts: 0, responseCommentId: comment.id });
|
|
expect(outbox.wakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each([false, true])("wakes the real reply worker only after outer commit, becomes idle and persists once (rollback=%s)", async rollback => {
|
|
const f = await fixture(true, { succeed: false }), outbox = await completion(f);
|
|
let queries = 0;
|
|
const debug = db.$client.options.debug;
|
|
db.$client.options.debug = () => { queries++; };
|
|
const errors: unknown[] = [];
|
|
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
|
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
|
|
const quiet = () => vi.waitFor(() => {
|
|
expect(coordinator.nextWakeAt()).toBeNull();
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
}, { timeout: 5000 });
|
|
const projectAndDefer = async (abort: boolean) => db.transaction(async tx => {
|
|
await tx.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
|
|
await chatCompletionDeliveryService(tx as unknown as Db, { wakeup: outbox.wakeup }).deliver(outbox.deliveryId);
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
if (abort) throw new Error("fixture-outbox-rollback");
|
|
});
|
|
try {
|
|
await worker.ready; await quiet();
|
|
if (rollback) {
|
|
await expect(projectAndDefer(true)).rejects.toThrow("fixture-outbox-rollback");
|
|
await quiet();
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("running");
|
|
expect(await outbox.delivery()).toMatchObject({ status: "queued", targetRunId: f.runId, attempts: 0, nextAttemptAt: new Date(0) });
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
}
|
|
await projectAndDefer(false);
|
|
await vi.waitFor(async () => expect(await comments(f.runId)).toHaveLength(1), { timeout: 5000 });
|
|
await quiet();
|
|
const [comment] = await comments(f.runId);
|
|
expect(comment).toMatchObject({ body: f.reply, authorAgentId: f.agentId, createdByRunId: f.runId });
|
|
expect(await issue(f.issueId)).toMatchObject({ status: "in_review", conversationState: "waiting" });
|
|
expect(await outbox.delivery()).toMatchObject({ status: "delivered", targetRunId: f.runId, attempts: 0, responseCommentId: comment.id });
|
|
const settledQueries = queries;
|
|
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
|
await vi.advanceTimersByTimeAsync(60_000);
|
|
expect(queries).toBe(settledQueries);
|
|
expect(coordinator.nextWakeAt()).toBeNull();
|
|
vi.useRealTimers();
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
expect(await comments(f.runId)).toHaveLength(1);
|
|
expect(outbox.wakeup).not.toHaveBeenCalled();
|
|
expect(errors).toEqual([]);
|
|
} finally {
|
|
vi.useRealTimers(); worker.stop(); await coordinator.stop(); db.$client.options.debug = debug;
|
|
}
|
|
});
|
|
|
|
it.each(["failed", "cancelled", "missing-final", "no-accepted", "old-generation", "foreign-company", "foreign-agent"])(
|
|
"keeps the existing retry path for %s targets", async fault => {
|
|
const f = await fixture(), outbox = await completion(f);
|
|
if (fault === "failed" || fault === "cancelled") await db.update(heartbeatRuns).set({ status: fault }).where(eq(heartbeatRuns.id, f.runId));
|
|
if (fault === "missing-final") await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId));
|
|
if (fault === "no-accepted") await db.update(nativeRunResults).set({ schemaStatus: "rejected" }).where(eq(nativeRunResults.runId, f.runId));
|
|
if (fault === "old-generation") {
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
|
await db.update(heartbeatRuns).set({ contextSnapshot: { ...run.contextSnapshot, conversationSessionGeneration: 1 } }).where(eq(heartbeatRuns.id, f.runId));
|
|
}
|
|
let foreign: Awaited<ReturnType<typeof fixture>> | undefined;
|
|
if (fault === "foreign-company") {
|
|
foreign = await fixture();
|
|
await db.update(chatCompletionDeliveries).set({ targetRunId: foreign.runId }).where(eq(chatCompletionDeliveries.id, outbox.deliveryId));
|
|
}
|
|
if (fault === "foreign-agent") {
|
|
const otherAgent = randomUUID();
|
|
await db.insert(agents).values({ id: otherAgent, companyId: f.companyId, name: "Other agent", adapterType: "paperclip_runner" });
|
|
await db.update(heartbeatRuns).set({ agentId: otherAgent }).where(eq(heartbeatRuns.id, f.runId));
|
|
}
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
expect(outbox.wakeup).toHaveBeenCalledTimes(1);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "pending", targetRunId: null, attempts: 1, responseCommentId: null });
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
if (foreign) expect(await comments(foreign.runId)).toEqual([]);
|
|
},
|
|
);
|
|
|
|
it.each(["reset-source", "reopened-task"])("keeps %s delivery supersession", async fault => {
|
|
const f = await fixture(), outbox = await completion(f);
|
|
if (fault === "reset-source") await db.update(issues).set({ conversationSessionGeneration: 1 }).where(eq(issues.id, f.issueId));
|
|
else {
|
|
await issueService(db).update(outbox.taskId, { status: "todo" });
|
|
await issueService(db).update(outbox.taskId, { status: "done" });
|
|
expect(await issue(outbox.taskId)).toMatchObject({ status: "done", statusVersion: 4 });
|
|
}
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "superseded", attempts: 0, responseCommentId: null });
|
|
expect(outbox.wakeup).not.toHaveBeenCalled();
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
});
|
|
|
|
it("lets an in-flight presentation acknowledge before expiry can retarget the delivery", async () => {
|
|
const started = Date.now();
|
|
vi.useFakeTimers({ toFake: ["Date"] }); vi.setSystemTime(started);
|
|
const f = await fixture(), outbox = await completion(f);
|
|
const suffix = randomUUID().replaceAll("-", ""), lockId = Number.parseInt(suffix.slice(0, 6), 16) + 1;
|
|
const trigger = `completion_reply_hold_${suffix}`;
|
|
let release!: () => void, acquired!: () => void;
|
|
const hold = new Promise<void>(resolve => { release = resolve; });
|
|
const locked = new Promise<void>(resolve => { acquired = resolve; });
|
|
let blocker: Promise<unknown> | undefined, presentation: Promise<unknown> | undefined, expiry: Promise<unknown> | undefined;
|
|
try {
|
|
await db.execute(sql.raw(`create function ${trigger}() returns trigger language plpgsql as $$
|
|
begin
|
|
if NEW.created_by_run_id = '${f.runId}'::uuid then perform pg_advisory_xact_lock(${lockId}); end if;
|
|
return NEW;
|
|
end $$`));
|
|
await db.execute(sql.raw(`create trigger ${trigger} before insert on issue_comments for each row execute function ${trigger}()`));
|
|
blocker = db.transaction(async tx => { await tx.execute(sql`select pg_advisory_xact_lock(${lockId}::bigint)`); acquired(); await hold; });
|
|
await Promise.race([locked, blocker]);
|
|
const work = nativeResponsePresentationWork(db), [entry] = await work.load(f.input);
|
|
presentation = work.check(entry); void presentation.catch(() => undefined);
|
|
|
|
|
|
await vi.waitFor(async () => {
|
|
const [row] = await db.execute(sql`select exists (select 1 from pg_locks
|
|
where locktype = 'advisory' and classid = 0 and objid = ${lockId}::oid and not granted
|
|
and database = (select oid from pg_database where datname = current_database())) as waiting`);
|
|
expect(row.waiting).toBe(true);
|
|
}, { timeout: 5000 });
|
|
vi.setSystemTime(started + 300_000);
|
|
let expirySettled = false;
|
|
expiry = outbox.service.deliver(outbox.deliveryId).then(() => { expirySettled = true; });
|
|
void expiry.catch(() => undefined);
|
|
await vi.waitFor(async () => {
|
|
const [row] = await db.execute(sql`select exists (select 1 from pg_stat_activity
|
|
where datname = current_database() and pid <> pg_backend_pid() and wait_event_type = 'Lock'
|
|
and query like '%from "issues"%' and query like '%for update%') as waiting`);
|
|
expect(expirySettled || row.waiting).toBe(true);
|
|
}, { timeout: 5000 });
|
|
expect(expirySettled).toBe(false);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "queued", targetRunId: f.runId, attempts: 0, responseCommentId: null });
|
|
release(); await blocker; await presentation; await expiry;
|
|
const [comment] = await comments(f.runId);
|
|
expect(comment).toMatchObject({ body: f.reply, authorAgentId: f.agentId, createdByRunId: f.runId });
|
|
expect(await comments(f.runId)).toHaveLength(1);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "delivered", targetRunId: f.runId, attempts: 0, responseCommentId: comment.id });
|
|
expect(outbox.wakeup).not.toHaveBeenCalled();
|
|
expect(await issue(f.issueId)).toMatchObject({ status: "in_review", conversationState: "waiting" });
|
|
} finally {
|
|
release?.(); await Promise.allSettled([blocker, presentation, expiry].filter((value): value is Promise<unknown> => Boolean(value)));
|
|
await db.execute(sql.raw(`drop trigger if exists ${trigger} on issue_comments`));
|
|
await db.execute(sql.raw(`drop function if exists ${trigger}()`));
|
|
vi.useRealTimers();
|
|
}
|
|
});
|
|
|
|
it("lets expiry win before presentation and releases the child lock before wake admission", async () => {
|
|
const started = Date.now();
|
|
vi.useFakeTimers({ toFake: ["Date"] }); vi.setSystemTime(started);
|
|
try {
|
|
const f = await fixture(), outbox = await completion(f), lockErrors: unknown[] = [];
|
|
const work = nativeResponsePresentationWork(db), [entry] = await work.load(f.input);
|
|
vi.setSystemTime(started + 300_000);
|
|
const wakeup = vi.fn(async (agentId: string, options: { idempotencyKey: string }) => {
|
|
try {
|
|
await db.transaction(async tx => {
|
|
await tx.execute(sql`set local lock_timeout = '500ms'`);
|
|
await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, f.issueId)).for("update");
|
|
await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, outbox.taskId)).for("update");
|
|
});
|
|
} catch (error) { lockErrors.push(error); throw error; }
|
|
return outbox.wakeup(agentId, options);
|
|
});
|
|
await chatCompletionDeliveryService(db, { wakeup }).deliver(outbox.deliveryId);
|
|
expect(lockErrors).toEqual([]);
|
|
expect(wakeup).toHaveBeenCalledTimes(1);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "pending", targetRunId: null, attempts: 1, responseCommentId: null });
|
|
await expect(work.check(entry)).rejects.toThrow("chat_completion_superseded");
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
expect(await issue(outbox.taskId)).toMatchObject({ status: "done", statusVersion: 2 });
|
|
} finally { vi.useRealTimers(); }
|
|
});
|
|
|
|
it("rolls back an expiry transition without waking or losing the original target", async () => {
|
|
const started = Date.now();
|
|
vi.useFakeTimers({ toFake: ["Date"] }); vi.setSystemTime(started);
|
|
try {
|
|
const f = await fixture(), outbox = await completion(f);
|
|
vi.setSystemTime(started + 300_000);
|
|
const rollback = new Error("fixture-expiry-rollback");
|
|
const connection = new Proxy(db, { get(target, key) {
|
|
return key === "transaction" ? async (callback: Parameters<Db["transaction"]>[0]) => target.transaction(async tx => {
|
|
await callback(tx); throw rollback;
|
|
}) : Reflect.get(target, key);
|
|
} });
|
|
await chatCompletionDeliveryService(connection, { wakeup: outbox.wakeup }).deliver(outbox.deliveryId);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "queued", targetRunId: f.runId, attempts: 0, responseCommentId: null,
|
|
error: rollback.message });
|
|
expect(outbox.wakeup).not.toHaveBeenCalled();
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
} finally { vi.useRealTimers(); }
|
|
});
|
|
|
|
it.each(["new-claim", "delivered", "retargeted", "superseded"])("respects a %s winner while expiry waits for the child lock", async winner => {
|
|
const started = Date.now();
|
|
vi.useFakeTimers({ toFake: ["Date"] }); vi.setSystemTime(started);
|
|
const f = await fixture(), outbox = await completion(f);
|
|
const replacementId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: replacementId, companyId: f.companyId, agentId: f.agentId, status: "queued" });
|
|
let release!: () => void, acquired!: () => void;
|
|
const hold = new Promise<void>(resolve => { release = resolve; });
|
|
const locked = new Promise<void>(resolve => { acquired = resolve; });
|
|
let first: Promise<unknown> | undefined, second: Promise<unknown> | undefined;
|
|
const blocker = db.transaction(async tx => {
|
|
await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, outbox.taskId)).for("update"); acquired(); await hold;
|
|
if (winner === "delivered") await tx.update(chatCompletionDeliveries).set({ status: "delivered" }).where(eq(chatCompletionDeliveries.id, outbox.deliveryId));
|
|
if (winner === "retargeted") await tx.update(chatCompletionDeliveries).set({ targetRunId: replacementId, attempts: 1 })
|
|
.where(eq(chatCompletionDeliveries.id, outbox.deliveryId));
|
|
if (winner === "superseded") {
|
|
await tx.update(issues).set({ status: "todo", statusVersion: 3 }).where(eq(issues.id, outbox.taskId));
|
|
await tx.update(chatCompletionDeliveries).set({ status: "superseded" }).where(eq(chatCompletionDeliveries.id, outbox.deliveryId));
|
|
}
|
|
});
|
|
const waiting = async (count: number) => vi.waitFor(async () => {
|
|
const [row] = await db.execute(sql`select count(*)::int as waiting from pg_stat_activity
|
|
where datname = current_database() and pid <> pg_backend_pid() and wait_event_type = 'Lock'
|
|
and query like '%from "issues"%' and query like '%for update%'`);
|
|
expect(row.waiting).toBeGreaterThanOrEqual(count);
|
|
}, { timeout: 5000 });
|
|
try {
|
|
await Promise.race([locked, blocker]);
|
|
vi.setSystemTime(started + 300_000);
|
|
first = outbox.service.deliver(outbox.deliveryId); void first.catch(() => undefined);
|
|
await waiting(1);
|
|
if (winner === "new-claim") {
|
|
vi.setSystemTime(started + 360_000);
|
|
second = outbox.service.deliver(outbox.deliveryId); void second.catch(() => undefined);
|
|
await waiting(2);
|
|
}
|
|
release(); await blocker; await first; await second;
|
|
if (winner === "new-claim") {
|
|
expect(outbox.wakeup).toHaveBeenCalledTimes(1);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "pending", targetRunId: null, attempts: 1 });
|
|
} else {
|
|
expect(outbox.wakeup).not.toHaveBeenCalled();
|
|
if (winner === "delivered") expect((await outbox.delivery()).status).toBe("delivered");
|
|
if (winner === "retargeted") expect(await outbox.delivery()).toMatchObject({ status: "queued", targetRunId: replacementId, attempts: 1 });
|
|
if (winner === "superseded") expect((await outbox.delivery()).status).toBe("superseded");
|
|
}
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
} finally {
|
|
release?.(); await Promise.allSettled([blocker, first, second].filter((value): value is Promise<unknown> => Boolean(value)));
|
|
vi.useRealTimers();
|
|
}
|
|
});
|
|
|
|
it("rechecks the fresh delivery when retry holds the child lock before presentation reads", async () => {
|
|
const started = Date.now();
|
|
vi.useFakeTimers({ toFake: ["Date"] }); vi.setSystemTime(started);
|
|
const f = await fixture(), outbox = await completion(f);
|
|
const suffix = randomUUID().replaceAll("-", ""), lockId = Number.parseInt(suffix.slice(0, 6), 16) + 1;
|
|
const trigger = `completion_retry_hold_${suffix}`;
|
|
let release!: () => void, acquired!: () => void;
|
|
const hold = new Promise<void>(resolve => { release = resolve; });
|
|
const locked = new Promise<void>(resolve => { acquired = resolve; });
|
|
let blocker: Promise<unknown> | undefined, presentation: Promise<unknown> | undefined, expiry: Promise<unknown> | undefined;
|
|
try {
|
|
await db.execute(sql.raw(`create function ${trigger}() returns trigger language plpgsql as $$
|
|
begin
|
|
if OLD.id = '${outbox.deliveryId}'::uuid and NEW.target_run_id is null then perform pg_advisory_xact_lock(${lockId}); end if;
|
|
return NEW;
|
|
end $$`));
|
|
await db.execute(sql.raw(`create trigger ${trigger} before update of target_run_id on chat_completion_deliveries for each row execute function ${trigger}()`));
|
|
blocker = db.transaction(async tx => { await tx.execute(sql`select pg_advisory_xact_lock(${lockId}::bigint)`); acquired(); await hold; });
|
|
await Promise.race([locked, blocker]);
|
|
const work = nativeResponsePresentationWork(db), [entry] = await work.load(f.input);
|
|
vi.setSystemTime(started + 300_000);
|
|
expiry = outbox.service.deliver(outbox.deliveryId); void expiry.catch(() => undefined);
|
|
|
|
|
|
await vi.waitFor(async () => {
|
|
const [row] = await db.execute(sql`select exists (select 1 from pg_locks
|
|
where locktype = 'advisory' and classid = 0 and objid = ${lockId}::oid and not granted
|
|
and database = (select oid from pg_database where datname = current_database())) as waiting`);
|
|
expect(row.waiting).toBe(true);
|
|
}, { timeout: 5000 });
|
|
presentation = work.check(entry); void presentation.catch(() => undefined);
|
|
await vi.waitFor(async () => {
|
|
const [row] = await db.execute(sql`select exists (select 1 from pg_stat_activity
|
|
where datname = current_database() and pid <> pg_backend_pid() and wait_event_type = 'Lock'
|
|
and query like '%from "issues"%' and query like '%for update%') as waiting`);
|
|
expect(row.waiting).toBe(true);
|
|
}, { timeout: 5000 });
|
|
release(); await blocker; await expiry;
|
|
const [outcome] = await Promise.allSettled([presentation]);
|
|
expect(outcome.status).toBe("rejected");
|
|
if (outcome.status === "rejected") expect(outcome.reason.message).toBe("chat_completion_superseded");
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "pending", targetRunId: null, attempts: 1, responseCommentId: null });
|
|
expect(outbox.wakeup).toHaveBeenCalledTimes(1);
|
|
} finally {
|
|
release?.(); await Promise.allSettled([blocker, presentation, expiry].filter((value): value is Promise<unknown> => Boolean(value)));
|
|
await db.execute(sql.raw(`drop trigger if exists ${trigger} on chat_completion_deliveries`));
|
|
await db.execute(sql.raw(`drop function if exists ${trigger}()`));
|
|
vi.useRealTimers();
|
|
}
|
|
});
|
|
|
|
it.each(["within-grace", "at-expiry"])("preserves a Done reassignment's completion event %s and still retries at expiry", async when => {
|
|
const started = Date.now();
|
|
vi.useFakeTimers({ toFake: ["Date"] }); vi.setSystemTime(started);
|
|
try {
|
|
const f = await fixture(), outbox = await completion(f), reassignedAgent = randomUUID();
|
|
await db.insert(agents).values({ id: reassignedAgent, companyId: f.companyId, name: "New task owner", adapterType: "paperclip_runner" });
|
|
await db.update(nativeRunResults).set({ canonicalSha256: "invalid-digest" }).where(eq(nativeRunResults.runId, f.runId));
|
|
await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(started + 120_000) }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const updated = await issueService(db).update(outbox.taskId, { assigneeAgentId: reassignedAgent });
|
|
expect(updated).toMatchObject({ status: "done", statusVersion: 3, assigneeAgentId: reassignedAgent });
|
|
expect(await outbox.delivery()).toMatchObject({ status: "queued", statusVersion: 2, targetRunId: f.runId, attempts: 0 });
|
|
if (when === "within-grace") {
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "queued", targetRunId: f.runId, attempts: 0,
|
|
nextAttemptAt: new Date(started + 120_000) });
|
|
expect(outbox.wakeup).not.toHaveBeenCalled();
|
|
}
|
|
vi.setSystemTime(started + 300_000);
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "pending", targetRunId: null, attempts: 1, statusVersion: 2 });
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
expect(outbox.wakeup).toHaveBeenCalledTimes(1);
|
|
expect(await issue(outbox.taskId)).toMatchObject({ status: "done", statusVersion: 3, assigneeAgentId: reassignedAgent });
|
|
} finally { vi.useRealTimers(); }
|
|
});
|
|
|
|
it("returns the fresh saved reply receipt after waiting for the child lock", async () => {
|
|
const f = await fixture(), outbox = await completion(f);
|
|
const [saved] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
|
authorAgentId: f.agentId, createdByRunId: f.runId, body: f.reply }).returning();
|
|
let release!: () => void, acquired!: () => void;
|
|
const hold = new Promise<void>(resolve => { release = resolve; });
|
|
const locked = new Promise<void>(resolve => { acquired = resolve; });
|
|
const blocker = db.transaction(async tx => {
|
|
await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, outbox.taskId)).for("update"); acquired(); await hold;
|
|
await tx.update(chatCompletionDeliveries).set({ status: "delivered", responseCommentId: saved.id })
|
|
.where(eq(chatCompletionDeliveries.id, outbox.deliveryId));
|
|
});
|
|
let reader: Promise<Awaited<ReturnType<typeof existingChatCompletionReply>>> | undefined;
|
|
try {
|
|
await Promise.race([locked, blocker]);
|
|
reader = db.transaction(tx => existingChatCompletionReply(tx, f.runId, f.issueId)); void reader.catch(() => undefined);
|
|
await vi.waitFor(async () => {
|
|
const [row] = await db.execute(sql`select exists (select 1 from pg_stat_activity
|
|
where datname = current_database() and pid <> pg_backend_pid() and wait_event_type = 'Lock'
|
|
and query like '%from "issues"%' and query like '%for update%') as waiting`);
|
|
expect(row.waiting).toBe(true);
|
|
}, { timeout: 5000 });
|
|
release(); await blocker;
|
|
expect(await reader).toMatchObject({ id: saved.id, body: f.reply, createdByRunId: f.runId });
|
|
expect(await comments(f.runId)).toHaveLength(1);
|
|
} finally { release?.(); await Promise.allSettled([blocker, reader]); }
|
|
});
|
|
|
|
it("does not let an old hold shorten a newer claim's lease", async () => {
|
|
const started = Date.now();
|
|
vi.useFakeTimers({ toFake: ["Date"] }); vi.setSystemTime(started);
|
|
const f = await fixture(), outbox = await completion(f);
|
|
const notify = nativeResponseNotifications.notifyNativeResponsePresentation;
|
|
let release!: () => void, notified!: () => void;
|
|
const hold = new Promise<void>(resolve => { release = resolve; });
|
|
const ready = new Promise<void>(resolve => { notified = resolve; });
|
|
const spy = vi.spyOn(nativeResponseNotifications, "notifyNativeResponsePresentation").mockImplementationOnce(async (tx, scope) => {
|
|
await notify(tx, scope); notified(); await hold;
|
|
});
|
|
let old: Promise<unknown> | undefined;
|
|
try {
|
|
|
|
|
|
old = outbox.service.deliver(outbox.deliveryId); void old.catch(() => undefined);
|
|
await vi.waitFor(() => expect(spy).toHaveBeenCalledTimes(1), { timeout: 5000 }); await ready;
|
|
vi.setSystemTime(started + 60_000);
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "queued", targetRunId: f.runId, attempts: 0,
|
|
nextAttemptAt: new Date(started + 120_000) });
|
|
release(); await old;
|
|
expect(await outbox.delivery()).toMatchObject({ status: "queued", targetRunId: f.runId, attempts: 0,
|
|
nextAttemptAt: new Date(started + 120_000) });
|
|
expect(outbox.wakeup).not.toHaveBeenCalled();
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
} finally { release?.(); await Promise.allSettled(old ? [old] : []); spy.mockRestore(); vi.useRealTimers(); }
|
|
});
|
|
|
|
it("saves a legitimate delayed reply before the fixed limit without another model run", async () => {
|
|
const started = Date.now();
|
|
vi.useFakeTimers({ toFake: ["Date"] }); vi.setSystemTime(started);
|
|
try {
|
|
const f = await fixture(), outbox = await completion(f);
|
|
for (const elapsed of [0, 60_000, 120_000, 240_000]) {
|
|
vi.setSystemTime(started + elapsed);
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "queued", targetRunId: f.runId, attempts: 0, responseCommentId: null });
|
|
expect(outbox.wakeup).not.toHaveBeenCalled();
|
|
}
|
|
vi.setSystemTime(started + 299_999);
|
|
const work = nativeResponsePresentationWork(db), [entry] = await work.load(f.input);
|
|
await expect(work.check(entry)).resolves.toBeNull();
|
|
const [comment] = await comments(f.runId);
|
|
expect(comment).toMatchObject({ body: f.reply, authorAgentId: f.agentId, createdByRunId: f.runId });
|
|
expect(await issue(f.issueId)).toMatchObject({ status: "in_review", conversationState: "waiting" });
|
|
expect(await outbox.delivery()).toMatchObject({ status: "delivered", targetRunId: f.runId, attempts: 0, responseCommentId: comment.id });
|
|
vi.setSystemTime(started + 300_000);
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
expect(await comments(f.runId)).toHaveLength(1);
|
|
expect(outbox.wakeup).not.toHaveBeenCalled();
|
|
} finally { vi.useRealTimers(); }
|
|
});
|
|
|
|
it("clamps an external native retry deadline to the immutable five-lease presentation limit", async () => {
|
|
const started = Date.now();
|
|
vi.useFakeTimers({ toFake: ["Date"] }); vi.setSystemTime(started);
|
|
try {
|
|
const f = await fixture(), outbox = await completion(f);
|
|
await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(started + 86_400_000) })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "queued", targetRunId: f.runId, attempts: 0,
|
|
nextAttemptAt: new Date(started + 300_000) });
|
|
expect(outbox.wakeup).not.toHaveBeenCalled();
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
vi.setSystemTime(started + 300_000);
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "pending", targetRunId: null, attempts: 1 });
|
|
expect(outbox.wakeup).toHaveBeenCalledTimes(1);
|
|
} finally { vi.useRealTimers(); }
|
|
});
|
|
|
|
it.each(["digest", "external-context"])("bounds permanent %s declines without sliding, publishing or changing accepted work", async fault => {
|
|
const started = Date.now();
|
|
vi.useFakeTimers({ toFake: ["Date"] }); vi.setSystemTime(started);
|
|
try {
|
|
const f = await fixture(), outbox = await completion(f);
|
|
if (fault === "digest") await db.update(nativeRunResults).set({ canonicalSha256: "invalid-digest" }).where(eq(nativeRunResults.runId, f.runId));
|
|
else {
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
|
await db.update(heartbeatRuns).set({ contextSnapshot: { ...run.contextSnapshot, externalChatContinuation: true } }).where(eq(heartbeatRuns.id, f.runId));
|
|
}
|
|
const [acceptedBefore] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId));
|
|
const work = nativeResponsePresentationWork(db), [entry] = await work.load(f.input);
|
|
for (const elapsed of [0, 60_000, 120_000, 240_000]) {
|
|
vi.setSystemTime(started + elapsed);
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
await expect(work.check(entry)).resolves.toBeNull();
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "queued", targetRunId: f.runId, attempts: 0,
|
|
responseCommentId: null, nextAttemptAt: new Date(started + Math.min(elapsed + 60_000, 300_000)) });
|
|
expect(outbox.wakeup).not.toHaveBeenCalled();
|
|
}
|
|
vi.setSystemTime(started + 299_999);
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
expect((await outbox.delivery()).nextAttemptAt).toEqual(new Date(started + 300_000));
|
|
vi.setSystemTime(started + 300_000);
|
|
await outbox.service.deliver(outbox.deliveryId);
|
|
expect(await outbox.delivery()).toMatchObject({ status: "pending", targetRunId: null, attempts: 1, responseCommentId: null });
|
|
expect(outbox.wakeup).toHaveBeenCalledTimes(1);
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId)))[0]).toEqual(acceptedBefore);
|
|
expect(await issue(outbox.taskId)).toMatchObject({ status: "done", statusVersion: 2 });
|
|
} finally { vi.useRealTimers(); }
|
|
});
|
|
|
|
it("finds a committed final Agent Chat turn, persists the exact reply once and only then settles waiting", async () => {
|
|
const f = await fixture();
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
expect(await issue(f.issueId)).toMatchObject({ status: "in_progress", conversationState: "active" });
|
|
const work = nativeResponsePresentationWork(db), entries = await work.load(f.input);
|
|
expect(entries).toEqual([expect.objectContaining({ id: f.runId })]);
|
|
await expect(work.check(entries[0])).resolves.toBeNull();
|
|
expect(await comments(f.runId)).toEqual([expect.objectContaining({ body: f.reply, authorAgentId: f.agentId, createdByRunId: f.runId })]);
|
|
expect(await issue(f.issueId)).toMatchObject({ status: "in_review", conversationState: "waiting" });
|
|
expect(await work.load(f.input)).toEqual([]);
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
|
|
expect(await comments(f.runId)).toHaveLength(1);
|
|
});
|
|
it("recovers a server-finished yielded conversation from its completed unknown-channel reply", async () => {
|
|
const f = await fixture(true, { disposition: "yielded", channel: "unknown" });
|
|
expect((await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId)))[0])
|
|
.toMatchObject({ reasonCode: "conversation_turn_finished" });
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(true);
|
|
expect(await comments(f.runId)).toEqual([expect.objectContaining({ body: f.reply, createdByRunId: f.runId })]);
|
|
expect(await issue(f.issueId)).toMatchObject({ status: "in_review", conversationState: "waiting" });
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
|
|
expect(await comments(f.runId)).toHaveLength(1);
|
|
});
|
|
|
|
it("presents an invite join link from database run events, never from the redacted run log", async () => {
|
|
// The run log redacts invitation tokens; the person must still receive the
|
|
// working join link, so the reply must come from the stored run events.
|
|
const token = "pcp_invite_FixtureInviteToken-0123456789_abcdefghijklmnopqrstu";
|
|
const reply = `Created the invite.\n\n**Invite link:** [Join Company](http://127.0.0.1:3100/invite/${token})`;
|
|
expect(compactRunLogChunk(reply)).not.toContain(token);
|
|
const f = await fixture(true, { reply, logRef: `${randomUUID()}/${randomUUID()}/redacted-run.ndjson` });
|
|
runLogReads.length = 0;
|
|
const work = nativeResponsePresentationWork(db), entries = await work.load(f.input);
|
|
await expect(work.check(entries[0])).resolves.toBeNull();
|
|
expect(await comments(f.runId)).toEqual([expect.objectContaining({ body: reply, createdByRunId: f.runId })]);
|
|
expect(runLogReads).toEqual([]);
|
|
});
|
|
it("keeps a later user message actionable after recovering the earlier exact reply", async () => {
|
|
const f = await fixture();
|
|
await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, authorUserId: f.userId, body: "Another instruction",
|
|
createdAt: new Date("2020-01-01T00:01:00Z") });
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(true);
|
|
expect(await comments(f.runId)).toHaveLength(1);
|
|
expect(await issue(f.issueId)).toMatchObject({ status: "in_progress", conversationState: "active" });
|
|
});
|
|
it("does not duplicate a reply under concurrent recovery", async () => {
|
|
const f = await fixture();
|
|
expect((await Promise.all([recoverCommittedNativeChatResponse(db, f.input), recoverCommittedNativeChatResponse(db, f.input)])).sort())
|
|
.toEqual([false, true]);
|
|
expect(await comments(f.runId)).toHaveLength(1);
|
|
});
|
|
it.each(["skip", "generation", "agent", "user", "execution", "waiting", "turn", "event-run", "session", "channel", "kind", "digest", "partial"])(
|
|
"keeps %s mismatches unchanged instead of publishing an unauthorized or incomplete reply", async fault => {
|
|
const f = await fixture();
|
|
if (["skip", "generation"].includes(fault)) await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId,
|
|
source: "issue.comment", wakeCommentId: f.wakeId, conversationSessionGeneration: fault === "generation" ? 1 : 0,
|
|
...(fault === "skip" ? { skipIssueComment: true } : {}) } }).where(eq(heartbeatRuns.id, f.runId));
|
|
if (fault === "user") await db.update(heartbeatRuns).set({ responsibleUserId: "another-user" }).where(eq(heartbeatRuns.id, f.runId));
|
|
if (fault === "agent") {
|
|
const other = randomUUID(); await db.insert(agents).values({ id: other, companyId: f.companyId, name: "Other agent" });
|
|
await db.update(issues).set({ conversationAgentId: other, assigneeAgentId: other }).where(eq(issues.id, f.issueId));
|
|
}
|
|
if (fault === "execution") {
|
|
const other = randomUUID(); await db.insert(heartbeatRuns).values({ id: other, companyId: f.companyId, agentId: f.agentId });
|
|
await db.update(issues).set({ executionRunId: other }).where(eq(issues.id, f.issueId));
|
|
}
|
|
if (fault === "waiting") await db.update(issues).set({ executionRunId: null, status: "in_review", conversationState: "waiting" }).where(eq(issues.id, f.issueId));
|
|
if (["turn", "event-run", "session", "channel", "kind"].includes(fault)) {
|
|
const event = structuredClone(f.prpEvent);
|
|
if (fault === "turn") event.turnId = "foreign-turn";
|
|
if (fault === "event-run") event.runId = randomUUID();
|
|
if (fault === "session") event.normalizedSessionId = randomUUID();
|
|
if (fault === "channel") event.payload.channel = "analysis";
|
|
if (fault === "kind") event.payload.kind = "reasoning";
|
|
await db.update(heartbeatRunEvents).set({ payload: { prpEvent: event } }).where(and(eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.seq, 170)));
|
|
}
|
|
if (fault === "digest") await db.update(nativeRunResults).set({ canonicalSha256: "invalid-digest" }).where(eq(nativeRunResults.runId, f.runId));
|
|
if (fault === "partial") await db.update(nativeRunFinalizations).set({ phase: "copying_back" }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const before = await issue(f.issueId);
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
|
|
expect(await comments(f.runId)).toEqual([]); expect(await issue(f.issueId)).toEqual(before);
|
|
},
|
|
);
|
|
it.each([false, true])("preserves a selected reply without replacing or resurrecting it (deleted=%s)", async deleted => {
|
|
const f = await fixture();
|
|
const [saved] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, authorAgentId: f.agentId,
|
|
createdByRunId: f.runId, body: "Already selected", ...(deleted ? { deletedAt: new Date("2020-01-01T00:03:00Z") } : {}) }).returning();
|
|
await db.update(heartbeatRuns).set({ resultJson: { presentationDecision: { commentId: saved.id, commentAction: "create" } } }).where(eq(heartbeatRuns.id, f.runId));
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
|
|
expect(await comments(f.runId)).toEqual([saved]);
|
|
});
|
|
it("never crosses company scope and keeps ordinary internal task response recovery intact", async () => {
|
|
const f = await fixture(false);
|
|
await expect(recoverCommittedNativeChatResponse(db, { ...f.input, companyId: randomUUID() })).resolves.toBe(false);
|
|
const work = nativeResponsePresentationWork(db), entries = await work.load(f.input);
|
|
expect(entries).toHaveLength(1); await expect(work.check(entries[0])).resolves.toBeNull();
|
|
expect(await comments(f.runId)).toEqual([expect.objectContaining({ body: f.reply })]);
|
|
expect((await issue(f.issueId)).conversationAgentId).toBeNull();
|
|
});
|
|
it("recovers a deferred review reply once without resolving the review or rerunning work", async () => {
|
|
const f = await fixture(false, { disposition: "needs_review" });
|
|
const before = await issue(f.issueId);
|
|
const reviews = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, f.issueId));
|
|
expect(before.status).toBe("in_review");
|
|
expect(reviews).toEqual([expect.objectContaining({ status: "pending" })]);
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
const work = nativeResponsePresentationWork(db), entries = await work.load(f.input);
|
|
expect(entries).toHaveLength(1);
|
|
await expect(work.check(entries[0])).resolves.toBeNull();
|
|
expect(await comments(f.runId)).toEqual([expect.objectContaining({ body: f.reply, createdByRunId: f.runId })]);
|
|
expect(await issue(f.issueId)).toEqual({ ...before, updatedAt: expect.any(Date) });
|
|
expect(await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, f.issueId))).toEqual(reviews);
|
|
expect(await work.load(f.input)).toEqual([]);
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
|
|
expect(await comments(f.runId)).toHaveLength(1);
|
|
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.nativeIssueId, f.issueId))).toHaveLength(1);
|
|
});
|
|
it.each(["missing-final", "wrong-turn", "digest", "skip-comment"])(
|
|
"keeps deferred review response recovery closed for %s", async fault => {
|
|
const f = await fixture(false, { disposition: "needs_review" });
|
|
if (fault === "missing-final") await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId));
|
|
if (fault === "wrong-turn") await db.update(heartbeatRunEvents).set({ payload: { prpEvent: { ...f.prpEvent, turnId: "another-turn" } } })
|
|
.where(and(eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.seq, 170)));
|
|
if (fault === "digest") await db.update(nativeRunResults).set({ canonicalSha256: "invalid-digest" }).where(eq(nativeRunResults.runId, f.runId));
|
|
if (fault === "skip-comment") await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, skipIssueComment: true } }).where(eq(heartbeatRuns.id, f.runId));
|
|
const before = await issue(f.issueId);
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
expect(await issue(f.issueId)).toEqual(before);
|
|
},
|
|
);
|
|
it("rolls back a partial reply write without changing the waiting state or consuming recovery", async () => {
|
|
const f = await fixture(), abort = new Error("fixture-commit-aborted");
|
|
const failing = new Proxy(db, { get(target, key) { return key === "transaction"
|
|
? (callback: Parameters<Db["transaction"]>[0]) => db.transaction(async tx => { await callback(tx); throw abort; })
|
|
: Reflect.get(target, key); } });
|
|
await expect(recoverCommittedNativeChatResponse(failing, f.input)).rejects.toBe(abort);
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
expect(await issue(f.issueId)).toMatchObject({ status: "in_progress", conversationState: "active" });
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(true);
|
|
expect(await comments(f.runId)).toHaveLength(1);
|
|
});
|
|
|
|
// A chat turn can end while its task stays held: a resubmitted approval or
|
|
// an execution stage keeps the status, but the turn still answered the
|
|
// person. A restart between the status commit and the live presentation
|
|
// must not lose that answer. The run is saved exactly as the previous
|
|
// release's finalizer saves it, with no presentation recorded yet.
|
|
type Hold = "pending approval" | "approval sent back for revision" | "execution stage" | "own question card"
|
|
| "someone else's confirmation card";
|
|
async function heldTurn(hold: Hold, disposition: "yielded" | "done" = "yielded") {
|
|
return fixture(true, { disposition, beforeFinalize: async ({ companyId, issueId, runId, agentId }) => {
|
|
if (hold.includes("approval")) {
|
|
const [approval] = await db.insert(approvals).values({ companyId, type: "hire_agent",
|
|
status: hold === "pending approval" ? "pending" : "revision_requested", payload: { name: "Researcher" } }).returning();
|
|
await db.insert(issueApprovals).values({ companyId, issueId, approvalId: approval.id });
|
|
}
|
|
if (hold === "execution stage") await db.update(issues).set({ executionState: { status: "pending" } }).where(eq(issues.id, issueId));
|
|
if (hold === "own question card") {
|
|
await db.insert(issueThreadInteractions).values({ companyId, issueId, kind: "ask_user_questions", status: "pending",
|
|
sourceRunId: runId, createdByAgentId: agentId, payload: { version: 1, questions: [{ id: "model", prompt: "Which model?",
|
|
selectionMode: "single", required: true, options: [{ id: "flash", label: "Flash" }] }] } });
|
|
}
|
|
// The board addressed this card to another person; the run did not create it.
|
|
if (hold === "someone else's confirmation card") {
|
|
await db.insert(issueThreadInteractions).values({ companyId, issueId, kind: "request_confirmation", status: "pending",
|
|
continuationPolicy: "none", title: "Riley's website proposal", requestedResolverPolicy: "human_only",
|
|
effectiveResolverPolicy: "human_only", resolverPolicyProvenance: "explicit", addresseeUserId: "riley",
|
|
createdByUserId: "local-board", payload: { version: 1, prompt: "Riley: approve creating the website implementation task?" } });
|
|
}
|
|
} });
|
|
}
|
|
it.each([
|
|
["pending approval", "yielded"], ["approval sent back for revision", "yielded"], ["execution stage", "yielded"],
|
|
["pending approval", "done"], ["someone else's confirmation card", "yielded"],
|
|
] as const)(
|
|
"recovers a held conversation turn's answer exactly once after a restart: %s (%s)", async (hold, disposition) => {
|
|
const f = await heldTurn(hold, disposition);
|
|
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, f.runId));
|
|
expect(decision.reasonCode).not.toBe("conversation_turn_finished");
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].resultJson)
|
|
.not.toHaveProperty("presentationDecision");
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
const work = nativeResponsePresentationWork(db), entries = await work.load(f.input);
|
|
expect(entries).toEqual([expect.objectContaining({ id: f.runId })]);
|
|
await expect(work.check(entries[0])).resolves.toBeNull();
|
|
expect(await comments(f.runId)).toEqual([expect.objectContaining({ body: f.reply, authorAgentId: f.agentId, createdByRunId: f.runId })]);
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].resultJson).toMatchObject({
|
|
presentationDecision: { chosenSource: "final_agent_message", commentAction: "create", commentId: expect.any(String) } });
|
|
expect(await issue(f.issueId)).toMatchObject({ conversationState: "waiting" });
|
|
expect(await work.load(f.input)).toEqual([]);
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
|
|
expect(await comments(f.runId)).toHaveLength(1);
|
|
},
|
|
);
|
|
it("does not repost a held conversation turn's answer that the live path already presented", async () => {
|
|
const f = await heldTurn("pending approval");
|
|
const [saved] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, authorAgentId: f.agentId,
|
|
createdByRunId: f.runId, body: f.reply }).returning();
|
|
await db.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({
|
|
presentationDecision: { chosenSource: "final_agent_message", commentAction: "create", commentId: saved.id, reasonCodes: [] } })}::jsonb` })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
expect(await nativeResponsePresentationWork(db).load(f.input)).toEqual([]);
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
|
|
expect(await comments(f.runId)).toEqual([saved]);
|
|
});
|
|
it("keeps a held turn's presentation that the previous release's live path already recorded", async () => {
|
|
// Before the live answered rule, a held yielded turn was recorded as a
|
|
// control-plane wait. That decision stands; recovery never re-presents it.
|
|
const f = await heldTurn("pending approval");
|
|
await db.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({
|
|
presentationDecision: { chosenSource: "none", commentAction: "none", reasonCodes: ["yielded_control_plane_wait"] } })}::jsonb` })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
expect(await nativeResponsePresentationWork(db).load(f.input)).toEqual([]);
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
});
|
|
it("settles a held conversation turn presented by its own question card without posting its prose", async () => {
|
|
const f = await heldTurn("own question card");
|
|
const cards = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, f.issueId));
|
|
expect(cards).toEqual([expect.objectContaining({ kind: "ask_user_questions", status: "pending", sourceRunId: f.runId })]);
|
|
const work = nativeResponsePresentationWork(db), entries = await work.load(f.input);
|
|
expect(entries).toEqual([expect.objectContaining({ id: f.runId })]);
|
|
await expect(work.check(entries[0])).resolves.toBeNull();
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].resultJson).toMatchObject({
|
|
presentationDecision: { chosenSource: "none", commentAction: "none" } });
|
|
// Settled like the live path: the card is the turn's response.
|
|
expect(await issue(f.issueId)).toMatchObject({ conversationState: "waiting" });
|
|
expect(await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, f.issueId))).toEqual(cards);
|
|
expect(await work.load(f.input)).toEqual([]);
|
|
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
|
|
expect(await comments(f.runId)).toEqual([]);
|
|
});
|
|
});
|