Files
PaperClipAI/server/src/services/heartbeat/queue.test.ts
T
Nicky LeachandPaperclip 87ef7097b6 refactor(wake-queue): own blocked admission deferral (#22332)
## Thinking Path

> - Paperclip runs agents in response to work on a task.
> - A person can leave a comment while execution recovery holds that
task.
> - Wake-queue checks the recovery gate under the task lock. Heartbeat
used to choose how to save the blocked input.
> - The deferred-work worker later retries the saved receipt by ID.
> - This change gives wake-queue responsibility for saving the input
before it confirms deferral.

## Linked Issues or Issue Description

**Current behavior**

The gate and the write that makes a blocked wake recoverable have
different owners. Heartbeat also reports an admitted deferral when an
update to an existing receipt matches no row.

**Proposed behavior**

Let the recovery admission operation save the first receipt or update
the existing one in the same transaction. Confirm admission only when
the write succeeds. Keep replaceable automatic signals on their existing
diagnostic path.

**Reason and benefit**

Changes to recovery admission can now be checked against one rule. A
blocked comment must have a durable receipt and remain recoverable
across delivery attempts. Related changes include
[#22292](https://github.com/paperclipai/paperclip/pull/22292) and
[#22290](https://github.com/paperclipai/paperclip/pull/22290).

## What Changed

- Added a transaction-bound deferral operation to wake-queue recovery.
It handles existing receipts, saved input, and replaceable system
signals.
- Scoped retry updates to the company, issue, agent, deferred status,
and receipt ID. A stale receipt cannot confirm admission or cause a new
receipt to be inserted.
- Kept case refusals and issue state callbacks in heartbeat. A saved
comment uses the existing delivery worker and post-commit effects.

## Verification

- Retried a saved comment by receipt ID through the real heartbeat and
run-control entry points. While the hold remained, no run started. After
the hold cleared, repeated retry delivered the comment to exactly one
run.
- Forced a rollback after the wake-queue Postgres adapter saved a
receipt. The test observed the receipt within the transaction and
confirmed its absence after rollback. Tested missing receipts and
receipts for another issue or agent. The admission application test
covers error propagation and post-commit effects when the transaction
fails.
- The test-only revision `9f443f9e6b` passed 9 focused PostgreSQL and
recovery checks and all 15 issue-wake admission application tests.
- The preceding production revision `eb37328472` passed the server
TypeScript check, the module boundary check, and 498 tests across five
queue and recovery suites. Twelve pairs of ordinary comments issued the
same SQL before and after that production change (42 statements for a
fresh wake and 35 for a join).

- Both updated test files passed in full (249 tests). [CI attempt
3](https://github.com/paperclipai/paperclip/actions/runs/44572369812/attempts/3)
passed on `9f443f9e6b` after retrying interrupted jobs. [Standard
Greptile
review](https://github.com/paperclipai/paperclip/pull/22332#issuecomment-7014835241)
scored the same commit 5/5 with no open findings.

## Risks

The affected path handles blocked admissions that may race a retry or a
reassignment. Existing recovery gates, saved comment IDs, receipt
identity, and notifications must survive that race. This change has no
migration or new worker. The healthy admission path is expected to issue
the same SQL.

## Model Used

OpenAI Codex with GPT-6.2 Sol for implementation and verification. Codex
directed and reviewed the change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-19 10:16:52 -08:00

1638 lines
112 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { contextCompactionService } from "../context-compaction.js";
import { claimNativeSessionResumptions } from "../native-runtime/native-finalization-reconciler.js";
import type { Db } from "@paperclipai/db";
import { authUsers, agents, agentTaskSessions, agentWakeupRequests, budgetIncidents, budgetPolicies, caseHandoffs, cases, chatCompletionDeliveries, companies, companyMemberships, costEvents, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, heartbeatRuns, issueComments, issueRecoveryActions, issueThreadInteractions, issues, nativeRunFinalizations, routines, routineRuns } from "@paperclipai/db";
import { eq, inArray, sql } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { createFreshIssueRunStore, createPostgresWakeQueueAdapter, createWakeAdmissionWriter } from "../../modules/wake-queue/adapters/postgres.js";
import { createReleaseIssueExecution } from "../../modules/wake-queue/application/use-cases.js";
import { closeProcessRunAdmission, resetProcessRunAdmissionForTests } from "./run-admission.js";
import { createHeartbeatQueue, DEFERRED_WAKE_CONTEXT_KEY, notifyOtherAgentsQueuedOnSettledIssue, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake, type HeartbeatQueueDependencies } from "./queue.js";
import { createWakeQueue } from "../../modules/wake-queue/index.js";
import { enrichWakeContextSnapshot, mergeCoalescedContextSnapshot } from "./run-preparation.js";
import { withQueuedCommentIdsInWakePayload, withQueuedCommentIdsInRunContext } from "../issue-queued-comment-queue.js";
import { settleConversationTurn } from "../agent-conversations.js";
import { chatCompletionDeliveryService, recordChatCompletion, recordChatHandoff } from "../chat-completion-delivery.js";
import { createDurableChatWakeupRequest } from "../durable-chat-wakeup.js";
import { instanceSettingsService } from "../instance-settings.js";
import { createHeartbeatRunControl } from "./run-control.js";
import { callbacks as runControlCallbacks } from "./__tests__/run-control-fixtures.js";
import { createHeartbeatQueueTestDependencies as callbacks } from "../../__tests__/helpers/heartbeat-queue.js";
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../delivery-work-notifications.js";
function guardedDatabase() {
const access = vi.fn(() => { throw new Error("Unexpected queue database access"); });
return { db: new Proxy({}, { get: access }) as Db, access };
}
describe("heartbeat queue module boundary", () => {
it("constructs without querying, dispatching, or registering background work", () => {
const database = guardedDatabase();
const deps = callbacks(database.db);
createHeartbeatQueue(database.db, deps);
expect(database.access).not.toHaveBeenCalled();
for (const callback of Object.values(deps)) if (vi.isMockFunction(callback)) expect(callback).not.toHaveBeenCalled();
expect(deps.activeRunExecutionPromises.size).toBe(0);
expect(deps.activeWakeupPromises.size).toBe(0);
});
it.each(["startNextQueuedRunForAgent", "resumeQueuedRuns"] as const)("honors suppression before %s reads or resumes work", async (operation) => {
const database = guardedDatabase();
const deps = callbacks(database.db);
deps.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" });
const queue = createHeartbeatQueue(database.db, deps);
await queue[operation]("agent");
expect(database.access).not.toHaveBeenCalled();
expect(deps.getAgent).not.toHaveBeenCalled();
expect(deps.resumeExecutionWaitComments).not.toHaveBeenCalled();
expect(deps.executeRun).not.toHaveBeenCalled();
});
it.each(["startNextQueuedRunForAgent", "resumeQueuedRuns"] as const)("starts no run once shutdown closed run admission (%s)", async (operation) => {
const database = guardedDatabase();
const deps = callbacks(database.db);
deps.isRunAdmissionClosed = () => true;
const queue = createHeartbeatQueue(database.db, deps);
await queue[operation]("agent");
expect(database.access).not.toHaveBeenCalled();
expect(deps.getAgent).not.toHaveBeenCalled();
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("claims no queued run once run admission is closed, whatever path asks (draining or yielding server)", async () => {
// Release-swap wave: an old server started a delivered continuation 3 minutes after traffic
// had moved to its dispatching replacement, through executeRun of a queued run, which no gate covered.
const database = guardedDatabase();
const deps = callbacks(database.db);
deps.isRunAdmissionClosed = () => true;
const queue = createHeartbeatQueue(database.db, deps);
const queued = { id: "run-1", status: "queued" } as Parameters<typeof queue.claimQueuedRun>[0];
expect(await queue.claimQueuedRun(queued)).toBeNull();
expect(database.access).not.toHaveBeenCalled();
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("honors the process-wide run admission gate in every heartbeat instance", async () => {
const database = guardedDatabase();
const deps = callbacks(database.db);
closeProcessRunAdmission();
try {
await createHeartbeatQueue(database.db, deps).startNextQueuedRunForAgent("agent");
} finally {
resetProcessRunAdmissionForTests();
}
expect(database.access).not.toHaveBeenCalled();
expect(deps.getAgent).not.toHaveBeenCalled();
});
it("tracks pending wakes across instances and clears each rejected wake", async () => {
const database = guardedDatabase();
const deps = callbacks(database.db);
let rejectFirst!: (reason: Error) => void;
let rejectSecond!: (reason: Error) => void;
deps.getAgent.mockImplementationOnce(() => new Promise((_resolve, reject) => { rejectFirst = reject; }));
deps.getAgent.mockImplementationOnce(() => new Promise((_resolve, reject) => { rejectSecond = reject; }));
const first = createHeartbeatQueue(database.db, deps).trackWakeup("first");
const second = createHeartbeatQueue(database.db, deps).trackWakeup("second");
const firstCheck = expect(first).rejects.toThrow("first lookup failed");
const secondCheck = expect(second).rejects.toThrow("second lookup failed");
expect(deps.activeWakeupPromises).toEqual(new Set([first, second]));
rejectFirst(new Error("first lookup failed"));
await firstCheck;
await Promise.resolve();
expect(deps.activeWakeupPromises).toEqual(new Set([second]));
rejectSecond(new Error("second lookup failed"));
await secondCheck;
await Promise.resolve();
expect(deps.activeWakeupPromises.size).toBe(0);
expect(deps.activeRunExecutionPromises.size).toBe(0);
expect(database.access).not.toHaveBeenCalled();
});
it("uses the same policy limits for dispatch and execution", () => {
const database = guardedDatabase();
const queue = createHeartbeatQueue(database.db, callbacks(database.db));
const agent = { adapterType: "paperclip_runner", adapterConfig: { provider: "openai_dot" }, runtimeConfig: { heartbeat: { maxConcurrentRuns: 20, maxDailyRuns: 3.8 } } } as unknown as typeof agents.$inferSelect;
expect(queue.parseHeartbeatPolicy(agent)).toMatchObject({ maxConcurrentRuns: 1, maxDailyRuns: 3 });
expect(queue.parseHeartbeatPolicy({ ...agent, adapterConfig: { provider: "codex" } })).toMatchObject({ maxConcurrentRuns: 20, maxDailyRuns: 3 });
expect(database.access).not.toHaveBeenCalled();
});
});
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_QUEUE_TEST_DATABASE_URL");
const support = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
if (!support.supported) console.warn(`Skipping queue module database tests: ${support.reason}`);
describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: Db;
beforeAll(async () => {
if (!external) database = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-queue-module");
db = createDb(external ?? database!.connectionString, { maxConnections: 4 });
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => {
await db?.$client.end();
await database?.cleanup();
});
beforeEach(async () => {
await db.execute(sql`truncate table companies restart identity cascade`);
});
it.each([
{ provider: "codex" }, { provider: "acpx", acpxAgent: "antigravity" }, { provider: "muse" },
])("rejects direct taskless timer wakes for $provider before a run or request can be persisted", async adapterConfig => {
const [company] = await db.insert(companies).values({ name: "Timer scope", issuePrefix: "TS" }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Ready for a task", status: "idle",
adapterType: "paperclip_runner", adapterConfig, runtimeConfig: { heartbeat: { enabled: true, intervalSec: 60 } } }).returning();
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(agent);
const queue = createHeartbeatQueue(db, deps);
expect(await queue.enqueueWakeup(agent.id, { source: "timer", reason: "heartbeat_timer", requestedByActorType: "system" })).toBeNull();
expect(await db.select().from(heartbeatRuns)).toHaveLength(0);
expect(await db.select().from(agentWakeupRequests)).toHaveLength(0);
expect((await db.select().from(agents).where(eq(agents.id, agent.id)))[0])
.toMatchObject({ status: "idle", lastHeartbeatAt: null });
expect(deps.executeRun).not.toHaveBeenCalled();
expect(deps.budgets.getInvocationBlock).not.toHaveBeenCalled();
});
it.each(["ordinary recovery", "mismatched compaction"] as const)(
"does not admit %s after the conversation settles between precheck and admission", async cause => {
const [user] = await db.insert(authUsers).values({
id: "queue-conversation-race-user", name: "Conversation owner", email: "queue-race@paperclip.test",
createdAt: new Date(), updatedAt: new Date(),
}).onConflictDoUpdate({ target: authUsers.id, set: { name: "Conversation owner" } }).returning();
const [company] = await db.insert(companies).values({ name: "Conversation race", issuePrefix: "CR" }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Chat agent",
status: "idle", adapterType: "process", adapterConfig: {} }).returning();
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Answered conversation",
assigneeAgentId: agent.id, responsibleUserId: user.id, conversationAgentId: agent.id,
conversationUserId: user.id, conversationState: "active", status: "in_progress" }).returning();
const [completed] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
status: "succeeded", finishedAt: new Date(), invocationSource: "on_demand", triggerDetail: "manual",
contextSnapshot: { issueId: issue.id, conversationSessionGeneration: issue.conversationSessionGeneration },
}).returning();
await db.insert(issueComments).values({ companyId: company.id, issueId: issue.id, authorAgentId: agent.id,
createdByRunId: completed.id, origin: "comment", body: "The requested answer." });
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(agent);
deps.instanceSettings.getExperimental.mockResolvedValue({ enableAgentChat: true } as Awaited<ReturnType<typeof deps.instanceSettings.getExperimental>>);
// Admit through the normal transaction, but hold dispatch so no adapter runs.
deps.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" });
deps.getSchedulingSuppression.mockResolvedValueOnce({ suppressed: false, reason: null });
const preflight = deps.wakeQueue.preflightIssueWake;
let reached!: () => void;
let release!: () => void;
const atBarrier = new Promise<void>(resolve => { reached = resolve; });
const admissionMayContinue = new Promise<void>(resolve => { release = resolve; });
deps.budgets.getInvocationBlock.mockImplementationOnce(async () => {
reached();
await admissionMayContinue;
return null;
});
const requestId = randomUUID();
const compaction = cause === "mismatched compaction";
const pending = createHeartbeatQueue(db, deps).enqueueWakeup(agent.id, {
source: compaction ? "on_demand" : "automation", triggerDetail: compaction ? "manual" : "system",
reason: compaction ? "context_compaction" : "issue.continuation_recovery",
requestedByActorType: compaction ? "user" : "system", requestedByActorId: compaction ? user.id : undefined,
...(compaction ? { manualUserWake: true, allowRunCoalescing: false } : {}),
payload: { issueId: issue.id, ...(compaction ? { intent: "context_compaction", requestId: `foreign-${requestId}` } : {}) },
contextSnapshot: { issueId: issue.id, retryOfRunId: completed.id, source: "issue.continuation_recovery",
...(compaction ? { contextCompactionRequestId: requestId } : {}) },
});
try {
await atBarrier;
expect(preflight).toHaveBeenCalledOnce();
await expect(preflight.mock.results[0]?.value).resolves.toMatchObject({ kind: "eligible" });
expect(await settleConversationTurn(db, completed)).toBe(true);
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0])
.toMatchObject({ status: "in_review", conversationState: "waiting" });
} finally { release(); }
const admitted = await pending;
expect(deps.executeRun).not.toHaveBeenCalled();
expect(admitted).toBeNull();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agent.id))).map(run => run.id))
.toEqual([completed.id]);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, agent.id)))
.toHaveLength(0);
});
async function waitingConversationFixture() {
const [user] = await db.insert(authUsers).values({ id: "queue-conversation-owner", name: "Owner",
email: "queue-owner@paperclip.test", createdAt: new Date(), updatedAt: new Date() })
.onConflictDoUpdate({ target: authUsers.id, set: { name: "Owner" } }).returning();
const [company] = await db.insert(companies).values({ name: "Waiting conversation", issuePrefix: "WC" }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Chat agent", status: "idle", adapterType: "process" }).returning();
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Waiting for input",
assigneeAgentId: agent.id, responsibleUserId: user.id, conversationAgentId: agent.id,
conversationUserId: user.id, conversationState: "waiting", status: "in_review" }).returning();
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(agent);
deps.instanceSettings.getExperimental.mockResolvedValue({ enableAgentChat: true } as Awaited<ReturnType<typeof deps.instanceSettings.getExperimental>>);
deps.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" });
deps.getSchedulingSuppression.mockResolvedValueOnce({ suppressed: false, reason: null });
await instanceSettingsService(db).updateExperimental({ enableAgentChat: true });
return { user, company, agent, issue, deps, queue: createHeartbeatQueue(db, deps) };
}
async function waitingCompactionFixture() {
const f = await waitingConversationFixture();
const [agent] = await db.update(agents).set({ adapterType: "paperclip_runner", adapterConfig: { provider: "codex" } })
.where(eq(agents.id, f.agent.id)).returning();
if (!agent) throw new Error("Missing compaction agent");
f.deps.getAgent.mockResolvedValue(agent);
const service = contextCompactionService(db, { wakeup: f.queue.enqueueWakeup });
return { ...f, agent, service };
}
it.each([
["fresh", "authorization contention"], ["busy", "authorization contention"],
["fresh", "wrong locked issue"], ["busy", "wrong locked issue"],
] as const)("rolls back a %s chat admission on %s without publishing", async (kind, failure) => {
const { user, company, agent, issue, deps, queue } = await waitingConversationFixture();
await db.update(issues).set({ conversationAgentId: null, conversationUserId: null, conversationState: null, status: "todo" })
.where(eq(issues.id, issue.id));
if (kind === "busy") {
const [active] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
status: "running", invocationSource: "assignment", contextSnapshot: { issueId: issue.id } }).returning();
await db.update(issues).set({ executionRunId: active.id, executionAgentNameKey: agent.name, executionLockedAt: new Date() })
.where(eq(issues.id, issue.id));
deps.liveRunExecutions.has.mockImplementation((id) => id === active.id);
}
const [comment] = await db.insert(issueComments).values({ companyId: company.id, issueId: issue.id,
authorType: "user", authorUserId: user.id, body: "Please continue" }).returning();
const id = randomUUID();
const authorize = vi.fn(async () => {});
const onAdmitted = vi.fn(async (tx: Db, { runId }: { runId: string | null }) => {
const [receipt] = await tx.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, id));
expect(receipt).toMatchObject({ status: kind === "fresh" ? "queued" : "deferred_issue_execution", runId });
expect(Boolean(runId)).toBe(kind === "fresh");
throw Object.assign(new Error("authorization lock was held"), { code: "55P03" });
});
const request = createDurableChatWakeupRequest({ id, companyId: company.id, issueId: issue.id, agentId: agent.id,
commentId: comment.id, requestedByActorType: "user", requestedByActorId: user.id, requestedAt: new Date(), authorize, onAdmitted });
const [otherIssue] = failure === "wrong locked issue" ? await db.insert(issues).values({
companyId: company.id, assigneeAgentId: agent.id, title: "Another task",
}).returning() : [];
const publishIssueActivity = vi.fn();
const admit = deps.wakeQueue.admitIssueWake.getMockImplementation()!;
deps.wakeQueue.admitIssueWake.mockImplementation((target, evaluate, effects) => admit(target,
(tx, admission) => evaluate(tx, otherIssue ? {
...admission,
async prepareRecovery(input) {
const recovery = await admission.prepareRecovery(input);
return {
...recovery,
async checkBusyAdmission() {
const busy = await recovery.checkBusyAdmission();
if (busy.kind !== "allowed") return busy;
return {
...busy,
admitBusy: (input) => busy.admitBusy({ ...input, issueId: otherIssue.id,
contextSnapshot: { ...input.contextSnapshot, issueId: otherIssue.id } }),
async checkFreshAdmission() {
const fresh = await busy.checkFreshAdmission();
if (fresh.kind !== "allowed") return fresh;
return {
...fresh,
admitFresh: (input) => fresh.admitFresh({ ...input,
issue: { ...input.issue, id: otherIssue.id }, contextSnapshot: { ...input.contextSnapshot, issueId: otherIssue.id } }),
};
},
};
},
};
},
} : admission), { ...effects, publishIssueActivity }));
const wake = queue.enqueueWakeup(agent.id, {
source: "assignment", triggerDetail: "system", reason: "External chat message received",
requestedByActorType: "user", requestedByActorId: user.id, allowRunCoalescing: false, durableChatRequest: request,
payload: { issueId: issue.id, commentId: comment.id, mutation: "chat_message_received" },
});
if (otherIssue) await expect(wake).rejects.toThrow(/does not match the locked/);
else await expect(wake).resolves.toBeNull();
expect(authorize).toHaveBeenCalledTimes(1);
expect(onAdmitted).toHaveBeenCalledTimes(otherIssue ? 0 : 1);
expect(publishIssueActivity).not.toHaveBeenCalled();
expect(deps.executeRun).not.toHaveBeenCalled();
expect(await db.select().from(agentWakeupRequests)).toEqual([]);
expect(await db.select().from(heartbeatRuns)).toHaveLength(kind === "fresh" ? 0 : 1);
});
it("delivers a comment saved behind a concurrent recovery hold to one run on repeated retry", async () => {
const { user, company, agent, issue, deps, queue } = await waitingConversationFixture();
await db.update(issues).set({ conversationAgentId: null, conversationUserId: null, conversationState: null, status: "todo" })
.where(eq(issues.id, issue.id));
const [comment] = await db.insert(issueComments).values({
companyId: company.id, issueId: issue.id, authorType: "user", authorUserId: user.id, body: "Please continue.",
}).returning();
const actionId = randomUUID();
function enqueue() {
return queue.enqueueWakeup(agent.id, {
source: "automation", triggerDetail: "system", reason: "issue_commented",
requestedByActorType: "user", requestedByActorId: user.id, payload: { issueId: issue.id, commentId: comment.id },
});
}
let pending: ReturnType<typeof enqueue> | undefined;
try {
await db.transaction(async writer => {
const [{ pid }] = await writer.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
await writer.insert(issueRecoveryActions).values({
id: actionId, companyId: company.id, sourceIssueId: issue.id, kind: "active_run_watchdog",
cause: "legacy_execution_requires_reconciliation", fingerprint: actionId, status: "resolved",
evidence: { automaticRecovery: { replay: "blocked" } }, nextAction: "Inspect the stopped run.",
});
pending = enqueue();
await vi.waitFor(async () => {
const [waiter] = await db.execute<{ waiting: boolean }>(sql`select exists (
select 1 from pg_stat_activity
where wait_event_type = 'Lock' and ${pid} = any(pg_blocking_pids(pg_stat_activity.pid))
and query ilike '%for update%'
) as waiting`);
expect(waiter.waiting).toBe(true);
}, { timeout: 5_000, interval: 20 });
});
} finally {
await pending;
}
await expect(pending).resolves.toBeNull();
expect(await db.select().from(heartbeatRuns)).toEqual([]);
const receipts = await db.select().from(agentWakeupRequests);
expect(receipts).toEqual([
expect.objectContaining({ status: "deferred_issue_execution", runId: null,
payload: expect.objectContaining({ issueId: issue.id, commentId: comment.id,
executionWait: expect.objectContaining({ recoveryActionId: actionId }) }) }),
]);
const [receipt] = receipts;
const controlDeps = runControlCallbacks(db);
vi.mocked(controlDeps.enqueueWakeup).mockImplementation(queue.enqueueWakeup);
const control = createHeartbeatRunControl(db, controlDeps);
deps.getSchedulingSuppression.mockResolvedValueOnce({ suppressed: false, reason: null });
await control.resumeSavedLegacyComments(company.id, receipt.id);
expect(await db.select().from(heartbeatRuns)).toEqual([]);
await db.update(issueRecoveryActions).set({ evidence: {} }).where(eq(issueRecoveryActions.id, actionId));
deps.getSchedulingSuppression.mockResolvedValueOnce({ suppressed: false, reason: null });
await control.resumeSavedLegacyComments(company.id, receipt.id);
await control.resumeSavedLegacyComments(company.id, receipt.id);
expect(await db.select().from(heartbeatRuns)).toEqual([expect.objectContaining({
contextSnapshot: expect.objectContaining({ wakeCommentIds: [comment.id] }),
})]);
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("uses issue changes written by durable chat authorization before admission", async () => {
const { user, company, agent, issue, queue } = await waitingConversationFixture();
await db.update(issues).set({ conversationAgentId: null, conversationUserId: null, conversationState: null, status: "todo" })
.where(eq(issues.id, issue.id));
const [comment] = await db.insert(issueComments).values({ companyId: company.id, issueId: issue.id,
authorType: "user", authorUserId: user.id, body: "Please continue" }).returning();
const requestId = randomUUID();
const request = createDurableChatWakeupRequest({
id: requestId, companyId: company.id, issueId: issue.id, agentId: agent.id, commentId: comment.id,
requestedByActorType: "user", requestedByActorId: user.id, requestedAt: new Date(),
authorize: async (tx) => {
await tx.update(issues).set({ status: "blocked" }).where(eq(issues.id, issue.id));
},
});
await expect(queue.enqueueWakeup(agent.id, {
source: "assignment", triggerDetail: "system", reason: "External chat message received",
requestedByActorType: "user", requestedByActorId: user.id, durableChatRequest: request,
payload: { issueId: issue.id, commentId: comment.id, mutation: "chat_message_received" },
issueStateGuard: { assigneeAgentId: agent.id, statuses: ["todo"] },
})).resolves.toBeNull();
const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, requestId));
expect(receipt).toMatchObject({ status: "skipped", reason: "issue_state_guard_mismatch",
payload: { heartbeatSkip: { actualStatus: "blocked" } } });
expect(await db.select().from(heartbeatRuns)).toEqual([]);
});
it("admits the owner's context compaction maintenance request while its conversation waits", async () => {
const f = await waitingCompactionFixture();
const accepted = await f.service.request({ companyId: f.company.id, issue: f.issue, userId: f.user.id });
const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agent.id));
expect(runs).toHaveLength(1);
expect(runs[0]).toMatchObject({ id: accepted.runId, status: "queued", contextSnapshot: {
issueId: f.issue.id, contextCompactionRequestId: accepted.requestId, skipIssueComment: true,
} });
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agent.id)))
.toHaveLength(1);
expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0])
.toMatchObject({ status: "in_review", conversationState: "waiting", assigneeAgentId: f.agent.id });
expect(await db.select().from(issueComments).where(eq(issueComments.issueId, f.issue.id))).toHaveLength(0);
expect(f.deps.executeRun).not.toHaveBeenCalled();
});
it("admits only one of two simultaneous waiting-conversation compaction requests", async () => {
const f = await waitingCompactionFixture();
// Both service requests must finish their real availability read before
// either enters queue admission. Dispatch remains suppressed, not mocked.
// task_drain explicitly retains admission (queue.ts's admission guard
// excludes it), while startNextQueuedRunForAgent stops before any claim.
// Do not tie two admissions to a call-count-sensitive suppression sequence.
f.deps.getSchedulingSuppression.mockReset().mockResolvedValue({ suppressed: true, reason: "task_drain" });
const admission = createWakeQueue(db, {
resolveResponsibleUserId: async () => f.user.id,
getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }),
resolveSessionBeforeForWakeup: async () => null,
wakeAdmissionHelpers: { filterZombieCoalesceTarget: f.deps.filterZombieCoalesceTarget,
mergeCoalescedContextSnapshot, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake },
recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined },
});
f.deps.wakeQueue.admitIssueWake.mockImplementation(admission.admitIssueWake);
let releaseAvailability!: () => void;
const bothAvailable = new Promise<void>(resolve => { releaseAvailability = resolve; });
let arrivals = 0;
const service = contextCompactionService(db, { wakeup: async (...args) => {
if (++arrivals === 2) releaseAvailability();
await bothAvailable;
return f.queue.enqueueWakeup(...args);
} });
const results = await Promise.allSettled([0, 1].map(() => service.request({
companyId: f.company.id, issue: f.issue, userId: f.user.id,
})));
// Inspect both forms of durable execution authority before asserting.
// A saved deferred maintenance request can replay after the first run;
// counting only currently admitted runs would miss that extra work.
const maintenanceRuns = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agent.id));
const receipts = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agent.id));
const maintenanceReceipts = receipts.filter(row => row.status !== "skipped"
&& row.payload?.intent === "context_compaction" && row.payload.issueId === f.issue.id);
expect(arrivals).toBe(2);
expect(maintenanceReceipts.map(row => ({ id: row.id, status: row.status, runId: row.runId })),
"one durable compaction authority including deferred").toHaveLength(1);
expect(maintenanceRuns.map(run => run.id)).toHaveLength(1);
expect(maintenanceRuns[0]).toMatchObject({ companyId: f.company.id, contextSnapshot: { issueId: f.issue.id } });
expect(results.filter(result => result.status === "fulfilled")).toHaveLength(1);
const rejected = results.filter(result => result.status === "rejected");
expect(rejected).toHaveLength(1);
if (rejected[0]?.status === "rejected") expect(rejected[0].reason).toMatchObject({ code: "already_pending" });
expect(receipts.filter(row => row.status === "deferred_issue_execution")).toEqual([]);
expect(receipts.filter(row => row.reason === "context_compaction" && row.status !== "skipped")).toHaveLength(1);
expect(f.deps.executeRun).not.toHaveBeenCalled();
});
it("preserves an existing deferred compaction authority until cancellation allows a new request", async () => {
const f = await waitingCompactionFixture();
f.deps.getSchedulingSuppression.mockReset().mockResolvedValue({ suppressed: true, reason: "task_drain" });
// Reproduce a pre-existing receipt through the actual normal deferred
// writer. In particular, its informational reason is NOT context_compaction.
// The service builds the real request; no provider/result rows are seeded.
const historical = contextCompactionService(db, { wakeup: async (agentId, options) => {
const source = options.source ?? "on_demand";
const triggerDetail = options.triggerDetail ?? null;
const context = enrichWakeContextSnapshot({ contextSnapshot: { ...options.contextSnapshot },
reason: options.reason ?? null, source, triggerDetail, payload: options.payload ?? null });
await db.transaction(async tx => {
await createWakeAdmissionWriter(tx, f.company.id).insertNewDeferredWake({
companyId: f.company.id, agentId, source, triggerDetail,
payload: { ...options.payload, manualUserWake: true, [DEFERRED_WAKE_CONTEXT_KEY]: context.contextSnapshot },
requestedByActorType: options.requestedByActorType ?? null,
requestedByActorId: options.requestedByActorId ?? null, idempotencyKey: options.idempotencyKey ?? null,
});
});
return null;
} });
await expect(historical.request({ companyId: f.company.id, issue: f.issue, userId: f.user.id }))
.rejects.toMatchObject({ code: "wake_skipped" });
const [pending] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agent.id));
expect(pending).toMatchObject({ reason: "issue_execution_deferred", status: "deferred_issue_execution", runId: null,
payload: { issueId: f.issue.id, intent: "context_compaction" } });
await expect(f.service.request({ companyId: f.company.id, issue: f.issue, userId: f.user.id }))
.rejects.toMatchObject({ code: "already_pending" });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agent.id))).toEqual([]);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agent.id))).toEqual([pending]);
// This never-dispatched fixture authority is now cancelled, not completed.
// Nonpending history must not permanently block a fresh normal request.
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() })
.where(eq(agentWakeupRequests.id, pending.id));
const accepted = await f.service.request({ companyId: f.company.id, issue: f.issue, userId: f.user.id });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agent.id)))
.toEqual([expect.objectContaining({ id: accepted.runId, status: "queued" })]);
const receipts = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agent.id));
expect(receipts).toHaveLength(2);
expect(receipts.find(row => row.id === pending.id)).toMatchObject({ status: "cancelled", runId: null });
expect(receipts.find(row => row.runId === accepted.runId)).toMatchObject({ status: "queued" });
expect(f.deps.executeRun).not.toHaveBeenCalled();
});
it.each([
{ actor: "missing actor", userId: "", message: "Manual wake requires an authenticated user" },
{ actor: "foreign owner", userId: "another-conversation-owner", message: "Only the conversation owner can start a chat run" },
])("refuses waiting-conversation compaction from a $actor", async ({ userId, message }) => {
const f = await waitingCompactionFixture();
await expect(f.service.request({ companyId: f.company.id, issue: f.issue, userId })).rejects.toMatchObject({ status: 403, message });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agent.id))).toHaveLength(0);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agent.id))).toHaveLength(0);
expect(f.deps.executeRun).not.toHaveBeenCalled();
});
it.each(["missing request", "malformed request", "mismatched request"] as const)(
"rejects a waiting-conversation compaction envelope with a %s", async kind => {
const f = await waitingCompactionFixture();
const service = contextCompactionService(db, { wakeup: (agentId, options) => {
const contextSnapshot = { ...options.contextSnapshot };
const payload = { ...options.payload };
if (kind === "missing request") delete contextSnapshot.contextCompactionRequestId;
if (kind === "malformed request") payload.requestId = { unexpected: true };
if (kind === "mismatched request") payload.requestId = randomUUID();
return f.queue.enqueueWakeup(agentId, { ...options, contextSnapshot, payload });
} });
await expect(service.request({ companyId: f.company.id, issue: f.issue, userId: f.user.id }))
.rejects.toMatchObject({ code: "wake_skipped" });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agent.id))).toHaveLength(0);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agent.id))).toHaveLength(0);
expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0])
.toMatchObject({ status: "in_review", conversationState: "waiting" });
expect(f.deps.executeRun).not.toHaveBeenCalled();
});
it("does not turn an ordinary waiting-conversation wake into compaction authority", async () => {
const f = await waitingCompactionFixture();
expect(await f.queue.enqueueWakeup(f.agent.id, { source: "on_demand", triggerDetail: "manual",
reason: "issue.continuation_recovery", requestedByActorType: "user", requestedByActorId: f.user.id,
manualUserWake: true, allowRunCoalescing: false, payload: { issueId: f.issue.id },
contextSnapshot: { issueId: f.issue.id } })).toBeNull();
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agent.id))).toHaveLength(0);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agent.id))).toHaveLength(0);
expect(f.deps.executeRun).not.toHaveBeenCalled();
});
it("keeps closed tasks ineligible for a compaction request", async () => {
const f = await waitingCompactionFixture();
const [closed] = await db.insert(issues).values({ companyId: f.company.id, title: "Closed task",
status: "done", assigneeAgentId: f.agent.id }).returning();
if (!closed) throw new Error("Missing closed task");
await expect(f.service.request({ companyId: f.company.id, issue: closed, userId: f.user.id }))
.rejects.toMatchObject({ code: "task_closed" });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agent.id))).toHaveLength(0);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agent.id))).toHaveLength(0);
});
it("keeps a busy conversation ineligible for a second compaction run", async () => {
const f = await waitingCompactionFixture();
const [running] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id,
issueId: f.issue.id, scopeKind: "issue", status: "running" }).returning();
if (!running) throw new Error("Missing active conversation run");
await expect(f.service.request({ companyId: f.company.id, issue: f.issue, userId: f.user.id }))
.rejects.toMatchObject({ code: "agent_busy" });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agent.id))).map(run => run.id))
.toEqual([running.id]);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agent.id))).toHaveLength(0);
expect(f.deps.executeRun).not.toHaveBeenCalled();
});
it.each(["comment", "interaction", "ordinary task"] as const)("preserves admission for a current %s", async cause => {
const f = await waitingConversationFixture();
const context: Record<string, unknown> = { issueId: f.issue.id, source: "issue.continuation_recovery" };
const payload: Record<string, unknown> = { issueId: f.issue.id };
if (cause === "comment") {
const [comment] = await db.insert(issueComments).values({ companyId: f.company.id, issueId: f.issue.id,
authorUserId: f.user.id, body: "A new question" }).returning();
context.wakeCommentId = comment.id;
payload.commentId = comment.id;
} else if (cause === "interaction") {
const [interaction] = await db.insert(issueThreadInteractions).values({ companyId: f.company.id,
issueId: f.issue.id, createdByAgentId: f.agent.id, kind: "ask_user_questions", status: "answered",
resolvedByUserId: f.user.id, resolvedAt: new Date(),
payload: { version: 1, questions: [{ id: "q", prompt: "Continue?", selectionMode: "single",
options: [{ id: "a", label: "Continue" }] }] },
result: { version: 1, answers: [{ questionId: "q", optionIds: ["a"] }] } }).returning();
Object.assign(context, { source: "issue.interaction.respond", interactionId: interaction.id,
interactionKind: interaction.kind, interactionStatus: interaction.status });
Object.assign(payload, { mutation: "interaction", interactionId: interaction.id });
} else {
await db.update(issues).set({ conversationAgentId: null, conversationUserId: null,
conversationState: null, status: "in_progress" }).where(eq(issues.id, f.issue.id));
}
const admitted = await f.queue.enqueueWakeup(f.agent.id, { source: "automation", triggerDetail: "system",
reason: cause === "ordinary task" ? "issue.continuation_recovery" : "issue_commented",
requestedByActorType: "system", payload, contextSnapshot: context });
expect(admitted).toMatchObject({ agentId: f.agent.id, status: "queued" });
expect(f.deps.executeRun).not.toHaveBeenCalled();
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agent.id)))
.toHaveLength(1);
});
it("delivers a real task completion to its waiting conversation", async () => {
const f = await waitingConversationFixture();
const [source] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id,
status: "succeeded", finishedAt: new Date(), contextSnapshot: { issueId: f.issue.id, conversationSessionGeneration: 0 } }).returning();
const [task] = await db.insert(issues).values({ companyId: f.company.id, title: "Handed-off task",
status: "in_progress", createdByAgentId: f.agent.id }).returning();
await recordChatHandoff(db, task, source.id);
const recorded = await db.transaction(async tx => {
const [done] = await tx.update(issues).set({ status: "done" }).where(eq(issues.id, task.id)).returning();
return recordChatCompletion(tx, task, done);
});
expect(recorded).toBe(true);
await chatCompletionDeliveryService(db, { wakeup: (agentId, options) => f.queue.enqueueWakeup(agentId, options) }).sweepPending();
const [delivery] = await db.select().from(chatCompletionDeliveries).where(eq(chatCompletionDeliveries.companyId, f.company.id));
expect(delivery).toMatchObject({ status: "queued" });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, delivery.targetRunId!));
expect(run).toMatchObject({ agentId: f.agent.id, status: "queued" });
expect(run.contextSnapshot).toMatchObject({ issueId: f.issue.id, wakeReason: "chat_task_completed" });
expect(f.deps.executeRun).not.toHaveBeenCalled();
});
async function fixture(status = "running") {
const [company] = await db.insert(companies).values({ name: "Queue module", issuePrefix: "QM" }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Queue agent", role: "engineer", status: "idle", adapterType: "process", adapterConfig: {} }).returning();
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Queue task", status: "todo", assigneeAgentId: agent.id }).returning();
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: company.id, agentId: agent.id, source: "assignment", status: "claimed", claimedAt: new Date() }).returning();
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, status, invocationSource: "assignment", wakeupRequestId: wake.id, startedAt: status === "running" ? new Date() : null, contextSnapshot: { issueId: issue.id } }).returning();
await db.update(issues).set({ executionRunId: run.id, executionAgentNameKey: agent.name, executionLockedAt: new Date() }).where(eq(issues.id, issue.id));
return { company, agent, issue, wake, run };
}
it.each([true, false])("adopts only eligible saved comments from the exact reassignment source (%s)", async (matchingIssue) => {
const { user, company, agent, issue, deps } = await waitingConversationFixture();
await db.update(issues).set({ conversationAgentId: null, conversationUserId: null,
conversationState: null, status: "todo" }).where(eq(issues.id, issue.id));
const [previousAgent] = await db.insert(agents).values({ companyId: company.id,
name: "Previous assignee", adapterType: "process" }).returning();
const [interruptedRun] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: previousAgent.id,
status: "cancelled", errorCode: "issue_reassigned", finishedAt: new Date(),
contextSnapshot: { issueId: matchingIssue ? issue.id : randomUUID() } }).returning();
const [earlier, later] = await db.insert(issueComments).values(["Earlier", "Later"].map(body => ({
companyId: company.id, issueId: issue.id, authorType: "user" as const, authorUserId: user.id, body,
}))).returning();
const [local, handoff, chat, interaction] = await db.insert(agentWakeupRequests).values([
{ agentId: agent.id, commentId: later.id, requestedAt: new Date(2) },
{ agentId: previousAgent.id, commentId: earlier.id, requestedAt: new Date(1) },
{ agentId: agent.id, commentId: randomUUID(), requestedAt: new Date(3), idempotencyKey: `chat-inbound:${randomUUID()}` },
{ agentId: agent.id, commentId: randomUUID(), requestedAt: new Date(4), mutation: "interaction" },
].map(({ agentId, commentId, requestedAt, idempotencyKey, mutation }) => ({
companyId: company.id, agentId, source: "automation", reason: "issue_commented", status: "deferred_issue_execution",
requestedAt, idempotencyKey, requestedByActorType: "user", requestedByActorId: user.id,
payload: withQueuedCommentIdsInWakePayload({ issueId: issue.id, mutation }, [commentId]),
}))).returning();
const loadExecutionContext = vi.spyOn(deps, "getIssueExecutionContext");
const queue = createHeartbeatQueue(db, deps);
const admit = deps.wakeQueue.admitIssueWake.getMockImplementation()!;
deps.wakeQueue.admitIssueWake.mockImplementation((target, evaluate, effects) => admit(target, async (tx, admission) => {
await tx.insert(agentTaskSessions).values({ companyId: company.id, agentId: agent.id,
adapterType: agent.adapterType, taskKey: issue.id, sessionDisplayId: "session-available-in-admission" });
return evaluate(tx, admission);
}, effects));
const run = await queue.enqueueWakeup(agent.id, {
source: "assignment", triggerDetail: "system", reason: "issue_assigned", requestedByActorType: "system",
payload: { issueId: issue.id }, contextSnapshot: { issueId: issue.id, interruptedRunId: interruptedRun.id },
});
expect(loadExecutionContext).not.toHaveBeenCalled();
const commentIds = matchingIssue ? [earlier.id, later.id] : [later.id];
expect(run).toMatchObject({ status: "queued", responsibleUserId: user.id, invocationSource: "assignment",
sessionIdBefore: "session-available-in-admission",
contextSnapshot: { wakeCommentIds: commentIds, wakeCommentId: later.id,
queuedCommentSourceWakeIds: matchingIssue ? [handoff.id, local.id] : [local.id] } });
const receipts = new Map((await db.select().from(agentWakeupRequests)).map(wake => [wake.id, wake]));
expect(receipts.get(run!.wakeupRequestId!)).toMatchObject({ runId: run!.id, source: "assignment",
requestedByActorType: "system", payload: { _paperclipWakeContext: { wakeCommentIds: commentIds } } });
expect(receipts.get(local.id)).toMatchObject({ status: "coalesced", runId: run!.id, requestedByActorId: user.id });
expect(receipts.get(handoff.id)).toMatchObject(matchingIssue
? { status: "coalesced", runId: run!.id, agentId: previousAgent.id }
: { status: "deferred_issue_execution", runId: null });
for (const receipt of [chat, interaction]) expect(receipts.get(receipt.id)).toEqual(receipt);
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("uses the routine run attribution visible in the admission transaction", async () => {
const { company, agent, issue, deps, queue } = await waitingConversationFixture();
const [routine] = await db.insert(routines).values({
companyId: company.id, title: "Routine", assigneeAgentId: agent.id, responsibleUserId: "current-routine-owner",
}).returning();
const [routineRun] = await db.insert(routineRuns).values({
companyId: company.id, routineId: routine.id, source: "schedule", responsibleUserId: "original-routine-owner",
}).returning();
await db.update(issues).set({
conversationAgentId: null, conversationUserId: null, conversationState: null, status: "todo",
originKind: "routine_execution", originId: routine.id,
}).where(eq(issues.id, issue.id));
const admit = deps.wakeQueue.admitIssueWake.getMockImplementation()!;
deps.wakeQueue.admitIssueWake.mockImplementation((target, evaluate, effects) => admit(target, async (tx, admission) => {
await tx.update(issues).set({ originRunId: routineRun.id }).where(eq(issues.id, issue.id));
return evaluate(tx, { ...admission, lockedIssue: await admission.lockIssue() });
}, effects));
const run = await queue.enqueueWakeup(agent.id, {
source: "assignment", triggerDetail: "system", reason: "issue_assigned", requestedByActorType: "system",
payload: { issueId: issue.id },
});
expect(run).toMatchObject({ status: "queued", responsibleUserId: "original-routine-owner" });
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("only promotes a deferred chat receipt for its agent and binds receipts within its company", async () => {
const local = await fixture("queued");
const [foreignCompany] = await db.insert(companies).values({ name: "Other company", issuePrefix: "OC" }).returning();
const [foreignAgent] = await db.insert(agents).values({
companyId: foreignCompany.id, name: "Other agent", adapterType: "process",
}).returning();
const [foreignWake] = await db.insert(agentWakeupRequests).values({
companyId: foreignCompany.id, agentId: foreignAgent.id, source: "automation",
status: "deferred_issue_execution", payload: { original: true },
}).returning();
const [localDeferredWake] = await db.insert(agentWakeupRequests).values({
companyId: local.company.id, agentId: local.agent.id, source: "automation",
status: "deferred_issue_execution",
}).returning();
await db.transaction(async (tx) => {
const store = createFreshIssueRunStore(tx, local.company.id);
expect(await store.saveReceipt({
kind: "promote", wakeId: localDeferredWake.id, agentId: foreignAgent.id, payload: null,
})).toBeUndefined();
await store.bindReceipts({
wakeId: foreignWake.id, runId: local.run.id, failedChatRetryReceiptId: foreignWake.id,
adopted: { wakeIds: [foreignWake.id, localDeferredWake.id], payload: { replaced: true } },
});
});
const [preserved] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, foreignWake.id));
expect(preserved).toMatchObject({ status: "deferred_issue_execution", runId: null, payload: { original: true } });
const [adopted] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, localDeferredWake.id));
expect(adopted).toMatchObject({ status: "coalesced", runId: local.run.id });
});
async function pendingPredecessorFixture() {
const f = await fixture("queued");
const [source] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id,
issueId: f.issue.id, status: "failed", runtimeMode: "legacy", finishedAt: new Date(),
executionStage: "preparing", errorCode: "adapter_failed",
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
contextSnapshot: { issueId: f.issue.id } }).returning();
const [lease] = await db.insert(environmentLeases).values({ companyId: f.company.id, issueId: f.issue.id,
heartbeatRunId: source.id, provider: "daytona", providerLeaseId: crypto.randomUUID(), status: "active" }).returning();
const [successor] = await db.update(heartbeatRuns).set({ retryOfRunId: source.id, responsibleUserId: "queue-test-user",
contextSnapshot: { issueId: f.issue.id, source: "issue.continuation_recovery", retryOfRunId: source.id,
wakeReason: "issue_continuation_needed" } }).where(eq(heartbeatRuns.id, f.run.id)).returning();
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(f.agent);
return { ...f, source, lease, successor, deps, queue: createHeartbeatQueue(db, deps) };
}
it.each([
["issue_continuation_needed", "cancelled"], ["issue_recovery_action_restored", "cancelled"], ["issue_children_completed", "cancelled"],
["issue_commented", "running"],
] as const)("does not start a %s run whose task another run already finished (expected %s)", async (wakeReason, expected) => {
// Release-swap wave t011: a delivered continuation queued while its task was open started after
// another run marked the task done, and took an extra turn on a finished task.
const f = await fixture("queued");
await db.update(issues).set({ status: "done", executionRunId: null }).where(eq(issues.id, f.issue.id));
// A person's message wake carries the comment it answers.
const wakeCommentId = wakeReason === "issue_commented"
? (await db.insert(issueComments).values({ companyId: f.company.id, issueId: f.issue.id, authorAgentId: f.agent.id,
body: "One more thing" }).returning())[0].id
: undefined;
const [queued] = await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user",
contextSnapshot: { issueId: f.issue.id, wakeReason, ...(wakeCommentId ? { wakeCommentId } : {}) } })
.where(eq(heartbeatRuns.id, f.run.id)).returning();
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(f.agent);
const queue = createHeartbeatQueue(db, deps);
const claimed = await queue.claimQueuedRun(queued);
if (expected === "cancelled") {
expect(claimed).toBeNull();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, queued.id)))[0])
.toMatchObject({ status: "cancelled", errorCode: "issue_closed_before_start" });
} else {
// A person's message on a done task still runs (and reopens it through the normal path).
expect(claimed).toMatchObject({ id: queued.id, status: "running" });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, queued.id)))[0])
.toMatchObject({ status: "running", errorCode: null });
}
});
it.each([
{ label: "still undelivered", delivered: false },
{ label: "already read by the closing run", delivered: true },
])("answers a person's message attached to a system continuation on a closed task once ($label)", async ({ delivered }) => {
// Admission can attach a deferred comment to a queued system continuation without changing its
// wake reason. The claim-time closed-task cancel must not consume that person's message.
const f = await fixture("queued");
const [comment] = await db.insert(issueComments).values({ companyId: f.company.id, issueId: f.issue.id,
authorType: "user", authorUserId: "queue-test-user", body: "Can you also update the changelog?" }).returning();
if (delivered) {
// The run that closed the task received the message as its own wake input.
const startedAt = new Date(comment.createdAt.getTime() + 1_000);
await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id, status: "succeeded",
invocationSource: "assignment", startedAt, finishedAt: new Date(startedAt.getTime() + 1_000),
contextSnapshot: { issueId: f.issue.id, wakeReason: "issue_commented", wakeCommentIds: [comment.id] } });
}
await db.update(issues).set({ status: "done", executionRunId: null }).where(eq(issues.id, f.issue.id));
const [queued] = await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user",
contextSnapshot: { issueId: f.issue.id, wakeReason: "issue_continuation_needed", wakeCommentIds: [comment.id] } })
.where(eq(heartbeatRuns.id, f.run.id)).returning();
// The continuation's wake is still queued and bound to the run, as admission leaves it.
await db.update(agentWakeupRequests).set({ status: "queued", claimedAt: null, runId: queued.id })
.where(eq(agentWakeupRequests.id, f.wake.id));
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(f.agent);
const claimed = await createHeartbeatQueue(db, deps).claimQueuedRun(queued);
const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agent.id));
if (delivered) {
expect(claimed).toBeNull();
expect(runs.find(row => row.id === queued.id)).toMatchObject({ status: "cancelled", errorCode: "issue_closed_before_start" });
expect(runs).toHaveLength(2);
} else {
// Exactly one run answers it: this one, claimed as the person's follow-up.
expect(claimed).toMatchObject({ id: queued.id, status: "running" });
expect(runs.filter(row => row.status !== "succeeded")).toEqual([expect.objectContaining({ id: queued.id, status: "running" })]);
}
});
it("keeps the result of a continuation another dispatcher already started and finished", async () => {
// Two servers overlap during a swap: both read the same queued continuation; one starts it and
// the run closes the task before the other reaches the closed-task check with its stale copy.
const f = await fixture("queued");
const [staleQueued] = await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issue.id, wakeReason: "issue_continuation_needed" } })
.where(eq(heartbeatRuns.id, f.run.id)).returning();
const finishedAt = new Date();
await db.update(heartbeatRuns).set({ status: "succeeded", startedAt: finishedAt, finishedAt,
resultJson: { summary: "Finished by the other server" } }).where(eq(heartbeatRuns.id, f.run.id));
await db.update(issues).set({ status: "done", executionRunId: null }).where(eq(issues.id, f.issue.id));
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(f.agent);
expect(await createHeartbeatQueue(db, deps).claimQueuedRun(staleQueued)).toBeNull();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0]).toMatchObject({
status: "succeeded", errorCode: null, resultJson: { summary: "Finished by the other server" } });
expect(deps.setWakeupStatus).not.toHaveBeenCalled();
expect(deps.releaseIssueExecutionAndPromote).not.toHaveBeenCalled();
});
it("returns the claim to the queue when run admission closes while the claim is being checked", async () => {
// SIGTERM (drain) or the hand-over yield can close admission during claimQueuedRun's awaited
// checks, after its first admission check passed. The run must stay exactly as it was.
const f = await fixture("queued");
const [queued] = await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user",
contextSnapshot: { issueId: f.issue.id, wakeReason: "issue_commented" } })
.where(eq(heartbeatRuns.id, f.run.id)).returning();
const [issueBefore] = await db.select().from(issues).where(eq(issues.id, f.issue.id));
let admissionClosed = false;
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(f.agent);
deps.isRunAdmissionClosed = () => admissionClosed;
deps.budgets.getInvocationBlock.mockImplementation(async () => { admissionClosed = true; return null; });
expect(await createHeartbeatQueue(db, deps).claimQueuedRun(queued)).toBeNull();
const [stored] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, queued.id));
expect(stored).toMatchObject({ status: "queued", startedAt: null, finishedAt: null, errorCode: null,
responsibleUserId: queued.responsibleUserId, runnerProfileJson: queued.runnerProfileJson });
expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0]).toMatchObject({
executionRunId: issueBefore.executionRunId, executionLockedAt: issueBefore.executionLockedAt });
expect(deps.setWakeupStatus).not.toHaveBeenCalled();
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("returns the claim to the queue when a reattach phase begins while the claim is being checked", async () => {
const f = await fixture("queued");
const [queued] = await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user",
contextSnapshot: { issueId: f.issue.id, wakeReason: "issue_commented" } })
.where(eq(heartbeatRuns.id, f.run.id)).returning();
const [issueBefore] = await db.select().from(issues).where(eq(issues.id, f.issue.id));
let reattachPhaseActive = false;
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(f.agent);
deps.isRunAdmissionClosed = () => false;
deps.isReattachPhaseActive = () => reattachPhaseActive;
deps.budgets.getInvocationBlock.mockImplementation(async () => { reattachPhaseActive = true; return null; });
expect(await createHeartbeatQueue(db, deps).claimQueuedRun(queued)).toBeNull();
const [stored] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, queued.id));
expect(stored).toMatchObject({ status: "queued", startedAt: null, finishedAt: null, errorCode: null,
responsibleUserId: queued.responsibleUserId, runnerProfileJson: queued.runnerProfileJson });
expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0]).toMatchObject({
executionRunId: issueBefore.executionRunId, executionLockedAt: issueBefore.executionLockedAt });
expect(deps.setWakeupStatus).not.toHaveBeenCalled();
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("keeps an automatic continuation queued while its legacy predecessor still owns cleanup", async () => {
const { successor, lease, queue, deps, ...f } = await pendingPredecessorFixture();
expect(await queue.claimQueuedRun(successor)).toBeNull();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, successor.id)))[0].status).toBe("queued");
expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0].executionRunId).toBe(successor.id);
expect(deps.executeRun).not.toHaveBeenCalled();
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success" })
.where(eq(environmentLeases.id, lease.id));
expect(await queue.claimQueuedRun(successor)).toMatchObject({ id: successor.id, status: "running" });
expect(await queue.claimQueuedRun(successor)).toBeNull();
});
async function reassignedPredecessorFixture(previousStatus = "cancelled") {
const f = await fixture("queued");
await db.update(issues).set({ executionRunId: null, executionAgentNameKey: null, executionLockedAt: null })
.where(eq(issues.id, f.issue.id));
const [previousAgent] = await db.insert(agents).values({ companyId: f.company.id, name: "Previous owner",
role: "engineer", status: "idle", adapterType: "paperclip_runner", adapterConfig: {} }).returning();
const [previous] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: previousAgent.id,
status: previousStatus, runtimeMode: "native", invocationSource: "assignment", startedAt: new Date(),
finishedAt: previousStatus === "running" ? null : new Date(), nativeIssueId: f.issue.id,
errorCode: previousStatus === "cancelled" ? "issue_reassigned" : null,
contextSnapshot: { issueId: f.issue.id, wakeReason: "issue_assigned" } }).returning();
const [next] = await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user",
contextSnapshot: { issueId: f.issue.id, wakeReason: "issue_assigned" } })
.where(eq(heartbeatRuns.id, f.run.id)).returning();
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(f.agent);
deps.activeRunExecutions.add(previous.id);
return { ...f, previousAgent, previous, next, deps, queue: createHeartbeatQueue(db, deps) };
}
it("keeps the new owner's run queued while the reassigned owner's stopped run still settles", async () => {
const f = await reassignedPredecessorFixture();
expect(await f.queue.claimQueuedRun(f.next)).toBeNull();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.next.id)))[0])
.toMatchObject({ status: "queued", startedAt: null });
expect(f.deps.executeRun).not.toHaveBeenCalled();
f.deps.activeRunExecutions.delete(f.previous.id);
expect(await f.queue.claimQueuedRun(f.next)).toMatchObject({ id: f.next.id, status: "running" });
});
it("does not hold a run behind another agent's live run on the same task", async () => {
const f = await reassignedPredecessorFixture("running");
expect(await f.queue.claimQueuedRun(f.next)).toMatchObject({ id: f.next.id, status: "running" });
});
it("wakes queued dispatch for the task's next owner once the previous owner's execution settles", async () => {
const f = await reassignedPredecessorFixture();
let wakes = 0;
const unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.queuedRunDispatch, () => { wakes += 1; });
try {
expect(await notifyOtherAgentsQueuedOnSettledIssue(db, f.previous)).toBe(true);
await vi.waitFor(() => expect(wakes).toBeGreaterThan(0));
await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.next.id));
expect(await notifyOtherAgentsQueuedOnSettledIssue(db, f.previous)).toBe(false);
} finally {
unsubscribe();
}
});
it("keeps a retry queued after failed cleanup and until its local executor releases", async () => {
const f = await pendingPredecessorFixture();
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "failed" })
.where(eq(environmentLeases.id, f.lease.id));
expect(await f.queue.claimQueuedRun(f.successor)).toBeNull();
await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, f.lease.id));
f.deps.liveRunExecutions.has.mockImplementation(id => id === f.source.id);
expect(await f.queue.claimQueuedRun(f.successor)).toBeNull();
f.deps.liveRunExecutions.has.mockReturnValue(false);
expect(await f.queue.claimQueuedRun(f.successor)).toMatchObject({ id: f.successor.id, status: "running" });
});
it.each(["missing", "foreign company", "other task", "other agent", "native task overrides context"] as const)(
"does not hold a queued run for a %s retry source", async mismatch => {
const f = await pendingPredecessorFixture();
let sourceId = f.source.id;
if (mismatch === "missing") {
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, sourceId));
expect(await f.queue.claimQueuedRun(f.successor)).toMatchObject({ id: f.successor.id, status: "running" });
return;
} else if (mismatch === "foreign company") {
const [company] = await db.insert(companies).values({ name: "Unrelated company", issuePrefix: "OTHER" }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Other company worker", adapterType: "process" }).returning();
const [source] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
status: "failed", finishedAt: new Date() }).returning();
sourceId = source.id;
await db.insert(environmentLeases).values({ companyId: company.id, heartbeatRunId: sourceId, status: "active", provider: "daytona" });
} else if (mismatch === "other agent") {
const [other] = await db.insert(agents).values({ companyId: f.company.id, name: "Another worker", adapterType: "process" }).returning();
const [source] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: other.id, issueId: f.issue.id,
status: "failed", finishedAt: new Date() }).returning();
sourceId = source.id;
await db.insert(environmentLeases).values({ companyId: f.company.id, heartbeatRunId: sourceId, status: "active", provider: "daytona" });
} else {
const [other] = await db.insert(issues).values({ companyId: f.company.id, title: "Another task" }).returning();
const [source] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id, issueId: other.id,
...(mismatch === "native task overrides context" ? { nativeIssueId: other.id } : {}),
contextSnapshot: { issueId: f.issue.id }, status: "failed", finishedAt: new Date() }).returning();
sourceId = source.id;
await db.insert(environmentLeases).values({ companyId: f.company.id, heartbeatRunId: sourceId, status: "active", provider: "daytona" });
}
const [successor] = await db.update(heartbeatRuns).set({ retryOfRunId: sourceId })
.where(eq(heartbeatRuns.id, f.successor.id)).returning();
expect(await f.queue.claimQueuedRun(successor)).toMatchObject({ id: successor.id, status: "running" });
},
);
it("retains the current execution owner's hold when retry provenance is unrelated", async () => {
const f = await pendingPredecessorFixture();
const [unrelated] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id,
status: "succeeded", finishedAt: new Date() }).returning();
const [successor] = await db.update(heartbeatRuns).set({ retryOfRunId: unrelated.id })
.where(eq(heartbeatRuns.id, f.successor.id)).returning();
await db.update(issues).set({ executionRunId: f.source.id }).where(eq(issues.id, f.issue.id));
expect(await f.queue.claimQueuedRun(successor)).toBeNull();
expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0].executionRunId).toBe(f.source.id);
});
it.each((["process", "paperclip_runner"] as const).flatMap(adapterType => (["coalesced", "missing-execution-key", "stale-execution-key", "claimed-primary", "legacy-envelope", "delete-old", "delete-new", "stopped", "running", "system-primary", "foreign-primary", "same-named-agent", "mixed-author"] as const).map(scenario => ({ adapterType, scenario }))))("protects a queued comment batch across $adapterType/$scenario", async ({ adapterType, scenario }) => {
const f = await fixture("queued");
let admissionAgent: typeof agents.$inferSelect = { ...f.agent, adapterType, adapterConfig: adapterType === "paperclip_runner" ? { provider: "codex" } : {} };
await db.update(agents).set({ adapterType, adapterConfig: admissionAgent.adapterConfig }).where(eq(agents.id, f.agent.id));
if (scenario === "same-named-agent") [admissionAgent] = await db.insert(agents).values({ companyId: f.company.id,
name: f.agent.name.toUpperCase(), role: "engineer", status: "idle", adapterType, adapterConfig: admissionAgent.adapterConfig,
}).returning();
const userId = `source-user-${f.company.id}`;
await db.insert(authUsers).values({ id: userId, name: "Source owner", email: `${f.company.id}@queue.test`, createdAt: new Date(), updatedAt: new Date() });
await db.insert(companyMemberships).values({ companyId: f.company.id, principalType: "user", principalId: userId, status: "active", membershipRole: "operator" });
await db.update(companies).set({ defaultResponsibleUserId: userId }).where(eq(companies.id, f.company.id));
const incomingActorId = scenario === "mixed-author" ? `second-user-${f.company.id}` : userId;
if (scenario === "mixed-author") {
await db.insert(authUsers).values({ id: incomingActorId, name: "Incoming source owner", email: `${incomingActorId}@queue.test`, createdAt: new Date(), updatedAt: new Date() });
await db.insert(companyMemberships).values({ companyId: f.company.id, principalType: "user", principalId: incomingActorId, status: "active", membershipRole: "operator" });
}
const [oldComment] = await db.insert(issueComments).values({ companyId: f.company.id, issueId: f.issue.id,
authorType: "user", authorUserId: userId, body: "Review source A" }).returning();
const [newComment] = await db.insert(issueComments).values({ companyId: f.company.id, issueId: f.issue.id,
authorType: "user", authorUserId: incomingActorId, body: "Review source B" }).returning();
await db.update(agentWakeupRequests).set({ status: scenario === "claimed-primary" ? "claimed" : "queued", runId: f.run.id, claimedAt: null,
requestedByActorType: scenario === "system-primary" ? "system" : "user", requestedByActorId: scenario === "system-primary" ? null : userId,
payload: scenario === "legacy-envelope" ? { issueId: f.issue.id, commentId: oldComment.id }
: withQueuedCommentIdsInWakePayload({ issueId: f.issue.id, retainedSource: "original-source" }, [oldComment.id]),
}).where(eq(agentWakeupRequests.id, f.wake.id));
const [scanned] = await db.update(heartbeatRuns).set({ responsibleUserId: userId,
contextSnapshot: withQueuedCommentIdsInRunContext({ issueId: f.issue.id, wakeReason: "issue_commented" }, [oldComment.id]),
}).where(eq(heartbeatRuns.id, f.run.id)).returning();
if (scenario === "missing-execution-key" || scenario === "stale-execution-key") {
const [staleRun] = scenario === "stale-execution-key" ? await db.insert(heartbeatRuns).values({
companyId: f.company.id, agentId: f.agent.id, status: "succeeded", contextSnapshot: { issueId: f.issue.id },
}).returning() : [];
await db.update(issues).set({ executionRunId: staleRun?.id ?? null,
executionAgentNameKey: staleRun ? "previous agent name" : null }).where(eq(issues.id, f.issue.id));
}
let foreignPrimary: typeof agentWakeupRequests.$inferSelect | undefined;
if (scenario === "foreign-primary") {
const [company] = await db.insert(companies).values({ name: "Foreign source", issuePrefix: "FOREIGN" }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Foreign source agent", adapterType: "process" }).returning();
[foreignPrimary] = await db.insert(agentWakeupRequests).values({ companyId: company.id, agentId: agent.id, source: "assignment", status: "queued", payload: { source: "foreign" } }).returning();
await db.update(heartbeatRuns).set({ wakeupRequestId: foreignPrimary.id }).where(eq(heartbeatRuns.id, f.run.id));
}
if (scenario === "delete-old") await db.update(issueComments).set({ deletedAt: new Date() }).where(eq(issueComments.id, oldComment.id));
const deps = callbacks(db);
const resolveSession = vi.spyOn(deps, "resolveSessionBeforeForWakeup");
deps.getAgent.mockResolvedValue(admissionAgent);
deps.canStartNewExternalBotWork.mockResolvedValue(false);
deps.instanceSettings.getExperimental.mockResolvedValue({ enableIsolatedWorkspaces: false, enableNativeRunner: true } as Awaited<ReturnType<HeartbeatQueueDependencies["instanceSettings"]["getExperimental"]>>);
const admission = createWakeQueue(db, {
resolveResponsibleUserId: async () => userId,
getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }),
resolveSessionBeforeForWakeup: async () => null,
wakeAdmissionHelpers: { filterZombieCoalesceTarget: deps.filterZombieCoalesceTarget,
mergeCoalescedContextSnapshot, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake },
recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined },
});
const busyAdmission = vi.fn();
deps.wakeQueue.admitIssueWake.mockImplementation((target, evaluate, effects) => admission.admitIssueWake(target,
(tx, stores) => evaluate(tx, { ...stores, async prepareRecovery(input) {
const recovery = await stores.prepareRecovery(input);
return { ...recovery, async checkBusyAdmission() {
const busy = await recovery.checkBusyAdmission();
if (busy.kind !== "allowed") return busy;
return { ...busy, admitBusy(input) {
busyAdmission(input.activeExecutionRun.id, input.issueExecutionAgentNameKey);
return busy.admitBusy(input);
} };
} };
} }), effects));
const queue = createHeartbeatQueue(db, deps);
if (scenario === "running") {
deps.canStartNewExternalBotWork.mockResolvedValue(true);
expect(await queue.claimQueuedRun(scanned)).toMatchObject({ status: "running" });
deps.canStartNewExternalBotWork.mockResolvedValue(false);
}
const [originalPrimary] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wake.id));
const coalesced = await queue.enqueueWakeup(admissionAgent.id, { source: "on_demand", triggerDetail: "manual", reason: "issue_commented",
payload: { issueId: f.issue.id, commentId: newComment.id }, contextSnapshot: { issueId: f.issue.id, wakeCommentId: newComment.id },
requestedByActorType: "user", requestedByActorId: incomingActorId });
if (["running", "foreign-primary", "same-named-agent"].includes(scenario)) {
expect(coalesced).toBeNull();
const [preserved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
expect(preserved.contextSnapshot).toMatchObject({ wakeCommentId: oldComment.id, wakeCommentIds: [oldComment.id] });
const receipts = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.company.id));
expect(receipts.find(row => row.id !== f.wake.id)).toMatchObject({ status: "deferred_issue_execution", runId: null });
expect(receipts.find(row => row.id === f.wake.id)).toEqual(originalPrimary);
if (scenario === "same-named-agent") {
expect(preserved).toEqual(scanned);
expect(receipts.find(row => row.id !== f.wake.id)).toMatchObject({ agentId: admissionAgent.id,
requestedByActorType: "user", requestedByActorId: incomingActorId, payload: { issueId: f.issue.id, commentId: newComment.id } });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.company.id))).toEqual([scanned]);
}
if (foreignPrimary) expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, foreignPrimary.id)))[0]).toEqual(foreignPrimary);
expect(deps.executeRun).not.toHaveBeenCalled();
return;
}
expect(coalesced).toMatchObject({ id: f.run.id, status: "queued", contextSnapshot: { wakeCommentIds: [oldComment.id, newComment.id] } });
expect(resolveSession).not.toHaveBeenCalled();
if (scenario === "missing-execution-key" || scenario === "stale-execution-key") {
expect(busyAdmission).toHaveBeenCalledExactlyOnceWith(f.run.id, f.agent.name.toLowerCase());
}
const accepted = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.company.id));
expect(accepted.find(row => row.id !== f.wake.id)).toMatchObject({ status: "coalesced", runId: f.run.id, finishedAt: expect.any(Date),
requestedByActorType: "user", requestedByActorId: incomingActorId });
const [updatedPrimary] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wake.id));
expect(updatedPrimary).toMatchObject({ id: originalPrimary.id, companyId: originalPrimary.companyId,
agentId: originalPrimary.agentId, source: originalPrimary.source, requestedByActorType: originalPrimary.requestedByActorType,
requestedByActorId: originalPrimary.requestedByActorId, idempotencyKey: originalPrimary.idempotencyKey, requestedAt: originalPrimary.requestedAt });
if (scenario === "legacy-envelope" || scenario === "claimed-primary") expect(updatedPrimary.payload).toEqual(originalPrimary.payload);
else expect(updatedPrimary.payload).toMatchObject({ retainedSource: "original-source", commentId: oldComment.id,
_paperclipWakeContext: { commentId: oldComment.id, wakeCommentId: oldComment.id, wakeCommentIds: [oldComment.id, newComment.id] } });
if (scenario === "claimed-primary") {
deps.canStartNewExternalBotWork.mockResolvedValue(true);
expect(await queue.claimQueuedRun(scanned)).toBeNull();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0])
.toMatchObject({ status: "queued", contextSnapshot: { wakeCommentIds: [oldComment.id, newComment.id] } });
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wake.id)))[0]).toEqual(originalPrimary);
expect(deps.executeRun).not.toHaveBeenCalled();
return;
}
if (scenario === "stopped") {
await createHeartbeatRunControl(db, runControlCallbacks(db)).cancelRunInternal(f.run.id, "operator Stop");
deps.canStartNewExternalBotWork.mockResolvedValue(true);
expect(await queue.claimQueuedRun(scanned)).toBeNull();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0])
.toMatchObject({ status: "cancelled", contextSnapshot: { wakeCommentIds: [oldComment.id, newComment.id] } });
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wake.id)))[0].status).toBe("cancelled");
expect(deps.executeRun).not.toHaveBeenCalled();
return;
}
if (scenario === "delete-new") await db.update(issueComments).set({ deletedAt: new Date() }).where(eq(issueComments.id, newComment.id));
deps.canStartNewExternalBotWork.mockResolvedValue(true);
const claimed = await queue.claimQueuedRun(scanned);
expect(claimed).toMatchObject({ id: f.run.id, status: "running",
contextSnapshot: { wakeCommentId: scenario === "delete-new" ? oldComment.id : newComment.id,
wakeCommentIds: scenario === "delete-old" ? [newComment.id] : scenario === "delete-new" ? [oldComment.id] : [oldComment.id, newComment.id] } });
const receipts = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.company.id));
expect(receipts.find(row => row.id !== f.wake.id)).toMatchObject({ status: "coalesced", runId: f.run.id });
expect(deps.executeRun).not.toHaveBeenCalled();
});
async function strandedMention(runtimeMode = "legacy") {
const seeded = await fixture("succeeded");
await db.update(heartbeatRuns).set({ runtimeMode, nativeIssueId: runtimeMode === "native" ? seeded.issue.id : null, finishedAt: new Date() }).where(eq(heartbeatRuns.id, seeded.run.id));
if (runtimeMode === "native") await db.insert(nativeRunFinalizations).values({ runId: seeded.run.id, companyId: seeded.company.id, issueId: seeded.issue.id, phase: "committed" });
await db.update(issues).set({ executionRunId: null, checkoutRunId: null }).where(eq(issues.id, seeded.issue.id));
const [recipient] = await db.insert(agents).values({ companyId: seeded.company.id, name: "Mention recipient", role: "engineer", status: "idle", adapterType: "process", adapterConfig: {} }).returning();
const [comment] = await db.insert(issueComments).values({ companyId: seeded.company.id, issueId: seeded.issue.id, body: "Saved feedback", authorAgentId: seeded.agent.id }).returning();
const [pending] = await db.insert(agentWakeupRequests).values({ companyId: seeded.company.id, agentId: recipient.id,
source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution",
payload: { issueId: seeded.issue.id, _paperclipWakeContext: { issueId: seeded.issue.id, wakeReason: "issue_comment_mentioned", wakeCommentIds: [comment.id] } },
}).returning();
return { ...seeded, recipient, pending };
}
async function addStrandedMessages(seeded: Awaited<ReturnType<typeof strandedMention>>) {
const messages = [seeded.pending];
for (let index = 0; index < 2; index++) {
const [comment] = await db.insert(issueComments).values({ companyId: seeded.company.id, issueId: seeded.issue.id,
body: `Additional saved feedback ${index}`, authorAgentId: seeded.agent.id }).returning();
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: seeded.company.id, agentId: seeded.recipient.id,
source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution",
payload: { issueId: seeded.issue.id, _paperclipWakeContext: { issueId: seeded.issue.id,
wakeReason: "issue_comment_mentioned", wakeCommentIds: [comment.id] } } }).returning();
messages.push(wake);
}
await db.update(agentWakeupRequests).set({ updatedAt: new Date(0) }).where(inArray(agentWakeupRequests.id, messages.map(row => row.id)));
return messages;
}
async function expectSavedMessages(messages: Awaited<ReturnType<typeof addStrandedMessages>>) {
const rows = await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, messages.map(row => row.id)));
expect(rows).toHaveLength(messages.length);
for (const before of messages) {
const row = rows.find(row => row.id === before.id)!;
expect(row).toMatchObject({ payload: before.payload, status: "deferred_issue_execution", runId: null,
finishedAt: null, error: null });
expect(row.updatedAt.getTime()).toBeGreaterThan(0);
}
}
it.each(["legacy", "native"])("keeps saved %s messages while avoiding repeated native source release", async runtimeMode => {
const seeded = await strandedMention(runtimeMode);
const messages = await addStrandedMessages(seeded);
const deps = callbacks(db), queue = createHeartbeatQueue(db, deps);
await queue.resumeQueuedRuns();
expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(runtimeMode === "native" ? 1 : 3);
await expectSavedMessages(messages);
await queue.resumeQueuedRuns();
expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(runtimeMode === "native" ? 2 : 6);
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("retries a rejected native release on the next saved message", async () => {
const seeded = await strandedMention("native"), messages = await addStrandedMessages(seeded);
const deps = callbacks(db);
deps.releaseIssueExecutionAndPromote.mockRejectedValueOnce(new Error("Fixture release remains pending"));
await createHeartbeatQueue(db, deps).resumeQueuedRuns();
expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(2);
await expectSavedMessages(messages);
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("joins the native release before it suppresses another message's duplicate check", async () => {
const seeded = await strandedMention("native"), messages = await addStrandedMessages(seeded);
const deps = callbacks(db);
let finish!: () => void;
deps.releaseIssueExecutionAndPromote.mockImplementationOnce(() => new Promise<void>(resolve => { finish = resolve; }));
let settled = false;
const pending = createHeartbeatQueue(db, deps).resumeQueuedRuns().then(() => { settled = true; });
try {
await vi.waitFor(() => expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(1));
expect(settled).toBe(false);
} finally { finish?.(); }
await pending;
expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(1);
await expectSavedMessages(messages);
});
it("rechecks a different native source that completes during the pass", async () => {
const seeded = await strandedMention("native"), messages = await addStrandedMessages(seeded);
await db.update(heartbeatRuns).set({ createdAt: new Date(0) }).where(eq(heartbeatRuns.id, seeded.run.id));
const deps = callbacks(db);
let nextId: string | undefined;
deps.releaseIssueExecutionAndPromote.mockImplementationOnce(async () => {
const [next] = await db.insert(heartbeatRuns).values({ companyId: seeded.company.id, agentId: seeded.agent.id,
runtimeMode: "native", nativeIssueId: seeded.issue.id, status: "succeeded", finishedAt: new Date(),
contextSnapshot: { issueId: seeded.issue.id } }).returning();
nextId = next.id;
});
await createHeartbeatQueue(db, deps).resumeQueuedRuns();
expect(deps.releaseIssueExecutionAndPromote.mock.calls.map(([run]) => run.id)).toEqual([seeded.run.id, nextId]);
await expectSavedMessages(messages);
});
it("rechecks changed native evidence on the same source run", async () => {
const seeded = await strandedMention("native"), messages = await addStrandedMessages(seeded);
const deps = callbacks(db);
deps.releaseIssueExecutionAndPromote.mockImplementationOnce(async () => {
await db.update(heartbeatRuns).set({ resultJson: { observedReceipt: "new evidence" } }).where(eq(heartbeatRuns.id, seeded.run.id));
});
await createHeartbeatQueue(db, deps).resumeQueuedRuns();
expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(2);
expect(deps.releaseIssueExecutionAndPromote.mock.calls.map(([run]) => run.id)).toEqual([seeded.run.id, seeded.run.id]);
await expectSavedMessages(messages);
});
it("checks each unrelated native task in the same pass", async () => {
const seeded = await strandedMention("native"), firstMessages = await addStrandedMessages(seeded);
const [issue] = await db.insert(issues).values({ companyId: seeded.company.id, title: "Another saved task", status: "todo", assigneeAgentId: seeded.agent.id }).returning();
const [run] = await db.insert(heartbeatRuns).values({ companyId: seeded.company.id, agentId: seeded.agent.id,
runtimeMode: "native", nativeIssueId: issue.id, status: "succeeded", finishedAt: new Date(), contextSnapshot: { issueId: issue.id } }).returning();
const [comment] = await db.insert(issueComments).values({ companyId: seeded.company.id, issueId: issue.id,
body: "Other task feedback", authorAgentId: seeded.agent.id }).returning();
const [pending] = await db.insert(agentWakeupRequests).values({ companyId: seeded.company.id, agentId: seeded.recipient.id,
source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution",
payload: { issueId: issue.id, _paperclipWakeContext: { issueId: issue.id, wakeReason: "issue_comment_mentioned", wakeCommentIds: [comment.id] } } }).returning();
const secondMessages = await addStrandedMessages({ ...seeded, issue, run, pending });
const deps = callbacks(db);
await createHeartbeatQueue(db, deps).resumeQueuedRuns();
expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(2);
expect(new Set(deps.releaseIssueExecutionAndPromote.mock.calls.map(([row]) => row.id))).toEqual(new Set([seeded.run.id, run.id]));
await expectSavedMessages([...firstMessages, ...secondMessages]);
});
// H1 bug C review: a queued run admission cancels for a board-owned budget
// block records which budget stopped it, so Raise budget and resume can
// continue it.
async function admissionBudgetBlock(options: { openIncident: boolean }) {
const fixtureRows = await fixture("queued");
const { company, agent } = fixtureRows;
const [policy] = await db.insert(budgetPolicies).values({ companyId: company.id, scopeType: "company", scopeId: company.id,
windowKind: "lifetime", amount: 1, hardStopEnabled: true, isActive: true }).returning();
await db.insert(costEvents).values({ companyId: company.id, agentId: agent.id, provider: "test", model: "test",
costCents: 5, occurredAt: new Date() });
if (options.openIncident) await db.insert(budgetIncidents).values({ companyId: company.id, policyId: policy.id,
scopeType: "company", scopeId: company.id, metric: "billed_cents", windowKind: "lifetime",
windowStart: new Date(0), windowEnd: new Date(Date.now() + 86_400_000), thresholdType: "hard",
amountLimit: 1, amountObserved: 5 });
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(agent);
deps.budgets.getInvocationBlock.mockResolvedValue({ scopeType: "company", scopeId: company.id,
scopeName: company.name, reason: "Company cannot start work because its budget hard-stop is still exceeded." });
deps.cancelRunInternal.mockImplementation(async (runId: string) => {
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
});
return { ...fixtureRows, deps };
}
it("leaves a queued run queued when a resume closed its budget incident first", async () => {
// Raise budget and resume closed the incident between the admission
// check and the cancel: the re-read finds no block, so the task runs.
const { run, deps } = await admissionBudgetBlock({ openIncident: false });
deps.budgets.getInvocationBlock.mockResolvedValueOnce({ scopeType: "company", scopeId: run.companyId,
scopeName: "Budget", reason: "Company cannot start work because its budget hard-stop is still exceeded." })
.mockResolvedValueOnce(null);
expect(await createHeartbeatQueue(db, deps).claimQueuedRun({ ...run, responsibleUserId: "queue-test-user" })).toBeNull();
expect(deps.cancelRunInternal).not.toHaveBeenCalled();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("queued");
});
it("records the budget that cancelled a queued run at admission", async () => {
const { company, agent, run, deps } = await admissionBudgetBlock({ openIncident: true });
void agent;
expect(await createHeartbeatQueue(db, deps).claimQueuedRun({ ...run, responsibleUserId: "queue-test-user" })).toBeNull();
expect(deps.cancelRunInternal).toHaveBeenCalledWith(run.id, expect.stringContaining("budget"), expect.anything());
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].resultJson)
.toMatchObject({ budgetAdmissionStop: { scopeType: "company", scopeId: company.id } });
});
it("rechecks a handoff's assignment version under the issue claim lock", async () => {
const { company, agent, issue, wake, run } = await fixture("queued");
const [record] = await db.insert(cases).values({ companyId: company.id, caseNumber: 1, identifier: "QM-CASE-1",
caseType: "work", title: issue.title, workIssueId: issue.id, handoffVersion: 1 }).returning();
await db.insert(caseHandoffs).values({ companyId: company.id, caseId: record.id, issueId: issue.id,
version: 1, toAgentId: agent.id, phase: "implementation", instructions: "Build the change", actorType: "system",
actorId: "test", idempotencyKey: "queue-version-race", inputFingerprint: "fixture", wakeupRequestId: wake.id,
expectedIssueStatusVersion: issue.statusVersion, appliedIssueStatusVersion: issue.statusVersion, state: "delivered" });
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(agent);
deps.instanceSettings.getExperimental.mockResolvedValue({ enableCases: true } as Awaited<ReturnType<HeartbeatQueueDependencies["instanceSettings"]["getExperimental"]>>);
// Simulate A → B → A after the optimistic receipt read. The agent ID is
// unchanged, but the original assignment no longer authorizes a new turn.
deps.budgets.getInvocationBlock.mockImplementationOnce(async () => {
await db.update(issues).set({ statusVersion: issue.statusVersion + 2 }).where(eq(issues.id, issue.id));
return null;
});
expect(await createHeartbeatQueue(db, deps).claimQueuedRun({ ...run, responsibleUserId: "queue-test-user" })).toBeNull();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0])
.toMatchObject({ status: "queued", startedAt: null });
expect(deps.executeRun).not.toHaveBeenCalled();
});
it.each(["legacy", "native"])("periodic recovery preserves a paused recipient through the real %s release adapter", async runtimeMode => {
const seeded = await strandedMention(runtimeMode);
const { company, issue, recipient } = seeded;
const messages = await addStrandedMessages(seeded);
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, recipient.id));
await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, issue.id));
const deps = callbacks(db);
const release = createReleaseIssueExecution({ issueLock: createPostgresWakeQueueAdapter(db, {
resolveResponsibleUserId: async () => "responsible-user",
getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }),
resolveSessionBeforeForWakeup: async () => null,
}), recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined } });
const controlDeps = runControlCallbacks(db);
vi.mocked(controlDeps.wakeQueue.releaseIssueExecution).mockImplementation(release);
const control = createHeartbeatRunControl(db, controlDeps);
deps.releaseIssueExecutionAndPromote.mockImplementation(control.releaseIssueExecutionAndPromote);
for (let pass = 0; pass < 2; pass++) {
await createHeartbeatQueue(db, deps).resumeQueuedRuns();
await expectSavedMessages(messages);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, company.id))).toHaveLength(1);
}
expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledTimes(runtimeMode === "native" ? 2 : 6);
expect(deps.executeRun).not.toHaveBeenCalled();
});
it.each(["legacy", "native"])("resumes an accepted non-assignee queue through its %s owner", async runtimeMode => {
const { run, pending } = await strandedMention(runtimeMode);
await db.update(heartbeatRuns).set({ contextSnapshot: { ...run.contextSnapshot, prompt: "large context".repeat(20_000) },
runnerProfileJson: { instructions: "large profile".repeat(20_000) }, stdoutExcerpt: "large output".repeat(20_000),
}).where(eq(heartbeatRuns.id, run.id));
const deps = callbacks(db);
await createHeartbeatQueue(db, deps).resumeQueuedRuns();
expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledExactlyOnceWith(
expect.objectContaining({ id: run.id }), { suppressImmediateRecovery: true, resumeStrandedQueue: true },
);
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, pending.id)))[0]).toMatchObject({
status: "deferred_issue_execution", runId: null, payload: pending.payload,
});
expect(deps.executeRun).not.toHaveBeenCalled();
const selected = deps.releaseIssueExecutionAndPromote.mock.calls[0][0];
expect(Buffer.byteLength(JSON.stringify(selected))).toBeLessThan(512);
expect(selected).not.toHaveProperty("contextSnapshot");
expect(selected).not.toHaveProperty("runnerProfileJson");
expect(selected).not.toHaveProperty("stdoutExcerpt");
});
it("preserves a former owner's ordinary messages for the assignment handoff", async () => {
const { pending } = await strandedMention();
const payload = { ...pending.payload, _paperclipWakeContext: {
...(pending.payload!._paperclipWakeContext as Record<string, unknown>), wakeReason: "issue_commented",
} };
await db.update(agentWakeupRequests).set({ payload }).where(eq(agentWakeupRequests.id, pending.id));
const deps = callbacks(db);
await createHeartbeatQueue(db, deps).resumeQueuedRuns();
expect(deps.releaseIssueExecutionAndPromote).not.toHaveBeenCalled();
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, pending.id)))[0])
.toMatchObject({ status: "deferred_issue_execution", runId: null, payload });
expect(deps.executeRun).not.toHaveBeenCalled();
});
it.each(["queued", "running", "scheduled_retry"])("does not replay a finishing owner while another agent has a %s turn", async status => {
const { company, issue, recipient } = await strandedMention();
await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: recipient.id, status, invocationSource: "assignment", contextSnapshot: { issueId: issue.id } });
const deps = callbacks(db);
deps.getAgent.mockResolvedValue({ ...recipient, status: "paused" });
await createHeartbeatQueue(db, deps).resumeQueuedRuns();
expect(deps.releaseIssueExecutionAndPromote).not.toHaveBeenCalled();
expect(deps.executeRun).not.toHaveBeenCalled();
});
it.each(["claimed", "cancelled"] as const)("releases a running claim while preserving a %s wake", async (wakeStatus) => {
const { run, issue, wake } = await fixture();
await db.update(agentWakeupRequests).set({ status: wakeStatus }).where(eq(agentWakeupRequests.id, wake.id));
await createHeartbeatQueue(db, callbacks(db)).releaseRunClaimedJustBeforeSuppression(run.id);
const [released] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
const [releasedIssue] = await db.select().from(issues).where(eq(issues.id, issue.id));
const [releasedWake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id));
expect(released).toMatchObject({ status: "queued", startedAt: null, responsibleUserId: null });
expect(releasedIssue).toMatchObject({ executionRunId: null, executionAgentNameKey: null, executionLockedAt: null });
expect(releasedWake.status).toBe(wakeStatus === "cancelled" ? "cancelled" : "queued");
if (wakeStatus === "claimed") expect(releasedWake.claimedAt).toBeNull();
});
it("preserves a newer execution owner when releasing an old claim", async () => {
const { run, issue, agent, company } = await fixture();
const [newOwner] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, status: "running", invocationSource: "assignment" }).returning();
await db.update(issues).set({ executionRunId: newOwner.id }).where(eq(issues.id, issue.id));
await createHeartbeatQueue(db, callbacks(db)).releaseRunClaimedJustBeforeSuppression(run.id);
const [current] = await db.select().from(issues).where(eq(issues.id, issue.id));
expect(current.executionRunId).toBe(newOwner.id);
});
it.each(["running", "failed"])("resumes the %s native owner before promoting its queued waiter", async status => {
const { run, issue, agent, company } = await fixture(status);
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issue.id, runnerProfileJson: {
nativeExecutionInput: { binding: { runId: run.id } },
sessionCheckpoint: { providerSessionId: "original-session" },
} }).where(eq(heartbeatRuns.id, run.id));
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: company.id,
issueId: issue.id, phase: "retryable_failure", attempt: 1 });
const [waiter] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
status: "queued", invocationSource: "assignment", responsibleUserId: "queue-test-user", contextSnapshot: { issueId: issue.id },
}).returning();
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(agent);
deps.instanceSettings.getExperimental.mockResolvedValue({ enableNativeRunner: true } as Awaited<ReturnType<HeartbeatQueueDependencies["instanceSettings"]["getExperimental"]>>);
const queue = createHeartbeatQueue(db, deps);
expect(await queue.claimQueuedRun(waiter)).toBeNull();
const claims = await claimNativeSessionResumptions({ db, runnerInstanceId: "owner", runIds: [run.id] });
expect(claims.map(claim => claim.runId)).toEqual([run.id]);
expect(await queue.claimQueuedRun(waiter)).toBeNull();
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, run.id));
await db.update(nativeRunFinalizations).set({ phase: "committed", leaseOwner: null, leaseExpiresAt: null })
.where(eq(nativeRunFinalizations.runId, run.id));
await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, issue.id));
expect(await queue.claimQueuedRun(waiter)).toMatchObject({ id: waiter.id, status: "running" });
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0].executionRunId).toBe(waiter.id);
});
it.each(["direct", "batch"] as const)("keeps fresh work queued across a held external connection on the %s path", async path => {
const { run, agent, issue, wake } = await fixture("queued");
await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user" }).where(eq(heartbeatRuns.id, run.id));
const deps = { ...callbacks(db), canStartNewExternalBotWork: vi.fn(async () => false) };
deps.getAgent.mockResolvedValue({ ...agent, adapterType: "paperclip_runner", adapterConfig: { provider: "grok_bot" } });
const queue = createHeartbeatQueue(db, deps);
for (let attempt = 0; attempt < 3; attempt++) {
if (path === "direct") expect(await queue.claimQueuedRun({ ...run, responsibleUserId: "queue-test-user" })).toBeNull();
else expect(await queue.startNextQueuedRunForAgent(agent.id)).toEqual([]);
}
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toMatchObject({ status: "queued", startedAt: null });
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0].status).toBe(wake.status);
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0].status).toBe("todo");
expect(deps.executeRun).not.toHaveBeenCalled();
expect(deps.setRunStatusFromLive).not.toHaveBeenCalled();
});
it("admits the same queued input after the external connection is repaired", async () => {
const { run, agent } = await fixture("queued");
await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user" }).where(eq(heartbeatRuns.id, run.id));
const deps = { ...callbacks(db), canStartNewExternalBotWork: vi.fn(async () => false) };
deps.getAgent.mockResolvedValue(agent);
const queue = createHeartbeatQueue(db, deps);
expect(await queue.startNextQueuedRunForAgent(agent.id)).toEqual([]);
deps.canStartNewExternalBotWork.mockResolvedValue(true);
expect(await queue.startNextQueuedRunForAgent(agent.id)).toEqual([expect.objectContaining({ id: run.id, status: "running" })]);
expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(run.id);
});
it("starts each claimed run before claiming the next, within the agent's concurrency cap", async () => {
// A claim commits a 60 s legacy controller lease that only the run's own
// execution renews. A batch whose later claims are slow must not hold the
// earlier claimed runs back from executing until their leases lapse.
const [company] = await db.insert(companies).values({ name: "Claim batch", issuePrefix: "CB" }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Busy agent", role: "engineer", status: "idle",
adapterType: "process", adapterConfig: {}, runtimeConfig: { heartbeat: { maxConcurrentRuns: 2 } } }).returning();
const queued = [];
for (let index = 0; index < 3; index++) {
const [issue] = await db.insert(issues).values({ companyId: company.id, title: `Batch task ${index}`, status: "todo",
assigneeAgentId: agent.id }).returning();
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: company.id, agentId: agent.id, source: "assignment",
status: "claimed", claimedAt: new Date() }).returning();
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, status: "queued",
invocationSource: "assignment", wakeupRequestId: wake.id, responsibleUserId: "queue-test-user", contextSnapshot: { issueId: issue.id },
createdAt: new Date(Date.now() - (3 - index) * 1_000) }).returning();
await db.update(issues).set({ executionRunId: run.id, executionAgentNameKey: agent.name, executionLockedAt: new Date() })
.where(eq(issues.id, issue.id));
queued.push(run);
}
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(agent);
let releaseSecondClaim!: () => void;
const secondClaimMayContinue = new Promise<void>(resolve => { releaseSecondClaim = resolve; });
let budgetChecks = 0;
deps.budgets.getInvocationBlock.mockImplementation(async () => {
budgetChecks += 1;
if (budgetChecks === 2) await secondClaimMayContinue;
return null;
});
const admission = vi.fn(() => false);
deps.isRunAdmissionClosed = admission;
const pass = createHeartbeatQueue(db, deps).startNextQueuedRunForAgent(agent.id);
try {
await vi.waitFor(() => expect(budgetChecks).toBe(2));
// The first run is executing while the second claim is still in progress.
expect(deps.executeRun).toHaveBeenCalledExactlyOnceWith(queued[0]!.id);
} finally { releaseSecondClaim(); }
const claimed = await pass;
expect(claimed.map(run => run.id)).toEqual([queued[0]!.id, queued[1]!.id]);
expect(deps.executeRun.mock.calls.map(([runId]) => runId)).toEqual([queued[0]!.id, queued[1]!.id]);
// The run-start admission gate is checked before every claim, and the cap holds.
expect(admission.mock.calls.length).toBeGreaterThanOrEqual(3);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, queued[2]!.id)))[0].status).toBe("queued");
});
it("leaves a queued run for the next server when shutdown closes admission during the queue scan", async () => {
const { run, agent, wake } = await fixture("queued");
await db.update(heartbeatRuns).set({ responsibleUserId: "queue-test-user" }).where(eq(heartbeatRuns.id, run.id));
let admissionClosed = false;
const deps = callbacks(db);
deps.isRunAdmissionClosed = () => admissionClosed;
deps.getAgent.mockImplementation(async () => { admissionClosed = true; return agent; });
expect(await createHeartbeatQueue(db, deps).startNextQueuedRunForAgent(agent.id)).toEqual([]);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toMatchObject({ status: "queued", startedAt: null });
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0].status).toBe(wake.status);
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("rechecks the connection hold when Stop races the queue scan", async () => {
const { run, agent } = await fixture("queued");
const deps = { ...callbacks(db), canStartNewExternalBotWork: vi.fn(async () => false) };
deps.getAgent.mockResolvedValue(agent);
deps.canStartNewExternalBotWork.mockResolvedValueOnce(true);
expect(await createHeartbeatQueue(db, deps).startNextQueuedRunForAgent(agent.id)).toEqual([]);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toMatchObject({ status: "queued", startedAt: null });
expect(deps.canStartNewExternalBotWork).toHaveBeenCalledTimes(2);
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("keeps admitted resumptions outside the fresh external connection gate", async () => {
const { run, agent } = await fixture("queued");
await db.update(heartbeatRuns).set({ startedAt: new Date(), responsibleUserId: "queue-test-user" }).where(eq(heartbeatRuns.id, run.id));
const [admitted] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
const deps = { ...callbacks(db), canStartNewExternalBotWork: vi.fn(async () => false) };
deps.getAgent.mockResolvedValue(agent);
expect(await createHeartbeatQueue(db, deps).claimQueuedRun(admitted)).toMatchObject({ id: run.id, status: "running" });
expect(deps.canStartNewExternalBotWork).not.toHaveBeenCalled();
});
it("keeps a claim queued while another task of its routine holds the execution slot", async () => {
const { company, agent, issue, run } = await fixture("queued");
const origin = { originKind: "routine_execution", originId: crypto.randomUUID(), originFingerprint: "scheduled" };
await db.update(issues).set({ ...origin, executionRunId: null, executionAgentNameKey: null, executionLockedAt: null })
.where(eq(issues.id, issue.id));
// An earlier task of the same routine keeps its lock while its failed run waits for a scheduled resume.
const [holder] = await db.insert(issues).values({ companyId: company.id, title: "Earlier routine task",
status: "in_progress", assigneeAgentId: agent.id, ...origin }).returning();
const [holderRun] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
status: "failed", finishedAt: new Date(), contextSnapshot: { issueId: holder.id } }).returning();
await db.update(issues).set({ executionRunId: holderRun.id }).where(eq(issues.id, holder.id));
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(agent);
await expect(createHeartbeatQueue(db, deps).claimQueuedRun({ ...run, responsibleUserId: "queue-test-user" }))
.rejects.toMatchObject({ status: 409 });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]).toMatchObject({ status: "queued" });
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0]).toMatchObject({ executionRunId: null });
expect((await db.select().from(issues).where(eq(issues.id, holder.id)))[0]).toMatchObject({ executionRunId: holderRun.id });
expect(deps.executeRun).not.toHaveBeenCalled();
});
it("cancels at the daily run cap before claiming or dispatching", async () => {
const { run, agent } = await fixture("queued");
const cappedAgent = { ...agent, runtimeConfig: { heartbeat: { maxDailyRuns: 0 } } };
const deps = callbacks(db);
deps.getAgent.mockResolvedValue(cappedAgent);
expect(await createHeartbeatQueue(db, deps).claimQueuedRun(run)).toBeNull();
const [cancelled] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
expect(cancelled).toMatchObject({ status: "cancelled", errorCode: "heartbeat.daily_run_limit" });
expect(deps.setWakeupStatus).toHaveBeenCalledWith(run.wakeupRequestId, "skipped", expect.any(Object));
expect(deps.releaseIssueExecutionAndPromote).toHaveBeenCalledWith(cancelled, { suppressImmediateRecovery: true });
expect(deps.treeControlSvc.getActivePauseHoldGate).not.toHaveBeenCalled();
expect(deps.executeRun).not.toHaveBeenCalled();
});
});