mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts an agent's queued runs when a slot is free. #21234, #21277, #21349 and #21391 order them: person work first, a recovery share, aging, and a share for the longest-waiting run > - On hosted staging, one agent with 20 slots had 295 queued runs. 139 of them were system recovery wakes that entered the queue within one hour. Runs that people asked for waited 7 hours > - Two rules let this happen. Recovery took every fourth start even when a person's run waited, and an aged recovery run moved into the agent-work tier ahead of todo tasks. The continuation producer also admitted any continuation held past 5 minutes, however deep the queue was > - This pull request keeps recovery behind people's work, gives a person's long-waiting run a bounded start, re-checks a recovery run before it starts, and admits recovery to an agent's queue a few runs at a time > - The benefit is that a task a person asked for starts within a bounded number of dispatches, and a recovery burst cannot flood an agent's queue ## Linked Issues or Issue Description Refs #21234, #21277, #21349, #21391, #21405, #21433, #21502 **What happened?** On hosted staging (2026-11-13), one agent had `maxConcurrentRuns` 20 and all 20 slots held hour-long runs. 295 runs were queued. 139 had wake reason `issue_recovery_action_restored` (source `execution.reconciled`), enqueued between about 10:46Z and 11:47Z. Two tasks that people created were stalled by a restart at 07:25Z. Their next runs were queued at 07:29Z and were still queued 7 hours later. Both were system wakes: one `issue_monitor_due`, one `issue_continuation_needed` from the assigned-work sweep. Recent starts had waited from 1 to 290 minutes. **Why did it happen?** 1. The recovery share gave every fourth start to system recovery, also while a person's run waited. 2. A recovery run that waited one hour aged into the agent-work tier. Recovery tasks are usually in progress, so they sorted ahead of todo tasks in that tier. 3. The oldest-run share served the oldest aged run of any class. Older agent work and older recovery came before a person's run. 4. Only a wake that a person requested counted as person work. A system wake on a task that a person created and waits on (a monitor check, a continuation after a restart) competed as ordinary work or recovery. 5. `deliverReconciledExecutions` admitted a continuation held past `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) whatever the queue depth. At one per agent per 15-second pass, one agent got about 139 continuations in an hour. **Expected behavior** A run that a person created or asked for never waits behind system recovery. Waiting never fails or cancels it. A long wait moves it forward, so it starts within a bounded number of dispatches. Recovery enters the queue a few runs at a time. A recovery run that is no longer needed is dropped quietly. **Steps to reproduce** 1. Give an agent 20 slots, all busy. 2. Queue 60 agent assignments, then two system wakes on tasks people created (a monitor check and a continuation), then 30 later person runs on in-progress tasks, then a burst of 139 recovery wakes. 3. Free one slot at a time while people and agents keep adding work. 4. See that the monitor check starts only at dispatch 244. ## What Changed - `server/src/services/heartbeat/queued-run-dispatch-order.ts`: - System recovery has its own tiers below agent work (fresh, then backlog). Aging moves old backlog only into the fresh recovery tier, never ahead of agent work. - The recovery share applies only when no ready person run waits. It comes out of agent work, never out of a person's turn. Within the share, the longest-waiting aged recovery run goes first. - The oldest-run share serves the longest-waiting aged run that a person asked for or whose task a person created (`issues.created_by_user_id`), first. This includes a system continuation of such a task. Then it serves the longest-waiting aged agent-work run. It never serves other recovery. - New dispatch fact `personTask`, read with the queued tasks' status and priority (same query). - `queuedRunDispatchLine` (the O(n log n) place-in-line order from #21405) follows the same rules. `queued-run-line.test.ts` still proves it equals the dispatcher's order over 400 random queues, now with `personTask` in them. - `server/src/services/heartbeat/queued-run-line.ts`: the start-history counters use `dispatchHistoryStart` and `dispatchCountersFromStarts`, so dispatch, the place-in-line view and the tests share one rule. - New pure helpers `oldestShareCandidate`, `dispatchHistoryStart` and `dispatchCountersFromStarts`. The queue and the tests use the same counter rule (which includes the #21391 rule that a person's recovery restart does not use up the recovery share). - `server/src/services/heartbeat/queue.ts`: - At claim, a system recovery run (`issue_recovery_action_restored`, `issue_continuation_needed`, not requested by a person) is re-checked. If the task is reassigned, or another run continues it (started after it was queued, running, a queued non-recovery run, or a newer queued recovery run), the run is cancelled before it starts with `recovery_no_longer_needed` and its wake is skipped. This is the same quiet path as the existing closed-task check: no failure, no provider work. A person's undelivered message on the run still keeps it. - `server/src/services/recovery-queue-admission.ts` (new): `QUEUED_RECOVERY_RUNS_PER_AGENT` = 3 and the queue-depth check. - `server/src/services/execution-recovery-resolution.ts`: a continuation is admitted only while its agent has fewer than 3 queued recovery runs, fresh or backlog. The 5-minute hold bound still orders held continuations, but no longer overrides the cap. The rest stays pending in its recovery action. The existing continuation pass (every 15 s while any continuation is pending) admits the next one after a queued recovery run starts or is dropped. No new polling. - `server/src/services/recovery/legacy-workspace-rehome.ts`: the pending-wake delivery for rehomed and released-writer tasks uses the same cap. A held wake waits 60 s for its next turn and does not count as a refusal. - The oldest-run share counts as served only by a start that people wait on while its candidate is one, so older agent work never uses up a person's turn (`servesOldestShare`, used by the dispatcher, the place-in-line order and the start history, which now records `personWaiting`). - `server/src/services/recovery/assigned-work-sweep.ts`: a rehome or released-writer wake held by the cap (`HELD_RECOVERY_WAKE_KEYS`) counts as the task's pending path while its assignee has queued recovery runs. - `server/src/services/recovery/assigned-work-sweep.ts` (with #21502's stalled-queued-run replacement): a start of work people wait on does not count as passing over a queued run that people do not wait on. Waiting behind people's work is intended, so the sweep does not cancel such a run and replace it with a recovery continuation. Test: `assigned-work-stalled-queued-run.test.ts`, "leaves an automatic run that waits behind work on tasks people created". It fails without the rule. - `server/src/services/recovery/assigned-work-sweep.ts`: a pending continuation still counts as the task's wake path past `PENDING_CONTINUATION_BOUND_MS` while its agent has queued recovery runs, so the sweep does not treat a held task as stranded. **Bounds** - A run a person asked for, or any run on a task a person created: once it has waited past `QUEUED_RUN_AGING_MS` (1 hour), it starts within `OLDEST_DISPATCH_SHARE` × (such aged runs queued before it + 1) dispatches. Before that, runs a person requested already go first. There is no time limit, and waiting never cancels it. - Recovery run, proposed bounded behavior: at most 3 queued per agent. When no person run waits, an aged recovery run starts within `RECOVERY_DISPATCH_SHARE` × (aged recovery runs before it + 1) dispatches. At start it is re-validated against the current task, and it is dropped quietly if stale. While people keep the agent busy, recovery waits. It is held durably in its recovery action, not lost and not failed. ## Verification Regression test that reproduces today's shape (`server/src/services/heartbeat/queued-run-dispatch-order.test.ts`, "deep per-agent queue: a recovery burst behind 20 busy slots"). It uses 20 busy slots, one dispatch per freed slot, 60 older agent assignments, the two 07:29 system wakes on tasks people created (monitor check and continuation), 30 later person runs, a 139-run recovery burst, and new person and assignment arrivals on every step. The same model, run against master's dispatcher and this one over 600 dispatches: | | Before (master) | After | | --- | --- | --- | | Dispatch at which the 07:29 monitor check starts | 244 (≈12 h at ~20 starts/h) | 4 | | Dispatch at which the 07:29 continuation starts | 5 | 8 | | Recovery starts while a person-requested run waited | 89 | 0 | With person-requested runs in the 07:29 slots instead (the first version of the test), master starts them at dispatch 244 and 248, and this change at 4 and 8. - Red on master: the new dispatch tests fail (`expected -1 to be greater than or equal to 0`, and recovery started ahead of agent assignments). Green with this change. - Admission cap, embedded Postgres (`recovery-wake-herd.test.ts`, "queues at most a few recovery runs per agent..."): with the cap removed, 12 of 12 continuations enter one agent's queue in 12 passes. With the cap, 3 enter, the rest stay pending, and all 12 are delivered as queued runs leave. The queue never holds more than 3 recovery runs. - Older agent work (`queued-run-dispatch-order.test.ts`): with 60 older, higher-priority assignments and no person arrivals, an aged `issue_monitor_due` run on a person-created task starts within `OLDEST_DISPATCH_SHARE` in dispatch and in the place-in-line order. With the old reset rule it does not start in 8 dispatches. - Held restored-task wake (`legacy-workspace-rehome.test.ts`): with the agent's recovery queue full, the wake stays pending with no failure counted, the sweep reads the task as `pending_intent`, and the wake is delivered after a queued recovery run leaves. - Start-time re-check, embedded Postgres, real heartbeat queue (`heartbeat-stale-queue-invalidation.test.ts`): a recovery run whose task was reassigned, or that a newer run already continued, is cancelled with `recovery_no_longer_needed`, its wake is `skipped`, and the adapter is never called. A recovery run that the task still needs starts. - Updated tests that encoded the old rules: recovery's share and aging now come out of agent work, not out of a person's turn. Sweep liveness while held is covered. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: passes. ## Risks - Behavior change: a system run on a task a person created now takes the oldest-run share ahead of older agent work. On an agent whose queue is mostly such tasks, agent-made subtasks get that share less often. They still start in normal order. - Behavior change: while people keep an agent fully busy, system recovery for that agent waits. This is intended. Recovery is held durably and starts once no person run waits. - Behavior change: a queued system recovery run can be cancelled at start as no longer needed. It is cancelled before any provider work, with the same quiet outcome as a continuation whose task closed. - A continuation held behind the cap stays pending longer than 7 minutes. The sweep now treats it as live while its agent has queued recovery runs. If such queued runs can never start (for example dependency-blocked), the next continuations wait behind them. - One extra indexed read per recovery claim, on the existing `(company_id, context_snapshot->>'issueId', created_at)` index. No schema change. Mixed versions are safe: an old server keeps its old order until it is replaced. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edit). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2071 lines
80 KiB
TypeScript
2071 lines
80 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { setImmediate as immediate } from "node:timers/promises";
|
|
import * as runState from "../services/heartbeat/run-state.js";
|
|
import { eq, sql } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
agents,
|
|
authUsers,
|
|
agentWakeupRequests,
|
|
agentRuntimeState,
|
|
companies,
|
|
companyMemberships,
|
|
costEvents,
|
|
createDb,
|
|
applyPendingMigrations,
|
|
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
|
|
documentRevisions,
|
|
documents,
|
|
environmentLeases,
|
|
nativeRunFinalizations,
|
|
heartbeatRuns,
|
|
issueComments,
|
|
issueDocuments,
|
|
issueThreadInteractions,
|
|
issues,
|
|
} from "@paperclipai/db";
|
|
import { ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY } from "@paperclipai/shared";
|
|
import {
|
|
getEmbeddedPostgresTestSupport,
|
|
startEmbeddedPostgresTestDatabase,
|
|
} from "./helpers/embedded-postgres.js";
|
|
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
|
import { drainHeartbeatRunsToQuiescence } from "./helpers/drain-heartbeat-runs.js";
|
|
import { truncateTablesWithDeadlockRetry } from "./helpers/truncate-with-deadlock-retry.js";
|
|
import {
|
|
MAX_TURN_CONTINUATION_RETRY_REASON,
|
|
MAX_TURN_CONTINUATION_WAKE_REASON,
|
|
heartbeatService,
|
|
} from "../services/heartbeat.ts";
|
|
import { runningProcesses, type AdapterExecutionContext, type AdapterExecutionResult } from "../adapters/index.ts";
|
|
import { recoveryService } from "../services/recovery/service.ts";
|
|
import { withQueuedCommentIdsInWakePayload } from "../services/issue-queued-comment-queue.ts";
|
|
|
|
const mockAdapterExecute = vi.hoisted(() =>
|
|
vi.fn(async (_context: AdapterExecutionContext): Promise<AdapterExecutionResult> => ({
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
errorMessage: null,
|
|
summary: "Stale-queue invalidation test run.",
|
|
provider: "test",
|
|
model: "test-model",
|
|
})),
|
|
);
|
|
|
|
vi.mock("../adapters/index.ts", async () => {
|
|
const actual = await vi.importActual<typeof import("../adapters/index.ts")>("../adapters/index.ts");
|
|
return {
|
|
...actual,
|
|
getServerAdapter: vi.fn(() => ({
|
|
supportsLocalAgentJwt: false,
|
|
execute: mockAdapterExecute,
|
|
})),
|
|
};
|
|
});
|
|
|
|
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_STALE_QUEUE_TEST_DATABASE_URL");
|
|
const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
|
|
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
|
|
|
if (!embeddedPostgresSupport.supported) {
|
|
console.warn(
|
|
`Skipping embedded Postgres heartbeat stale-queue invalidation tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
|
);
|
|
}
|
|
|
|
async function ensureIssueRelationsTable(db: ReturnType<typeof createDb>) {
|
|
await db.execute(sql.raw(`
|
|
CREATE TABLE IF NOT EXISTS "issue_relations" (
|
|
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
"company_id" uuid NOT NULL,
|
|
"issue_id" uuid NOT NULL,
|
|
"related_issue_id" uuid NOT NULL,
|
|
"type" text NOT NULL,
|
|
"created_by_agent_id" uuid,
|
|
"created_by_user_id" text,
|
|
"created_at" timestamptz NOT NULL DEFAULT now(),
|
|
"updated_at" timestamptz NOT NULL DEFAULT now()
|
|
);
|
|
`));
|
|
}
|
|
|
|
async function waitForCondition(fn: () => Promise<boolean>, timeoutMs = 3_000) {
|
|
const deadline = Date.now() + timeoutMs;
|
|
while (Date.now() < deadline) {
|
|
if (await fn()) return true;
|
|
await new Promise((resolve) => setTimeout(resolve, 50));
|
|
}
|
|
return fn();
|
|
}
|
|
|
|
async function cleanupHeartbeatInvalidationFixture(db: ReturnType<typeof createDb>) {
|
|
await truncateTablesWithDeadlockRetry(
|
|
db,
|
|
`
|
|
TRUNCATE TABLE
|
|
"company_skills",
|
|
"issue_comments",
|
|
"issue_documents",
|
|
"document_revisions",
|
|
"documents",
|
|
"issue_relations",
|
|
"issue_tree_holds",
|
|
"issues",
|
|
"heartbeat_run_events",
|
|
"cost_events",
|
|
"activity_log",
|
|
"heartbeat_runs",
|
|
"agent_wakeup_requests",
|
|
"agent_runtime_state",
|
|
"agents",
|
|
"companies"
|
|
RESTART IDENTITY CASCADE
|
|
`,
|
|
{ attempts: 10, delayMs: () => 100 },
|
|
);
|
|
}
|
|
|
|
type SeedOptions = {
|
|
agentName?: string;
|
|
agentRole?: string;
|
|
maxConcurrentRuns?: number;
|
|
heartbeatConfig?: Record<string, unknown>;
|
|
};
|
|
|
|
type SeedResult = {
|
|
companyId: string;
|
|
agentId: string;
|
|
};
|
|
|
|
describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => {
|
|
let db!: ReturnType<typeof createDb>;
|
|
let heartbeat!: ReturnType<typeof heartbeatService>;
|
|
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
let beforeContinuationDispatchCheck:
|
|
| ((input: { runId: string; issueId: string }) => Promise<void>)
|
|
| null = null;
|
|
let afterContinuationDispatchCheck:
|
|
| ((input: { runId: string; issueId: string }) => Promise<void>)
|
|
| null = null;
|
|
|
|
const countExecuteCallsForRun = (runId: string) =>
|
|
mockAdapterExecute.mock.calls.filter(([context]) => context?.runId === runId).length;
|
|
|
|
beforeAll(async () => {
|
|
if (external) await applyPendingMigrations(external);
|
|
else tempDb = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-stale-queue-");
|
|
db = createDb(external ?? tempDb!.connectionString);
|
|
heartbeat = heartbeatService(db, {
|
|
runtimeEnv: { ...process.env, PAPERCLIP_IN_WORKTREE: "false" },
|
|
beforeResolvedInteractionContinuationDispatchCheck: async (input) => {
|
|
await beforeContinuationDispatchCheck?.(input);
|
|
},
|
|
afterResolvedInteractionContinuationDispatchCheck: async (input) => {
|
|
await afterContinuationDispatchCheck?.(input);
|
|
},
|
|
});
|
|
await ensureIssueRelationsTable(db);
|
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
|
|
|
afterEach(async () => {
|
|
beforeContinuationDispatchCheck = null;
|
|
afterContinuationDispatchCheck = null;
|
|
mockAdapterExecute.mockReset();
|
|
mockAdapterExecute.mockImplementation(async () => ({
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
errorMessage: null,
|
|
summary: "Stale-queue invalidation test run.",
|
|
provider: "test",
|
|
model: "test-model",
|
|
}));
|
|
runningProcesses.clear();
|
|
// A table poll for "queued" or "running" rows cannot see a wakeup promise
|
|
// that has not yet written its run row. The shared drain helper awaits
|
|
// both in-flight wakeup promises and in-flight run executions first, then
|
|
// re-checks the run table, so a late writer cannot still be in flight when
|
|
// the TRUNCATE below runs.
|
|
await drainHeartbeatRunsToQuiescence(db, heartbeat);
|
|
await cleanupHeartbeatInvalidationFixture(db);
|
|
await db.delete(authUsers).where(eq(authUsers.id, "responsible-user"));
|
|
});
|
|
|
|
afterAll(async () => {
|
|
await tempDb?.cleanup();
|
|
});
|
|
|
|
async function seedCompanyAndAgent(opts: SeedOptions = {}): Promise<SeedResult> {
|
|
const companyId = randomUUID();
|
|
const agentId = randomUUID();
|
|
await db.insert(companies).values({
|
|
id: companyId,
|
|
name: "Paperclip",
|
|
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
|
defaultResponsibleUserId: "responsible-user",
|
|
requireBoardApprovalForNewAgents: false,
|
|
});
|
|
// Admission intersects agent authority with an existing responsible user
|
|
// and that user's active company membership.
|
|
await db.insert(authUsers).values({
|
|
id: "responsible-user",
|
|
name: "Fixture user",
|
|
email: "responsible-user@example.test",
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
}).onConflictDoNothing();
|
|
await db.insert(companyMemberships).values({
|
|
companyId,
|
|
principalType: "user",
|
|
principalId: "responsible-user",
|
|
status: "active",
|
|
membershipRole: "member",
|
|
});
|
|
await db.insert(agents).values({
|
|
id: agentId,
|
|
companyId,
|
|
name: opts.agentName ?? "ClaudeCoder",
|
|
role: opts.agentRole ?? "engineer",
|
|
status: "active",
|
|
adapterType: "codex_local",
|
|
adapterConfig: {},
|
|
runtimeConfig: {
|
|
heartbeat: {
|
|
wakeOnDemand: true,
|
|
maxConcurrentRuns: opts.maxConcurrentRuns ?? 1,
|
|
...(opts.heartbeatConfig ?? {}),
|
|
},
|
|
},
|
|
permissions: {},
|
|
});
|
|
return { companyId, agentId };
|
|
}
|
|
|
|
it.each(["legacy", "native"])("delivers an accepted non-assignee queue after its %s owner finished", async runtimeMode => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const [recipient] = await db.insert(agents).values({ companyId, name: "Saved feedback recipient", role: "engineer",
|
|
status: "idle", adapterType: "codex_local", adapterConfig: {}, runtimeConfig: { heartbeat: { wakeOnDemand: true } }, permissions: {},
|
|
}).returning();
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Existing accepted feedback", status: "in_progress", assigneeAgentId: agentId }).returning();
|
|
const [owner] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "assignment", status: "succeeded",
|
|
runtimeMode, nativeIssueId: runtimeMode === "native" ? issue.id : null, startedAt: new Date(), finishedAt: new Date(), contextSnapshot: { issueId: issue.id },
|
|
}).returning();
|
|
if (runtimeMode === "native") await db.insert(nativeRunFinalizations).values({ runId: owner.id, companyId, issueId: issue.id, phase: "committed" });
|
|
const [comment] = await db.insert(issueComments).values({ companyId, issueId: issue.id, body: "Saved feedback for the recipient",
|
|
authorAgentId: agentId, createdByRunId: owner.id,
|
|
}).returning();
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId, agentId: recipient.id, source: "automation",
|
|
reason: "issue_execution_deferred", status: "deferred_issue_execution", requestedByActorType: "agent", requestedByActorId: agentId,
|
|
payload: withQueuedCommentIdsInWakePayload({ issueId: issue.id, _paperclipWakeContext: {
|
|
issueId: issue.id, wakeReason: "issue_comment_mentioned", wakeSource: "automation",
|
|
} }, [comment.id]),
|
|
}).returning();
|
|
await heartbeat.resumeQueuedRuns();
|
|
const [delivered] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id));
|
|
expect(delivered.runId).toBeTruthy();
|
|
// The receipt can commit before execution enters liveRunExecutions.
|
|
// Observe actual adapter admission before awaiting its drain.
|
|
await vi.waitFor(async () => {
|
|
const current = await heartbeat.getRun(delivered.runId!);
|
|
expect(countExecuteCallsForRun(delivered.runId!), JSON.stringify({
|
|
status: current?.status, errorCode: current?.errorCode,
|
|
})).toBe(1);
|
|
}, { timeout: 5_000 });
|
|
await heartbeat.waitForRunExecutionDrain(delivered.runId!);
|
|
expect(countExecuteCallsForRun(delivered.runId!)).toBe(1);
|
|
expect((await heartbeat.getRun(delivered.runId!))?.contextSnapshot).toMatchObject({ wakeCommentIds: [comment.id] });
|
|
await heartbeat.resumeQueuedRuns();
|
|
expect(countExecuteCallsForRun(delivered.runId!)).toBe(1);
|
|
expect((await db.select().from(issueComments).where(eq(issueComments.id, comment.id)))[0]?.body).toBe(comment.body);
|
|
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0]?.assigneeAgentId).toBe(agentId);
|
|
});
|
|
|
|
it.each(["preparing", "verifying", "resuming"] as const)("keeps assigned work queued through %s, including recovery with timers disabled", async lifecycleState => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({ heartbeatConfig: { enabled: false } });
|
|
await db.update(agents).set({ status: "paused", lifecycleState }).where(eq(agents.id, agentId));
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Work accepted during setup", status: "todo", assigneeAgentId: agentId }).returning();
|
|
const run = await heartbeat.wakeup(agentId, { source: "assignment", triggerDetail: "system", reason: "issue_assigned",
|
|
requestedByActorType: "system", payload: { issueId: issue.id } });
|
|
expect(run?.status).toBe("queued");
|
|
await heartbeat.resumeQueuedRuns();
|
|
expect((await heartbeat.getRun(run!.id))?.status).toBe("queued");
|
|
expect(countExecuteCallsForRun(run!.id)).toBe(0);
|
|
expect((await heartbeat.reconcileStrandedAssignedIssues()).escalated).toBe(0);
|
|
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0]).toMatchObject({ status: "todo", assigneeAgentId: agentId });
|
|
await db.update(agents).set({ status: "idle", lifecycleState: "ready" }).where(eq(agents.id, agentId));
|
|
const recovered = heartbeatService(db, { runtimeEnv: { ...process.env, PAPERCLIP_IN_WORKTREE: "false" } });
|
|
await recovered.resumeQueuedRuns();
|
|
await recovered.drainActiveRunExecutions();
|
|
expect(countExecuteCallsForRun(run!.id)).toBe(1);
|
|
});
|
|
|
|
async function seedQueuedRun(input: {
|
|
companyId: string;
|
|
agentId: string;
|
|
issueId: string;
|
|
wakeReason: string;
|
|
contextExtras?: Record<string, unknown>;
|
|
invocationSource?: "assignment" | "automation";
|
|
scheduledRetryReason?: string | null;
|
|
}) {
|
|
const wakeupRequestId = randomUUID();
|
|
const runId = randomUUID();
|
|
await db.insert(agentWakeupRequests).values({
|
|
id: wakeupRequestId,
|
|
companyId: input.companyId,
|
|
agentId: input.agentId,
|
|
source: input.invocationSource ?? "assignment",
|
|
triggerDetail: "system",
|
|
reason: input.wakeReason,
|
|
payload: { issueId: input.issueId },
|
|
status: "queued",
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId: input.companyId,
|
|
agentId: input.agentId,
|
|
invocationSource: input.invocationSource ?? "assignment",
|
|
triggerDetail: "system",
|
|
status: "queued",
|
|
wakeupRequestId,
|
|
scheduledRetryReason: input.scheduledRetryReason ?? null,
|
|
contextSnapshot: {
|
|
issueId: input.issueId,
|
|
wakeReason: input.wakeReason,
|
|
...(input.contextExtras ?? {}),
|
|
},
|
|
});
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId })
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
return { runId, wakeupRequestId };
|
|
}
|
|
|
|
async function seedContinuationSummary(input: {
|
|
companyId: string;
|
|
issueId: string;
|
|
agentId: string;
|
|
body: string;
|
|
}) {
|
|
const documentId = randomUUID();
|
|
const revisionId = randomUUID();
|
|
await db.insert(documents).values({
|
|
id: documentId,
|
|
companyId: input.companyId,
|
|
title: "Continuation Summary",
|
|
format: "markdown",
|
|
latestBody: input.body,
|
|
latestRevisionId: revisionId,
|
|
latestRevisionNumber: 1,
|
|
createdByAgentId: input.agentId,
|
|
updatedByAgentId: input.agentId,
|
|
});
|
|
await db.insert(documentRevisions).values({
|
|
id: revisionId,
|
|
companyId: input.companyId,
|
|
documentId,
|
|
revisionNumber: 1,
|
|
title: "Continuation Summary",
|
|
format: "markdown",
|
|
body: input.body,
|
|
createdByAgentId: input.agentId,
|
|
});
|
|
await db.insert(issueDocuments).values({
|
|
companyId: input.companyId,
|
|
issueId: input.issueId,
|
|
documentId,
|
|
key: ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY,
|
|
});
|
|
}
|
|
|
|
it("joins a claimed execution whose initial run read has not reached live registration", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Held preparation", status: "todo", assigneeAgentId: agentId }).returning();
|
|
const queued = await seedQueuedRun({ companyId, agentId, issueId: issue.id, wakeReason: "issue_assigned" });
|
|
const entered = Promise.withResolvers<void>(), release = Promise.withResolvers<void>();
|
|
const factory = runState.createHeartbeatRunState;
|
|
let held = false;
|
|
const spy = vi.spyOn(runState, "createHeartbeatRunState").mockImplementation((...args) => {
|
|
const owner = factory(...args);
|
|
const read = owner.getRun;
|
|
return { ...owner, getRun: async (...readArgs: Parameters<typeof read>) => {
|
|
const row = await read(...readArgs);
|
|
if (!held && row?.id === queued.runId && row.status === "running") {
|
|
held = true; entered.resolve(); await release.promise;
|
|
}
|
|
return row;
|
|
} };
|
|
});
|
|
const recovered = heartbeatService(db, { runtimeEnv: { ...process.env, PAPERCLIP_IN_WORKTREE: "false" } });
|
|
try {
|
|
await recovered.resumeQueuedRuns();
|
|
await entered.promise;
|
|
// Receipt admission is earlier than live-run registration. The legacy
|
|
// run-specific drain can return here; the tracked execution must not.
|
|
await recovered.waitForRunExecutionDrain(queued.runId);
|
|
expect(countExecuteCallsForRun(queued.runId)).toBe(0);
|
|
let settled = false;
|
|
const draining = recovered.drainActiveRunExecutions().then(() => { settled = true; });
|
|
await immediate();
|
|
expect(settled).toBe(false);
|
|
release.resolve();
|
|
await draining;
|
|
expect(countExecuteCallsForRun(queued.runId)).toBe(1);
|
|
expect(await recovered.getRun(queued.runId)).toMatchObject({ status: "succeeded", runtimeMode: "legacy" });
|
|
} finally {
|
|
release.resolve();
|
|
await recovered.drainActiveRunExecutions();
|
|
spy.mockRestore();
|
|
}
|
|
});
|
|
|
|
it.each([
|
|
{ name: "assignments", sameIssue: true, first: "assignment", second: "assignment" },
|
|
{ name: "unrelated tasks", sameIssue: false, first: "assignment", second: "assignment" },
|
|
{ name: "assignment then direct comment", sameIssue: true, first: "assignment", second: "direct" },
|
|
{ name: "direct comment then assignment", sameIssue: true, first: "direct", second: "assignment" },
|
|
{ name: "assignment then queued comment", sameIssue: true, first: "assignment", second: "queued" },
|
|
{ name: "queued comment then assignment", sameIssue: true, first: "queued", second: "assignment" },
|
|
{ name: "queued comments", sameIssue: true, first: "queued", second: "queued" },
|
|
])("serializes issue claims without serializing unrelated work: $name", async ({ sameIssue, first, second }) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({ maxConcurrentRuns: 2 });
|
|
const firstIssueId = randomUUID();
|
|
const secondIssueId = sameIssue ? firstIssueId : randomUUID();
|
|
for (const id of new Set([firstIssueId, secondIssueId])) {
|
|
await db.insert(issues).values({ id, companyId, title: "Concurrent assignment", status: "todo", assigneeAgentId: agentId });
|
|
}
|
|
async function queue(issueId: string, kind: string) {
|
|
const commentId = kind === "assignment" ? null : randomUUID();
|
|
if (commentId) await db.insert(issueComments).values({ id: commentId, companyId, issueId, authorUserId: "local-board", body: "Continue this task." });
|
|
const queued = await seedQueuedRun({ companyId, agentId, issueId,
|
|
wakeReason: commentId ? "issue_commented" : "issue_assigned",
|
|
invocationSource: commentId ? "automation" : "assignment",
|
|
contextExtras: commentId ? { wakeCommentIds: [commentId], wakeCommentId: commentId } : {},
|
|
});
|
|
if (kind === "queued") await db.update(agentWakeupRequests).set({
|
|
payload: withQueuedCommentIdsInWakePayload({ issueId }, [commentId!]),
|
|
}).where(eq(agentWakeupRequests.id, queued.wakeupRequestId));
|
|
}
|
|
await queue(firstIssueId, first);
|
|
await queue(secondIssueId, second);
|
|
let release!: () => void;
|
|
const gate = new Promise<void>((resolve) => { release = resolve; });
|
|
const entered = Promise.withResolvers<void>();
|
|
const expectedExecutions = sameIssue ? 1 : 2;
|
|
let executions = 0;
|
|
mockAdapterExecute.mockImplementation(async () => {
|
|
if (++executions === expectedExecutions) entered.resolve();
|
|
await gate;
|
|
return { exitCode: 0, signal: null, timedOut: false, errorMessage: null, summary: "Assignment finished.", provider: "test", model: "test-model" };
|
|
});
|
|
try {
|
|
await heartbeat.resumeQueuedRuns();
|
|
await entered.promise;
|
|
expect(mockAdapterExecute).toHaveBeenCalledTimes(expectedExecutions);
|
|
const runs = await db.select().from(heartbeatRuns);
|
|
const running = runs.filter((run) => run.status === "running");
|
|
expect(running).toHaveLength(sameIssue ? 1 : 2);
|
|
expect(runs.filter((run) => run.status === "queued")).toHaveLength(sameIssue ? 1 : 0);
|
|
for (const run of running) {
|
|
const [issue] = await db.select().from(issues).where(eq(issues.id, (run.contextSnapshot as { issueId: string }).issueId));
|
|
expect(issue.executionRunId).toBe(run.id);
|
|
}
|
|
} finally {
|
|
release();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
}
|
|
});
|
|
|
|
it.each(["active_lease", "pending_cleanup", "failed_cleanup", "finalizer_lease", "workspace_finalization",
|
|
"retained_ready", "retained_missing_receipt", "retained_failed", "retained_wrong_policy"])(
|
|
"waits for durable cleanup from another controller: %s", async (pending) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID(), previousId = randomUUID();
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Remote cleanup", status: "todo", assigneeAgentId: agentId });
|
|
await db.insert(heartbeatRuns).values({ id: previousId, companyId, agentId,
|
|
status: "succeeded", runtimeMode: "native", nativeIssueId: issueId,
|
|
invocationSource: "assignment", contextSnapshot: { issueId }, finishedAt: new Date() });
|
|
await db.update(issues).set({ executionRunId: previousId }).where(eq(issues.id, issueId));
|
|
const leaseId = randomUUID();
|
|
const retained = pending.startsWith("retained_");
|
|
const hasLease = retained || ["active_lease", "pending_cleanup", "failed_cleanup"].includes(pending);
|
|
if (hasLease) await db.insert(environmentLeases).values({ id: leaseId, companyId, issueId,
|
|
heartbeatRunId: previousId, provider: "daytona",
|
|
status: retained ? "retained" : pending === "pending_cleanup" ? "pending_cleanup" : "released",
|
|
leasePolicy: pending === "retained_wrong_policy" ? "retain_on_failure" : retained ? "reuse_by_environment" : "ephemeral",
|
|
releasedAt: retained || pending === "active_lease" ? null : new Date(),
|
|
cleanupStatus: ["retained_ready", "retained_wrong_policy"].includes(pending) ? "success"
|
|
: ["failed_cleanup", "retained_failed"].includes(pending) ? "failed" : null });
|
|
await db.insert(nativeRunFinalizations).values({ runId: previousId, companyId, issueId,
|
|
phase: pending === "workspace_finalization" ? "workspace_finalizing" : "committed",
|
|
leaseOwner: pending === "finalizer_lease" ? "another-controller" : null });
|
|
const next = await seedQueuedRun({ companyId, agentId, issueId, wakeReason: "issue_assigned", invocationSource: "assignment" });
|
|
mockAdapterExecute.mockImplementation(async () => {
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, issueId));
|
|
return { exitCode: 0, signal: null, timedOut: false, errorMessage: null,
|
|
summary: "Task completed", provider: "test", model: "test-model" };
|
|
});
|
|
// No in-memory executor exists in this service. Durable ownership alone
|
|
// must fence a second controller until cleanup settles. A verified warm
|
|
// retention receipt is already a settled boundary and allows continuation.
|
|
await heartbeat.resumeQueuedRuns();
|
|
if (pending !== "retained_ready") {
|
|
expect((await heartbeat.getRun(next.runId))?.status).toBe("queued");
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].executionRunId).toBe(previousId);
|
|
if (hasLease) await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success" }).where(eq(environmentLeases.id, leaseId));
|
|
await db.update(nativeRunFinalizations).set({ phase: "committed", leaseOwner: null }).where(eq(nativeRunFinalizations.runId, previousId));
|
|
await heartbeat.resumeQueuedRuns();
|
|
}
|
|
await heartbeat.drainActiveRunExecutions();
|
|
expect((await heartbeat.getRun(next.runId))?.status).toBe("succeeded");
|
|
expect(mockAdapterExecute).toHaveBeenCalledTimes(1);
|
|
},
|
|
);
|
|
|
|
it("skips generic timer wakes with no actionable assigned work before adapter execution", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
enabled: true,
|
|
skipTimerWhenNoActionableWork: true,
|
|
},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "timer",
|
|
triggerDetail: "system",
|
|
});
|
|
|
|
expect(run).toBeNull();
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
const runRows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns);
|
|
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "heartbeat.timer.no_actionable_work",
|
|
});
|
|
expect(wakeup?.payload).toMatchObject({
|
|
heartbeatSkip: {
|
|
reason: expect.stringContaining("No assigned todo or in_progress issue"),
|
|
},
|
|
});
|
|
expect(runRows).toHaveLength(0);
|
|
});
|
|
|
|
it("checks guarded issue status and assignee under the enqueue lock", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const parkedIssueId = randomUUID();
|
|
const reassignedIssueId = randomUUID();
|
|
await db.insert(issues).values([
|
|
{
|
|
id: parkedIssueId,
|
|
companyId,
|
|
title: "Parked connection intent",
|
|
status: "backlog" as const,
|
|
priority: "medium" as const,
|
|
assigneeAgentId: agentId,
|
|
},
|
|
{
|
|
id: reassignedIssueId,
|
|
companyId,
|
|
title: "Reassigned connection intent",
|
|
status: "in_progress" as const,
|
|
priority: "medium" as const,
|
|
assigneeAgentId: null,
|
|
},
|
|
]);
|
|
|
|
for (const issueId of [parkedIssueId, reassignedIssueId]) {
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
reason: "issue_commented",
|
|
payload: { issueId, interactionId: randomUUID() },
|
|
contextSnapshot: { issueId, wakeReason: "issue_commented" },
|
|
requestedByActorType: "user",
|
|
requestedByActorId: "responsible-user",
|
|
issueStateGuard: {
|
|
statuses: ["in_progress"],
|
|
assigneeAgentId: agentId,
|
|
},
|
|
});
|
|
expect(run).toBeNull();
|
|
}
|
|
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
expect(await db.select().from(heartbeatRuns)).toHaveLength(0);
|
|
expect(await db.select({ status: agentWakeupRequests.status, reason: agentWakeupRequests.reason })
|
|
.from(agentWakeupRequests)).toEqual([
|
|
{ status: "skipped", reason: "issue_state_guard_mismatch" },
|
|
{ status: "skipped", reason: "issue_state_guard_mismatch" },
|
|
]);
|
|
});
|
|
|
|
it.each([
|
|
{ runtimeMode: "native", status: "done", reassigned: false },
|
|
{ runtimeMode: "legacy", status: "done", reassigned: false },
|
|
{ runtimeMode: "native", status: "cancelled", reassigned: false },
|
|
{ runtimeMode: "native", status: "backlog", reassigned: false },
|
|
{ runtimeMode: "native", status: "in_review", reassigned: false },
|
|
{ runtimeMode: "native", status: "blocked", reassigned: false },
|
|
{ runtimeMode: "native", status: "in_progress", reassigned: true },
|
|
] as const)("skips stale $runtimeMode productive recovery after status=$status reassigned=$reassigned commits under the enqueue lock", async ({ runtimeMode, status, reassigned }) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
const runId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Completion racing with productive recovery",
|
|
status: "in_progress",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "assignment",
|
|
runtimeMode,
|
|
status: "succeeded",
|
|
livenessState: "completed",
|
|
contextSnapshot: { issueId, wakeReason: "issue_assigned" },
|
|
startedAt: new Date(),
|
|
finishedAt: new Date(),
|
|
});
|
|
|
|
// Let the real sweep select its in-progress snapshot, then hold the issue
|
|
// lock until the real enqueue transaction is waiting on the newer state.
|
|
const enqueueWakeup = vi.fn(async (...[targetAgentId, options]: Parameters<typeof heartbeat.wakeup>) => {
|
|
let pendingWake!: ReturnType<typeof heartbeat.wakeup>;
|
|
await db.transaction(async (tx) => {
|
|
await tx.update(issues).set({
|
|
status,
|
|
...(reassigned ? { assigneeAgentId: null, assigneeUserId: "responsible-user" } : {}),
|
|
}).where(eq(issues.id, issueId));
|
|
const [{ pid }] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
|
|
pendingWake = heartbeat.wakeup(targetAgentId, options);
|
|
try {
|
|
expect(await waitForCondition(async () => {
|
|
const [{ waiting }] = await db.execute<{ waiting: boolean }>(sql`
|
|
select exists (
|
|
select 1 from pg_stat_activity
|
|
where ${pid} = any(pg_blocking_pids(pid))
|
|
) as waiting
|
|
`);
|
|
return waiting;
|
|
})).toBe(true);
|
|
} catch (error) {
|
|
// Observe a pending rejection even if the lock assertion fails.
|
|
void pendingWake.catch(() => {});
|
|
throw error;
|
|
}
|
|
});
|
|
return pendingWake;
|
|
});
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(enqueueWakeup).toHaveBeenCalledOnce();
|
|
expect(result).toMatchObject({ continuationRequeued: 0, escalated: 0, skipped: 1, issueIds: [] });
|
|
expect(await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)).toEqual([{ id: runId }]);
|
|
expect(await db.select().from(issueComments)).toHaveLength(0);
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
const [wakeup] = await db.select().from(agentWakeupRequests);
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "issue_state_guard_mismatch",
|
|
runId: null,
|
|
payload: {
|
|
heartbeatSkip: {
|
|
expectedStatuses: ["in_progress"],
|
|
actualStatus: status,
|
|
expectedAssigneeAgentId: agentId,
|
|
actualAssigneeAgentId: reassigned ? null : agentId,
|
|
},
|
|
},
|
|
});
|
|
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
|
expect(issue).toMatchObject({ status, assigneeAgentId: reassigned ? null : agentId });
|
|
});
|
|
|
|
it("cancels a resolved connection-intent wake parked before queued-run claim", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Connection intent parked after enqueue",
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
const { runId, wakeupRequestId } = await seedQueuedRun({
|
|
companyId,
|
|
agentId,
|
|
issueId,
|
|
wakeReason: "issue_commented",
|
|
invocationSource: "automation",
|
|
contextExtras: {
|
|
interactionId: randomUUID(),
|
|
interactionKind: "connection_intent",
|
|
interactionStatus: "accepted",
|
|
interactionResolvedAt: "2026-08-28T13:30:00.000Z",
|
|
mutation: "interaction",
|
|
source: "connection_intent.resolved",
|
|
},
|
|
});
|
|
|
|
await db.update(issues).set({ status: "backlog" }).where(eq(issues.id, issueId));
|
|
await heartbeat.resumeQueuedRuns();
|
|
|
|
const [run, wakeup, issue] = await Promise.all([
|
|
db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: agentWakeupRequests.status, error: agentWakeupRequests.error })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: issues.status }).from(issues).where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null),
|
|
]);
|
|
expect(run).toMatchObject({ status: "cancelled", errorCode: "issue_not_in_progress" });
|
|
expect(wakeup).toMatchObject({ status: "skipped", error: expect.stringContaining("no longer in_progress") });
|
|
expect(issue?.status).toBe("backlog");
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
});
|
|
|
|
it("does not re-open a resolved connection-intent issue parked after claim", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Connection intent parked between claim and checkout",
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
const { runId, wakeupRequestId } = await seedQueuedRun({
|
|
companyId,
|
|
agentId,
|
|
issueId,
|
|
wakeReason: "issue_commented",
|
|
invocationSource: "automation",
|
|
contextExtras: {
|
|
interactionId: randomUUID(),
|
|
interactionKind: "connection_intent",
|
|
interactionStatus: "accepted",
|
|
interactionResolvedAt: "2026-08-28T13:30:00.000Z",
|
|
mutation: "interaction",
|
|
source: "connection_intent.resolved",
|
|
},
|
|
});
|
|
|
|
await db.execute(sql.raw(`
|
|
CREATE OR REPLACE FUNCTION park_connection_intent_after_claim()
|
|
RETURNS trigger AS $trigger$
|
|
BEGIN
|
|
IF NEW.id = '${runId}'::uuid AND NEW.status = 'running' THEN
|
|
UPDATE issues SET status = 'backlog' WHERE id = '${issueId}'::uuid;
|
|
END IF;
|
|
RETURN NEW;
|
|
END;
|
|
$trigger$ LANGUAGE plpgsql;
|
|
|
|
CREATE TRIGGER park_connection_intent_after_claim
|
|
AFTER UPDATE OF status ON heartbeat_runs
|
|
FOR EACH ROW EXECUTE FUNCTION park_connection_intent_after_claim();
|
|
`));
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
const [run, wakeup, issue] = await Promise.all([
|
|
db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: agentWakeupRequests.status, error: agentWakeupRequests.error })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: issues.status }).from(issues).where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null),
|
|
]);
|
|
expect(run).toMatchObject({ status: "cancelled", errorCode: "issue_not_in_progress" });
|
|
expect(wakeup).toMatchObject({ status: "skipped", error: expect.stringContaining("no longer in_progress") });
|
|
expect(issue?.status).toBe("backlog");
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
});
|
|
|
|
it.each([
|
|
{
|
|
mutation: "parked",
|
|
expectedErrorCode: "issue_not_in_progress",
|
|
expectedError: "no longer in_progress",
|
|
},
|
|
{
|
|
mutation: "reassigned",
|
|
expectedErrorCode: "issue_assignee_changed",
|
|
expectedError: "changed assignee",
|
|
},
|
|
])(
|
|
"cancels a resolved connection-intent wake $mutation after checkout but before adapter dispatch",
|
|
async ({ mutation, expectedErrorCode, expectedError }) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const replacementAgentId = randomUUID();
|
|
if (mutation === "reassigned") {
|
|
await db.insert(agents).values({
|
|
id: replacementAgentId,
|
|
companyId,
|
|
name: "ReplacementCoder",
|
|
role: "engineer",
|
|
status: "active",
|
|
adapterType: "codex_local",
|
|
adapterConfig: {},
|
|
runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } },
|
|
permissions: {},
|
|
});
|
|
}
|
|
const issueId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: `Connection intent ${mutation} at final dispatch`,
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
const { runId, wakeupRequestId } = await seedQueuedRun({
|
|
companyId,
|
|
agentId,
|
|
issueId,
|
|
wakeReason: "issue_commented",
|
|
invocationSource: "automation",
|
|
contextExtras: {
|
|
interactionId: randomUUID(),
|
|
interactionKind: "connection_intent",
|
|
interactionStatus: "accepted",
|
|
interactionResolvedAt: "2026-08-28T13:30:00.000Z",
|
|
mutation: "interaction",
|
|
source: "connection_intent.resolved",
|
|
},
|
|
});
|
|
beforeContinuationDispatchCheck = async ({ runId: guardedRunId, issueId: guardedIssueId }) => {
|
|
expect(guardedRunId).toBe(runId);
|
|
expect(guardedIssueId).toBe(issueId);
|
|
await db
|
|
.update(issues)
|
|
.set(mutation === "parked"
|
|
? { status: "backlog", updatedAt: new Date() }
|
|
: { assigneeAgentId: replacementAgentId, updatedAt: new Date() })
|
|
.where(eq(issues.id, issueId));
|
|
};
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
const [run, wakeup, issue] = await Promise.all([
|
|
db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: agentWakeupRequests.status, error: agentWakeupRequests.error })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: issues.status, assigneeAgentId: issues.assigneeAgentId })
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null),
|
|
]);
|
|
expect(run).toMatchObject({ status: "cancelled", errorCode: expectedErrorCode });
|
|
expect(wakeup).toMatchObject({ status: "skipped", error: expect.stringContaining(expectedError) });
|
|
expect(issue).toMatchObject(mutation === "parked"
|
|
? { status: "backlog", assigneeAgentId: agentId }
|
|
: { status: "in_progress", assigneeAgentId: replacementAgentId });
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
},
|
|
);
|
|
|
|
it("rejects ownership changes immediately before the final continuation handoff", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Connection intent parked at the atomic dispatch gate",
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
const { runId } = await seedQueuedRun({
|
|
companyId,
|
|
agentId,
|
|
issueId,
|
|
wakeReason: "issue_commented",
|
|
invocationSource: "automation",
|
|
contextExtras: {
|
|
interactionId: randomUUID(),
|
|
interactionKind: "connection_intent",
|
|
interactionStatus: "accepted",
|
|
interactionResolvedAt: "2026-08-28T13:30:00.000Z",
|
|
mutation: "interaction",
|
|
source: "connection_intent.resolved",
|
|
},
|
|
});
|
|
|
|
const ordering: string[] = [];
|
|
let parkPromise: Promise<unknown> | null = null;
|
|
afterContinuationDispatchCheck = async ({ runId: guardedRunId, issueId: guardedIssueId }) => {
|
|
expect(guardedRunId).toBe(runId);
|
|
expect(guardedIssueId).toBe(issueId);
|
|
ordering.push("validated");
|
|
parkPromise = Promise.resolve(
|
|
db
|
|
.update(issues)
|
|
.set({
|
|
status: "backlog",
|
|
checkoutRunId: null,
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(issues.id, issueId))
|
|
.returning({ id: issues.id }),
|
|
).then((rows) => {
|
|
expect(rows).toHaveLength(1);
|
|
ordering.push("parked");
|
|
});
|
|
// Admission is committed before adapter-owned setup. This concurrent
|
|
// update must not wait on a lock held by the adapter callback.
|
|
await new Promise((resolve) => setTimeout(resolve, 25));
|
|
await parkPromise;
|
|
expect(ordering).toEqual(["validated", "parked"]);
|
|
};
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await parkPromise;
|
|
|
|
expect((await heartbeat.getRun(runId))?.status).toBe("cancelled");
|
|
const issue = await db
|
|
.select({ status: issues.status })
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null);
|
|
expect(issue?.status).toBe("backlog");
|
|
expect(ordering).toEqual(["validated", "parked"]);
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
});
|
|
|
|
it("leaves ineligible timers disarmed without writing skipped wakes or moving their baseline", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
enabled: true,
|
|
intervalSec: 60,
|
|
skipTimerWhenNoActionableWork: true,
|
|
},
|
|
});
|
|
const now = new Date();
|
|
await db
|
|
.update(agents)
|
|
.set({ lastHeartbeatAt: new Date(now.getTime() - 120_000) })
|
|
.where(eq(agents.id, agentId));
|
|
|
|
const firstTick = await heartbeat.tickTimers(now);
|
|
const secondTick = await heartbeat.tickTimers(now);
|
|
|
|
expect(firstTick.skipped).toBe(0);
|
|
expect(await heartbeat.nextAgentTimerAt()).toBeNull();
|
|
expect(secondTick.skipped).toBe(0);
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const wakeups = await db
|
|
.select({ reason: agentWakeupRequests.reason })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
const [agent] = await db
|
|
.select({ lastHeartbeatAt: agents.lastHeartbeatAt })
|
|
.from(agents)
|
|
.where(eq(agents.id, agentId));
|
|
|
|
expect(wakeups).toEqual([]);
|
|
expect(agent?.lastHeartbeatAt).toBeInstanceOf(Date);
|
|
expect(agent?.lastHeartbeatAt?.getTime()).toBe(now.getTime() - 120_000);
|
|
});
|
|
|
|
it("atomically claims a due timer interval across overlapping scheduler ticks", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
enabled: true,
|
|
intervalSec: 60,
|
|
},
|
|
});
|
|
const now = new Date();
|
|
await db
|
|
.update(agents)
|
|
.set({
|
|
createdAt: new Date(now.getTime() - 120_000),
|
|
lastHeartbeatAt: null,
|
|
})
|
|
.where(eq(agents.id, agentId));
|
|
|
|
const results = await Promise.all([
|
|
heartbeat.tickTimers(now),
|
|
heartbeat.tickTimers(now),
|
|
]);
|
|
|
|
expect(results.reduce((total, result) => total + result.enqueued, 0)).toBe(1);
|
|
|
|
const runs = await db
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
contextSnapshot: heartbeatRuns.contextSnapshot,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.agentId, agentId));
|
|
const [agent] = await db
|
|
.select({ lastHeartbeatAt: agents.lastHeartbeatAt })
|
|
.from(agents)
|
|
.where(eq(agents.id, agentId));
|
|
|
|
expect(runs).toHaveLength(1);
|
|
expect(runs[0]?.contextSnapshot).toMatchObject({
|
|
timerClaimWasFirstHeartbeat: true,
|
|
});
|
|
expect(agent?.lastHeartbeatAt?.getTime()).toBeGreaterThanOrEqual(now.getTime());
|
|
});
|
|
|
|
it("allows generic timer wakes when the agent has assigned todo work", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
enabled: true,
|
|
skipTimerWhenNoActionableWork: true,
|
|
},
|
|
});
|
|
await db.insert(issues).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
title: "Assigned work",
|
|
status: "todo",
|
|
priority: "high",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "timer",
|
|
triggerDetail: "system",
|
|
});
|
|
|
|
expect(run).not.toBeNull();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
expect(countExecuteCallsForRun(run!.id)).toBe(1);
|
|
});
|
|
|
|
it("allows legacy generic timer wakes by default when no skip policy is set", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
enabled: true,
|
|
},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "timer",
|
|
triggerDetail: "system",
|
|
});
|
|
|
|
expect(run).not.toBeNull();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
expect(countExecuteCallsForRun(run!.id)).toBe(1);
|
|
});
|
|
|
|
it("allows explicit proactive generic timer wakes without assigned issue work", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
enabled: true,
|
|
skipTimerWhenNoActionableWork: false,
|
|
},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "timer",
|
|
triggerDetail: "system",
|
|
});
|
|
|
|
expect(run).not.toBeNull();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
expect(countExecuteCallsForRun(run!.id)).toBe(1);
|
|
});
|
|
|
|
it("skips wakes before queueing when per-agent daily run cap is reached", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyRuns: 1,
|
|
},
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "succeeded",
|
|
createdAt: new Date(),
|
|
startedAt: new Date(),
|
|
finishedAt: new Date(),
|
|
contextSnapshot: {},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
});
|
|
|
|
expect(run).toBeNull();
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "heartbeat.daily_run_limit",
|
|
});
|
|
expect(wakeup?.payload).toMatchObject({
|
|
heartbeatSkip: {
|
|
observed: 1,
|
|
limit: 1,
|
|
},
|
|
});
|
|
});
|
|
|
|
it("treats zero daily run cap as a hard stop", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyRuns: 0,
|
|
},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
});
|
|
|
|
expect(run).toBeNull();
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "heartbeat.daily_run_limit",
|
|
});
|
|
expect(wakeup?.payload).toMatchObject({
|
|
heartbeatSkip: {
|
|
observed: 0,
|
|
limit: 0,
|
|
},
|
|
});
|
|
});
|
|
|
|
it("counts started cancelled runs toward the per-agent daily run cap", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyRuns: 1,
|
|
},
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "cancelled",
|
|
createdAt: new Date(),
|
|
startedAt: new Date(),
|
|
finishedAt: new Date(),
|
|
contextSnapshot: {},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
});
|
|
|
|
expect(run).toBeNull();
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "heartbeat.daily_run_limit",
|
|
});
|
|
expect(wakeup?.payload).toMatchObject({
|
|
heartbeatSkip: {
|
|
observed: 1,
|
|
limit: 1,
|
|
},
|
|
});
|
|
});
|
|
|
|
it("coalesces same-issue wakes before enforcing the daily run cap", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyRuns: 1,
|
|
},
|
|
});
|
|
const issueId = randomUUID();
|
|
const wakeupRequestId = randomUUID();
|
|
const queuedRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "succeeded",
|
|
createdAt: new Date(),
|
|
startedAt: new Date(),
|
|
finishedAt: new Date(),
|
|
contextSnapshot: {},
|
|
});
|
|
await db.insert(agentWakeupRequests).values({
|
|
id: wakeupRequestId,
|
|
companyId,
|
|
agentId,
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
reason: "manual",
|
|
payload: { issueId },
|
|
status: "queued",
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: queuedRunId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "queued",
|
|
wakeupRequestId,
|
|
contextSnapshot: { issueId },
|
|
});
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Queued issue work",
|
|
status: "in_progress",
|
|
priority: "high",
|
|
assigneeAgentId: agentId,
|
|
executionRunId: queuedRunId,
|
|
});
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId: queuedRunId })
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
payload: { issueId },
|
|
});
|
|
|
|
expect(run?.id).toBe(queuedRunId);
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const wakeups = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
runId: agentWakeupRequests.runId,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
|
|
expect(wakeups).toEqual(
|
|
expect.arrayContaining([
|
|
expect.objectContaining({
|
|
status: "coalesced",
|
|
reason: "issue_execution_same_name",
|
|
runId: queuedRunId,
|
|
}),
|
|
]),
|
|
);
|
|
});
|
|
|
|
it.each(["codex_local", "gemini_local"].flatMap(adapterType => [
|
|
{ adapterType, inputTokens: 50_000, cachedInputTokens: 550_000, rotate: true, rawInputIncludesCached: false },
|
|
{ adapterType, inputTokens: 50_000, cachedInputTokens: 449_999, rotate: false, rawInputIncludesCached: false },
|
|
{ adapterType, inputTokens: 0, cachedInputTokens: 500_000, rotate: true, rawInputIncludesCached: false },
|
|
{ adapterType, inputTokens: 400_000, cachedInputTokens: 350_000, rotate: false, rawInputIncludesCached: undefined },
|
|
{ adapterType, inputTokens: 500_000, cachedInputTokens: 350_000, rotate: true, rawInputIncludesCached: undefined },
|
|
]))("counts cached prompt input for $adapterType session rotation ($inputTokens + $cachedInputTokens; raw includes cache: $rawInputIncludesCached)", async ({ adapterType, inputTokens, cachedInputTokens, rotate, rawInputIncludesCached }) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({ heartbeatConfig: {
|
|
sessionCompaction: { enabled: true, maxSessionRuns: 0, maxRawInputTokens: 500_000, maxSessionAgeHours: 0 },
|
|
} });
|
|
const sessionId = randomUUID();
|
|
await db.update(agents).set({ adapterType }).where(eq(agents.id, agentId));
|
|
await db.insert(agentRuntimeState).values({ companyId, agentId, adapterType, sessionId });
|
|
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", sessionIdAfter: sessionId,
|
|
usageJson: { ...(rawInputIncludesCached === false ? { rawInputIncludesCached } : {}), inputTokens, cachedInputTokens, outputTokens: 10, rawInputTokens: inputTokens, rawCachedInputTokens: cachedInputTokens },
|
|
resultJson: { summary: "Continue from this completed turn" }, startedAt: new Date(), finishedAt: new Date(),
|
|
});
|
|
mockAdapterExecute.mockResolvedValueOnce({ exitCode: 0, signal: null, timedOut: false, errorMessage: null,
|
|
provider: "test", model: "test-model", costUsd: 0, usageComplete: true, usageBasis: "per_run",
|
|
usage: { inputTokens: 1, cachedInputTokens: 2, outputTokens: 3 },
|
|
});
|
|
const run = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual" });
|
|
expect(run).not.toBeNull();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
expect(countExecuteCallsForRun(run!.id)).toBe(1);
|
|
const [context] = mockAdapterExecute.mock.calls.find(([context]) => context?.runId === run!.id)!;
|
|
expect(context.runtime.sessionId).toBe(rotate ? null : sessionId);
|
|
if (rotate) {
|
|
expect(context.context.paperclipSessionRotationReason).toContain((inputTokens + (rawInputIncludesCached === false ? cachedInputTokens : 0)).toLocaleString("en-US"));
|
|
expect(context.context.paperclipSessionHandoffMarkdown).toContain("Continue from this completed turn");
|
|
} else expect(context.context.paperclipSessionRotationReason).toBeUndefined();
|
|
const [finished] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id));
|
|
expect(finished.usageJson).toMatchObject({ rawInputTokens: 1, rawCachedInputTokens: 2, rawInputIncludesCached: false });
|
|
});
|
|
|
|
it("skips wakes before queueing when per-agent daily cost cap is reached", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyCostCents: 75,
|
|
},
|
|
});
|
|
await db.insert(costEvents).values({
|
|
companyId,
|
|
agentId,
|
|
provider: "test",
|
|
biller: "test",
|
|
billingType: "metered_api",
|
|
model: "test-model",
|
|
inputTokens: 100,
|
|
outputTokens: 50,
|
|
costCents: 75,
|
|
occurredAt: new Date(),
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
});
|
|
|
|
expect(run).toBeNull();
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "heartbeat.daily_cost_limit",
|
|
});
|
|
expect(wakeup?.payload).toMatchObject({
|
|
heartbeatSkip: {
|
|
observed: 75,
|
|
limit: 75,
|
|
},
|
|
});
|
|
});
|
|
|
|
it("treats zero daily cost cap as a hard stop", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyCostCents: 0,
|
|
},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
});
|
|
|
|
expect(run).toBeNull();
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "heartbeat.daily_cost_limit",
|
|
});
|
|
expect(wakeup?.payload).toMatchObject({
|
|
heartbeatSkip: {
|
|
observed: 0,
|
|
limit: 0,
|
|
},
|
|
});
|
|
});
|
|
|
|
it("skips already queued runs before adapter execution when the daily cost cap is reached", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyCostCents: 75,
|
|
},
|
|
});
|
|
const wakeupRequestId = randomUUID();
|
|
const runId = randomUUID();
|
|
await db.insert(agentWakeupRequests).values({
|
|
id: wakeupRequestId,
|
|
companyId,
|
|
agentId,
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
reason: "manual",
|
|
payload: {},
|
|
status: "queued",
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "queued",
|
|
wakeupRequestId,
|
|
contextSnapshot: {},
|
|
});
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId })
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
await db.insert(costEvents).values({
|
|
companyId,
|
|
agentId,
|
|
provider: "test",
|
|
biller: "test",
|
|
billingType: "metered_api",
|
|
model: "test-model",
|
|
inputTokens: 100,
|
|
outputTokens: 50,
|
|
costCents: 75,
|
|
occurredAt: new Date(),
|
|
});
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [run] = await db
|
|
.select({
|
|
status: heartbeatRuns.status,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
resultJson: heartbeatRuns.resultJson,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
error: agentWakeupRequests.error,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
|
|
expect(run).toMatchObject({
|
|
status: "cancelled",
|
|
errorCode: "heartbeat.daily_cost_limit",
|
|
});
|
|
expect(run?.resultJson).toMatchObject({
|
|
stopReason: "heartbeat.daily_cost_limit",
|
|
observed: 75,
|
|
limit: 75,
|
|
});
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
error: expect.stringContaining("per-day heartbeat budget cap"),
|
|
});
|
|
});
|
|
|
|
it("skips already queued issue runs at the daily run cap and releases the execution lock", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyRuns: 1,
|
|
},
|
|
});
|
|
const issueId = randomUUID();
|
|
const wakeupRequestId = randomUUID();
|
|
const queuedRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "succeeded",
|
|
createdAt: new Date(),
|
|
startedAt: new Date(),
|
|
finishedAt: new Date(),
|
|
contextSnapshot: {},
|
|
});
|
|
await db.insert(agentWakeupRequests).values({
|
|
id: wakeupRequestId,
|
|
companyId,
|
|
agentId,
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
reason: "manual",
|
|
payload: { issueId },
|
|
status: "queued",
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: queuedRunId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "queued",
|
|
wakeupRequestId,
|
|
contextSnapshot: { issueId },
|
|
});
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Queued issue work",
|
|
status: "in_progress",
|
|
priority: "high",
|
|
assigneeAgentId: agentId,
|
|
executionRunId: queuedRunId,
|
|
});
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId: queuedRunId })
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [run] = await db
|
|
.select({
|
|
status: heartbeatRuns.status,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, queuedRunId));
|
|
const [wakeup] = await db
|
|
.select({ status: agentWakeupRequests.status })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
const [issue] = await db
|
|
.select({ executionRunId: issues.executionRunId })
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId));
|
|
|
|
expect(run).toMatchObject({
|
|
status: "cancelled",
|
|
errorCode: "heartbeat.daily_run_limit",
|
|
});
|
|
expect(wakeup).toMatchObject({ status: "skipped" });
|
|
expect(issue?.executionRunId).toBeNull();
|
|
});
|
|
|
|
it("promotes deferred issue wakes when a queued holder is cancelled by the daily run cap", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyRuns: 1,
|
|
},
|
|
});
|
|
const peerAgentId = randomUUID();
|
|
const issueId = randomUUID();
|
|
const wakeupRequestId = randomUUID();
|
|
const queuedRunId = randomUUID();
|
|
const deferredWakeupId = randomUUID();
|
|
await db.insert(agents).values({
|
|
id: peerAgentId,
|
|
companyId,
|
|
name: "PeerAgent",
|
|
role: "engineer",
|
|
status: "active",
|
|
adapterType: "codex_local",
|
|
adapterConfig: {},
|
|
runtimeConfig: {
|
|
heartbeat: {
|
|
wakeOnDemand: true,
|
|
maxConcurrentRuns: 1,
|
|
},
|
|
},
|
|
permissions: {},
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "succeeded",
|
|
createdAt: new Date(),
|
|
startedAt: new Date(),
|
|
finishedAt: new Date(),
|
|
contextSnapshot: {},
|
|
});
|
|
await db.insert(agentWakeupRequests).values({
|
|
id: wakeupRequestId,
|
|
companyId,
|
|
agentId,
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
reason: "manual",
|
|
payload: { issueId },
|
|
status: "queued",
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: queuedRunId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "queued",
|
|
wakeupRequestId,
|
|
contextSnapshot: { issueId },
|
|
});
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Queued issue work",
|
|
status: "in_progress",
|
|
priority: "high",
|
|
assigneeAgentId: agentId,
|
|
executionRunId: queuedRunId,
|
|
});
|
|
await db.insert(agentWakeupRequests).values({
|
|
id: deferredWakeupId,
|
|
companyId,
|
|
agentId: peerAgentId,
|
|
source: "comment",
|
|
triggerDetail: "mention",
|
|
reason: "issue_execution_deferred",
|
|
payload: {
|
|
issueId,
|
|
_paperclipWakeContext: {
|
|
issueId,
|
|
wakeReason: "issue_mention",
|
|
},
|
|
},
|
|
status: "deferred_issue_execution",
|
|
});
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId: queuedRunId })
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
const [deferred] = await db
|
|
.select({ status: agentWakeupRequests.status, runId: agentWakeupRequests.runId })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, deferredWakeupId));
|
|
const [promotedRun] = deferred?.runId
|
|
? await db
|
|
.select({ agentId: heartbeatRuns.agentId })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, deferred.runId))
|
|
: [];
|
|
|
|
expect(deferred?.status).not.toBe("deferred_issue_execution");
|
|
expect(promotedRun?.agentId).toBe(peerAgentId);
|
|
});
|
|
|
|
it("delivers a person's message saved behind a system continuation cancelled at claim because its task closed", async () => {
|
|
// The claim-time closed-task cancel must leave the same never-started proof as every other
|
|
// queued-run cancel; without it task release treats the run as unfinished provider work and
|
|
// never promotes the person's message waiting behind it.
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Closed while its continuation waited",
|
|
status: "in_progress", assigneeAgentId: agentId }).returning();
|
|
const { runId, wakeupRequestId } = await seedQueuedRun({ companyId, agentId, issueId: issue.id,
|
|
wakeReason: "issue_continuation_needed", invocationSource: "automation" });
|
|
await db.update(issues).set({ status: "done", executionRunId: runId, executionLockedAt: new Date() })
|
|
.where(eq(issues.id, issue.id));
|
|
const [comment] = await db.insert(issueComments).values({ companyId, issueId: issue.id, authorType: "user",
|
|
authorUserId: "responsible-user", body: "Please also update the changelog" }).returning();
|
|
const [saved] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "comment",
|
|
triggerDetail: "manual", reason: "issue_execution_deferred", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: "responsible-user",
|
|
payload: withQueuedCommentIdsInWakePayload({ issueId: issue.id, _paperclipWakeContext: {
|
|
issueId: issue.id, wakeReason: "issue_commented", wakeSource: "comment",
|
|
} }, [comment.id]),
|
|
}).returning();
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
const [cancelled] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(cancelled).toMatchObject({ status: "cancelled", errorCode: "issue_closed_before_start",
|
|
resultJson: expect.objectContaining({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }) });
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeupRequestId)))[0]?.status)
|
|
.toBe("skipped");
|
|
await vi.waitFor(async () => {
|
|
const [promoted] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, saved.id));
|
|
expect(promoted.status).not.toBe("deferred_issue_execution");
|
|
expect(promoted.runId).toBeTruthy();
|
|
expect(countExecuteCallsForRun(promoted.runId!)).toBe(1);
|
|
}, { timeout: 10_000 });
|
|
});
|
|
|
|
describe("a queued system recovery run re-checks that its task still needs it when its turn comes", () => {
|
|
async function queueRecovery(task: Partial<typeof issues.$inferInsert>) {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Waited behind people's work",
|
|
status: "in_progress", assigneeAgentId: agentId, ...task }).returning();
|
|
const queued = await seedQueuedRun({ companyId, agentId, issueId: issue.id,
|
|
wakeReason: "issue_recovery_action_restored", invocationSource: "automation" });
|
|
return { companyId, agentId, issueId: issue.id, ...queued };
|
|
}
|
|
async function expectDroppedQuietly(runId: string, wakeupRequestId: string) {
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(run).toMatchObject({ status: "cancelled", errorCode: "recovery_no_longer_needed",
|
|
resultJson: expect.objectContaining({ executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }) });
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeupRequestId)))[0]?.status)
|
|
.toBe("skipped");
|
|
}
|
|
|
|
it("drops it when the task was reassigned while it waited", async () => {
|
|
const { issueId, runId, wakeupRequestId } = await queueRecovery({});
|
|
await db.update(issues).set({ assigneeAgentId: null, assigneeUserId: "responsible-user" }).where(eq(issues.id, issueId));
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await expectDroppedQuietly(runId, wakeupRequestId);
|
|
});
|
|
|
|
it("drops it when a newer run already continued the task", async () => {
|
|
const { companyId, agentId, issueId, runId, wakeupRequestId } = await queueRecovery({});
|
|
await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "assignment", triggerDetail: "system",
|
|
status: "succeeded", createdAt: new Date(Date.now() + 1_000), startedAt: new Date(Date.now() + 1_000),
|
|
finishedAt: new Date(Date.now() + 2_000), contextSnapshot: { issueId, wakeReason: "issue_commented" } });
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await expectDroppedQuietly(runId, wakeupRequestId);
|
|
});
|
|
|
|
it("starts it when the task still needs it", async () => {
|
|
const { runId } = await queueRecovery({ status: "todo" });
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
expect(countExecuteCallsForRun(runId)).toBe(1);
|
|
}, 60_000);
|
|
});
|
|
|
|
it("cancels queued max-turn continuations when another continuation owns the issue lock", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
const lockOwnerRunId = randomUUID();
|
|
|
|
await db.insert(heartbeatRuns).values({
|
|
id: lockOwnerRunId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "automation",
|
|
triggerDetail: "system",
|
|
status: "scheduled_retry",
|
|
scheduledRetryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
|
scheduledRetryAttempt: 1,
|
|
scheduledRetryAt: new Date("2026-04-20T12:00:00.000Z"),
|
|
contextSnapshot: {
|
|
issueId,
|
|
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
|
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
|
},
|
|
});
|
|
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Duplicate max-turn continuation",
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: agentId,
|
|
executionRunId: lockOwnerRunId,
|
|
executionAgentNameKey: "claudecoder",
|
|
executionLockedAt: new Date("2026-04-20T11:59:00.000Z"),
|
|
});
|
|
|
|
const { runId, wakeupRequestId } = await seedQueuedRun({
|
|
companyId,
|
|
agentId,
|
|
issueId,
|
|
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
|
invocationSource: "automation",
|
|
scheduledRetryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
|
contextExtras: {
|
|
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
|
},
|
|
});
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
const [run, wakeup, issue] = await Promise.all([
|
|
db
|
|
.select({
|
|
status: heartbeatRuns.status,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
resultJson: heartbeatRuns.resultJson,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db
|
|
.select({ status: agentWakeupRequests.status, error: agentWakeupRequests.error })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db
|
|
.select({ executionRunId: issues.executionRunId })
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null),
|
|
]);
|
|
|
|
expect(run?.status).toBe("cancelled");
|
|
expect(run?.errorCode).toBe("issue_execution_lock_changed");
|
|
expect(run?.resultJson).toMatchObject({ stopReason: "issue_execution_lock_changed" });
|
|
expect(wakeup?.status).toBe("skipped");
|
|
expect(wakeup?.error).toContain("execution lock");
|
|
expect(issue?.executionRunId).toBe(lockOwnerRunId);
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
});
|
|
|
|
it.each([
|
|
["accepted", "connection_intent"],
|
|
["rejected", "connection_intent"],
|
|
["rejected", "request_confirmation"],
|
|
])("resumes a %s %s outcome and promotes the claimed review task", async (interactionStatus, interactionKind) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
const interactionId = randomUUID();
|
|
const wakeCommentId = randomUUID();
|
|
let claimedIssue: { status: string; executionRunId: string | null } | null = null;
|
|
let resumedRunId: string | undefined;
|
|
let signalAdapterStarted!: () => void;
|
|
const adapterStarted = new Promise<void>(resolve => { signalAdapterStarted = resolve; });
|
|
const execute = mockAdapterExecute.getMockImplementation()!;
|
|
mockAdapterExecute.mockImplementation(async (...args: Parameters<typeof execute>) => {
|
|
if ((args[0] as { runId: string }).runId === resumedRunId) {
|
|
signalAdapterStarted();
|
|
// A claimed continuation can finish after the old terminal poll.
|
|
if (interactionStatus === "rejected" && interactionKind === "connection_intent") {
|
|
await new Promise(resolve => setTimeout(resolve, 3100));
|
|
}
|
|
}
|
|
return execute(...args);
|
|
});
|
|
afterContinuationDispatchCheck = async ({ runId: checkedRunId, issueId: checkedIssueId }) => {
|
|
if (checkedIssueId !== issueId || checkedRunId !== resumedRunId) return;
|
|
claimedIssue = await db
|
|
.select({ status: issues.status, executionRunId: issues.executionRunId })
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null);
|
|
expect(claimedIssue).toEqual({ status: "in_progress", executionRunId: checkedRunId });
|
|
};
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Waiting for connection", status: "in_review", priority: "medium", assigneeAgentId: agentId });
|
|
await db.insert(issueComments).values({
|
|
id: wakeCommentId,
|
|
companyId,
|
|
issueId,
|
|
authorUserId: "local-board",
|
|
body: "Continue after the interaction result.",
|
|
});
|
|
if (interactionKind === "request_confirmation") {
|
|
await db.insert(issueThreadInteractions).values({
|
|
id: interactionId,
|
|
companyId,
|
|
issueId,
|
|
kind: "request_confirmation",
|
|
status: "rejected",
|
|
continuationPolicy: "wake_assignee",
|
|
payload: { version: 1, prompt: "Approve the plan?" },
|
|
result: { version: 1, outcome: "rejected", reason: "Needs more work" },
|
|
createdByAgentId: agentId,
|
|
resolvedAt: new Date(),
|
|
});
|
|
}
|
|
const { runId } = await seedQueuedRun({ companyId, agentId, issueId, wakeReason: "issue_commented",
|
|
contextExtras: { interactionId, interactionKind, interactionStatus, wakeCommentId,
|
|
originCommentIds: [wakeCommentId],
|
|
interactionResolvedAt: new Date().toISOString(), mutation: "interaction", source: `${interactionKind}.resolved`, forceFreshSession: true } });
|
|
resumedRunId = runId;
|
|
await heartbeat.resumeQueuedRuns();
|
|
// Adapter entry guarantees this exact execution is registered. Await its
|
|
// owning completion barrier before terminal assertions and fixture cleanup.
|
|
await adapterStarted;
|
|
await heartbeat.waitForRunExecutionDrain(runId);
|
|
expect((await db.select({ status: heartbeatRuns.status }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]?.status).toBe("succeeded");
|
|
expect(countExecuteCallsForRun(runId)).toBe(1);
|
|
expect(claimedIssue).toEqual({ status: "in_progress", executionRunId: runId });
|
|
});
|
|
|
|
async function seedAnsweredQuestion(input: {
|
|
companyId: string;
|
|
issueId: string;
|
|
agentId: string;
|
|
sourceRunId: string | null;
|
|
rejectedConfirmation?: boolean;
|
|
}) {
|
|
const interactionId = randomUUID();
|
|
const resolved = {
|
|
id: interactionId,
|
|
companyId: input.companyId,
|
|
issueId: input.issueId,
|
|
continuationPolicy: "wake_assignee",
|
|
createdByAgentId: input.agentId,
|
|
sourceRunId: input.sourceRunId,
|
|
resolvedByUserId: "responsible-user",
|
|
resolvedAt: new Date(),
|
|
};
|
|
await db.insert(issueThreadInteractions).values(input.rejectedConfirmation
|
|
? {
|
|
...resolved,
|
|
kind: "request_confirmation",
|
|
status: "rejected",
|
|
payload: { version: 1, prompt: "Approve the plan?" },
|
|
result: { version: 1, outcome: "rejected", reason: "Needs more work" },
|
|
}
|
|
: {
|
|
...resolved,
|
|
kind: "ask_user_questions",
|
|
status: "answered",
|
|
payload: { version: 1, questions: [{ id: "first-task-opening", prompt: "What would you like to do?", selectionMode: "single",
|
|
required: true, options: [{ id: "interview", label: "Interview me" }] }] },
|
|
result: { version: 1, answers: [{ questionId: "first-task-opening", optionIds: ["interview"] }] },
|
|
});
|
|
// Mirrors the context question-response delivery queues for an answer.
|
|
return {
|
|
interactionId,
|
|
interactionKind: input.rejectedConfirmation ? "request_confirmation" : "ask_user_questions",
|
|
interactionStatus: input.rejectedConfirmation ? "rejected" : "answered",
|
|
mutation: "interaction",
|
|
sourceRunId: input.sourceRunId,
|
|
externalChatContinuation: false,
|
|
source: "issue.interaction.respond",
|
|
};
|
|
}
|
|
|
|
it("starts the assignee when a question no run asked is answered on a todo task that has never started", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
let answeredRunId: string | undefined;
|
|
let claimedIssue: { status: string; executionRunId: string | null } | null = null;
|
|
afterContinuationDispatchCheck = async ({ runId: checkedRunId, issueId: checkedIssueId }) => {
|
|
if (checkedIssueId !== issueId || checkedRunId !== answeredRunId) return;
|
|
claimedIssue = await db
|
|
.select({ status: issues.status, executionRunId: issues.executionRunId })
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null);
|
|
};
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Onboarding first task", status: "todo", priority: "medium", assigneeAgentId: agentId });
|
|
const contextExtras = await seedAnsweredQuestion({ companyId, issueId, agentId, sourceRunId: null });
|
|
const { runId } = await seedQueuedRun({ companyId, agentId, issueId, wakeReason: "issue_commented",
|
|
invocationSource: "automation", contextExtras });
|
|
answeredRunId = runId;
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await vi.waitFor(async () => {
|
|
const current = await heartbeat.getRun(runId);
|
|
expect(countExecuteCallsForRun(runId), JSON.stringify({
|
|
status: current?.status, errorCode: current?.errorCode,
|
|
})).toBe(1);
|
|
}, { timeout: 5_000 });
|
|
await heartbeat.waitForRunExecutionDrain(runId);
|
|
|
|
expect((await heartbeat.getRun(runId))?.status).toBe("succeeded");
|
|
expect(claimedIssue).toEqual({ status: "in_progress", executionRunId: runId });
|
|
expect((await db.select({ startedAt: issues.startedAt }).from(issues).where(eq(issues.id, issueId)))[0]?.startedAt)
|
|
.toBeInstanceOf(Date);
|
|
});
|
|
|
|
it.each([
|
|
{ name: "a started task parked back in todo", startedAt: new Date("2026-01-01T00:00:00.000Z"), askedByRun: false, rejectedConfirmation: false },
|
|
{ name: "a question an agent run asked", startedAt: null, askedByRun: true, rejectedConfirmation: false },
|
|
{ name: "a rejected confirmation no run asked", startedAt: null, askedByRun: false, rejectedConfirmation: true },
|
|
])("keeps cancelling an answer continuation for $name", async ({ startedAt, askedByRun, rejectedConfirmation }) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Parked or agent-asked", status: "todo", priority: "medium",
|
|
assigneeAgentId: agentId, startedAt });
|
|
const [sourceRun] = askedByRun
|
|
? await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "assignment", status: "succeeded",
|
|
startedAt: new Date(), finishedAt: new Date(), contextSnapshot: { issueId } }).returning()
|
|
: [null];
|
|
const contextExtras = await seedAnsweredQuestion({ companyId, issueId, agentId, sourceRunId: sourceRun?.id ?? null,
|
|
rejectedConfirmation });
|
|
const { runId, wakeupRequestId } = await seedQueuedRun({ companyId, agentId, issueId, wakeReason: "issue_commented",
|
|
invocationSource: "automation", contextExtras });
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
const [run, wakeup, issue] = await Promise.all([
|
|
db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: agentWakeupRequests.status, error: agentWakeupRequests.error })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: issues.status }).from(issues).where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null),
|
|
]);
|
|
expect(run).toMatchObject({ status: "cancelled", errorCode: "issue_not_in_progress" });
|
|
expect(wakeup).toMatchObject({ status: "skipped", error: expect.stringContaining("no longer in_progress") });
|
|
expect(issue?.status).toBe("todo");
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
});
|
|
});
|