mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 22:56:58 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A new company starts with an onboarding first task. The task opens
with a greeting and an opening question card that the assignee agent
posts.
> - Onboarding does not wake the agent when it creates the task. The
user's answer must start the agent.
> - When the user clicks an answer, run dispatch cancels the agent's run
0.4 seconds later with `issue_not_in_progress`. Answer continuations
only start on `in_progress` or `in_review` tasks, and the onboarding
task is still `todo`.
> - The user sees nothing until an unrelated bounded retry starts the
agent about two minutes later. This is the first thing every new user
does.
> - This pull request lets an answer start a `todo` task when the task
has never started and no run asked the question.
> - The benefit is that the agent starts immediately after the first
answer. The protection for parked tasks does not change.
## Linked Issues or Issue Description
**What happened?**
I created a new company and clicked an option on the onboarding opening
question ("What would you like to do?"). The agent's run was cancelled
immediately with `issue_not_in_progress`: "Cancelled because
resolved-interaction continuation issue is no longer in_progress
(current status: todo) before the queued run could start". The task
stayed in `todo`. About two minutes later, a `transient_failure` bounded
retry of the cancelled run started the agent through the normal checkout
path.
**Expected behavior**
The agent starts as soon as the user answers the opening question.
**Steps to reproduce**
1. Create a new company and hire the first agent through onboarding.
2. Open the onboarding first task.
3. Click an option on the opening question card and submit.
4. Look at the agent's runs. The first run is `cancelled` with
`issue_not_in_progress`.
**Paperclip version or commit**
`master` at `86bc3db6938`. The same code is on `00ad9503fe3`.
**Relevant logs or output**
```
heartbeat_runs: status=cancelled error_code=issue_not_in_progress
"Cancelled because resolved-interaction continuation issue is no longer in_progress (current status: todo) before the queued run could start"
two minutes later: "Scheduled bounded retry 1/2" (retryReason transient_failure)
```
Related: #13068 added the onboarding opening question. #13318 made
answer continuations require `in_progress` or `in_review`.
## What Changed
- New `server/src/services/unstarted-task-question-answer.ts`.
`isUnstartedTaskQuestionAnswer` returns true only when all of these are
true:
- the interaction is an answered question (`ask_user_questions`, status
`answered`) on this task, and it has no `sourceRunId` (no run asked it);
- the task is `todo`;
- the task's `startedAt` is null.
It reads only server-owned rows. `startedAt` is never cleared, so a task
that started and was later moved back to `todo` does not match.
- `run-dispatch/adapters/postgres.ts`: the staleness facts include
`isUnstartedTaskQuestionAnswer`. The adapter reads it under the issue
lock, and only for a resolved-interaction continuation on a `todo` task.
- `run-dispatch/domain/policy.ts`: `decideQueuedRunStaleness` lets such
a continuation start from `todo`. The other answer continuations still
require `in_progress` or `in_review`.
- `services/heartbeat.ts`: the final continuation checkout adds `todo`
to the expected statuses only for the same case. The task then moves to
`in_progress` in the same guarded update.
## Verification
- `heartbeat-stale-queue-invalidation.test.ts` (embedded Postgres):
- New test: an answer to a question that no run asked, on a
never-started `todo` task, starts the assignee. The task moves to
`in_progress` under the run's execution lock. With the fix reverted,
this test fails with the reported error (`cancelled`,
`issue_not_in_progress`).
- New test: three regression cases stay cancelled with
`issue_not_in_progress`:
- an answer on a task that started and was moved back to `todo`;
- an answer to a question that an agent run asked;
- a rejected confirmation that no run asked.
Without the kind and status check, the confirmation case fails.
- `policy.test.ts`: the new fact allows `todo`. The same fact on a
`backlog` task is still cancelled.
- `pnpm --filter @paperclipai/server typecheck`: exit 0.
- `npx vitest run` on these suites: 8 files, 497 tests passed.
- stale-queue invalidation
- run-dispatch policy
- onboarding first-task routes
- onboarding first-task admission
- question-response delivery
- issue-thread interaction routes
- heartbeat retry scheduling
- legacy continuation authority
## Risks
- Low risk. The change only affects answer continuations on `todo` tasks
that have never started, where the question has no source run. All other
continuations behave as before.
- The exception is not limited to the onboarding card. Any answered
question that no run asked, on a never-started `todo` task, can now
start the assignee. This matches what a plain comment wake does on a
`todo` task. Confirmations and other interaction kinds keep the existing
rule. For example, rejecting a confirmation that no run asked still does
not start the task.
- The final checkout does one extra indexed read. It only runs for
resolved-interaction continuations on `todo` tasks.
- I did not find which component scheduled the two-minute
`transient_failure` retry. After this change, onboarding does not depend
on that retry.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs). Investigation, fix, and tests were
AI-assisted, with human direction.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
docs describe this behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
1989 lines
75 KiB
TypeScript
1989 lines
75 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { setImmediate as immediate } from "node:timers/promises";
|
|
import * as runState from "../services/heartbeat/run-state.js";
|
|
import { eq, sql } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
agents,
|
|
authUsers,
|
|
agentWakeupRequests,
|
|
agentRuntimeState,
|
|
companies,
|
|
companyMemberships,
|
|
costEvents,
|
|
createDb,
|
|
applyPendingMigrations,
|
|
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
|
|
documentRevisions,
|
|
documents,
|
|
environmentLeases,
|
|
nativeRunFinalizations,
|
|
heartbeatRuns,
|
|
issueComments,
|
|
issueDocuments,
|
|
issueThreadInteractions,
|
|
issues,
|
|
} from "@paperclipai/db";
|
|
import { ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY } from "@paperclipai/shared";
|
|
import {
|
|
getEmbeddedPostgresTestSupport,
|
|
startEmbeddedPostgresTestDatabase,
|
|
} from "./helpers/embedded-postgres.js";
|
|
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
|
import { drainHeartbeatRunsToQuiescence } from "./helpers/drain-heartbeat-runs.js";
|
|
import { truncateTablesWithDeadlockRetry } from "./helpers/truncate-with-deadlock-retry.js";
|
|
import {
|
|
MAX_TURN_CONTINUATION_RETRY_REASON,
|
|
MAX_TURN_CONTINUATION_WAKE_REASON,
|
|
heartbeatService,
|
|
} from "../services/heartbeat.ts";
|
|
import { runningProcesses, type AdapterExecutionContext, type AdapterExecutionResult } from "../adapters/index.ts";
|
|
import { recoveryService } from "../services/recovery/service.ts";
|
|
import { withQueuedCommentIdsInWakePayload } from "../services/issue-queued-comment-queue.ts";
|
|
|
|
const mockAdapterExecute = vi.hoisted(() =>
|
|
vi.fn(async (_context: AdapterExecutionContext): Promise<AdapterExecutionResult> => ({
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
errorMessage: null,
|
|
summary: "Stale-queue invalidation test run.",
|
|
provider: "test",
|
|
model: "test-model",
|
|
})),
|
|
);
|
|
|
|
vi.mock("../adapters/index.ts", async () => {
|
|
const actual = await vi.importActual<typeof import("../adapters/index.ts")>("../adapters/index.ts");
|
|
return {
|
|
...actual,
|
|
getServerAdapter: vi.fn(() => ({
|
|
supportsLocalAgentJwt: false,
|
|
execute: mockAdapterExecute,
|
|
})),
|
|
};
|
|
});
|
|
|
|
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_STALE_QUEUE_TEST_DATABASE_URL");
|
|
const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
|
|
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
|
|
|
if (!embeddedPostgresSupport.supported) {
|
|
console.warn(
|
|
`Skipping embedded Postgres heartbeat stale-queue invalidation tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
|
);
|
|
}
|
|
|
|
async function ensureIssueRelationsTable(db: ReturnType<typeof createDb>) {
|
|
await db.execute(sql.raw(`
|
|
CREATE TABLE IF NOT EXISTS "issue_relations" (
|
|
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
"company_id" uuid NOT NULL,
|
|
"issue_id" uuid NOT NULL,
|
|
"related_issue_id" uuid NOT NULL,
|
|
"type" text NOT NULL,
|
|
"created_by_agent_id" uuid,
|
|
"created_by_user_id" text,
|
|
"created_at" timestamptz NOT NULL DEFAULT now(),
|
|
"updated_at" timestamptz NOT NULL DEFAULT now()
|
|
);
|
|
`));
|
|
}
|
|
|
|
async function waitForCondition(fn: () => Promise<boolean>, timeoutMs = 3_000) {
|
|
const deadline = Date.now() + timeoutMs;
|
|
while (Date.now() < deadline) {
|
|
if (await fn()) return true;
|
|
await new Promise((resolve) => setTimeout(resolve, 50));
|
|
}
|
|
return fn();
|
|
}
|
|
|
|
async function cleanupHeartbeatInvalidationFixture(db: ReturnType<typeof createDb>) {
|
|
await truncateTablesWithDeadlockRetry(
|
|
db,
|
|
`
|
|
TRUNCATE TABLE
|
|
"company_skills",
|
|
"issue_comments",
|
|
"issue_documents",
|
|
"document_revisions",
|
|
"documents",
|
|
"issue_relations",
|
|
"issue_tree_holds",
|
|
"issues",
|
|
"heartbeat_run_events",
|
|
"cost_events",
|
|
"activity_log",
|
|
"heartbeat_runs",
|
|
"agent_wakeup_requests",
|
|
"agent_runtime_state",
|
|
"agents",
|
|
"companies"
|
|
RESTART IDENTITY CASCADE
|
|
`,
|
|
{ attempts: 10, delayMs: () => 100 },
|
|
);
|
|
}
|
|
|
|
type SeedOptions = {
|
|
agentName?: string;
|
|
agentRole?: string;
|
|
maxConcurrentRuns?: number;
|
|
heartbeatConfig?: Record<string, unknown>;
|
|
};
|
|
|
|
type SeedResult = {
|
|
companyId: string;
|
|
agentId: string;
|
|
};
|
|
|
|
describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => {
|
|
let db!: ReturnType<typeof createDb>;
|
|
let heartbeat!: ReturnType<typeof heartbeatService>;
|
|
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
let beforeContinuationDispatchCheck:
|
|
| ((input: { runId: string; issueId: string }) => Promise<void>)
|
|
| null = null;
|
|
let afterContinuationDispatchCheck:
|
|
| ((input: { runId: string; issueId: string }) => Promise<void>)
|
|
| null = null;
|
|
|
|
const countExecuteCallsForRun = (runId: string) =>
|
|
mockAdapterExecute.mock.calls.filter(([context]) => context?.runId === runId).length;
|
|
|
|
beforeAll(async () => {
|
|
if (external) await applyPendingMigrations(external);
|
|
else tempDb = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-stale-queue-");
|
|
db = createDb(external ?? tempDb!.connectionString);
|
|
heartbeat = heartbeatService(db, {
|
|
runtimeEnv: { ...process.env, PAPERCLIP_IN_WORKTREE: "false" },
|
|
beforeResolvedInteractionContinuationDispatchCheck: async (input) => {
|
|
await beforeContinuationDispatchCheck?.(input);
|
|
},
|
|
afterResolvedInteractionContinuationDispatchCheck: async (input) => {
|
|
await afterContinuationDispatchCheck?.(input);
|
|
},
|
|
});
|
|
await ensureIssueRelationsTable(db);
|
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
|
|
|
afterEach(async () => {
|
|
beforeContinuationDispatchCheck = null;
|
|
afterContinuationDispatchCheck = null;
|
|
mockAdapterExecute.mockReset();
|
|
mockAdapterExecute.mockImplementation(async () => ({
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
errorMessage: null,
|
|
summary: "Stale-queue invalidation test run.",
|
|
provider: "test",
|
|
model: "test-model",
|
|
}));
|
|
runningProcesses.clear();
|
|
// A table poll for "queued" or "running" rows cannot see a wakeup promise
|
|
// that has not yet written its run row. The shared drain helper awaits
|
|
// both in-flight wakeup promises and in-flight run executions first, then
|
|
// re-checks the run table, so a late writer cannot still be in flight when
|
|
// the TRUNCATE below runs.
|
|
await drainHeartbeatRunsToQuiescence(db, heartbeat);
|
|
await cleanupHeartbeatInvalidationFixture(db);
|
|
await db.delete(authUsers).where(eq(authUsers.id, "responsible-user"));
|
|
});
|
|
|
|
afterAll(async () => {
|
|
await tempDb?.cleanup();
|
|
});
|
|
|
|
async function seedCompanyAndAgent(opts: SeedOptions = {}): Promise<SeedResult> {
|
|
const companyId = randomUUID();
|
|
const agentId = randomUUID();
|
|
await db.insert(companies).values({
|
|
id: companyId,
|
|
name: "Paperclip",
|
|
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
|
defaultResponsibleUserId: "responsible-user",
|
|
requireBoardApprovalForNewAgents: false,
|
|
});
|
|
// Admission intersects agent authority with an existing responsible user
|
|
// and that user's active company membership.
|
|
await db.insert(authUsers).values({
|
|
id: "responsible-user",
|
|
name: "Fixture user",
|
|
email: "responsible-user@example.test",
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
}).onConflictDoNothing();
|
|
await db.insert(companyMemberships).values({
|
|
companyId,
|
|
principalType: "user",
|
|
principalId: "responsible-user",
|
|
status: "active",
|
|
membershipRole: "member",
|
|
});
|
|
await db.insert(agents).values({
|
|
id: agentId,
|
|
companyId,
|
|
name: opts.agentName ?? "ClaudeCoder",
|
|
role: opts.agentRole ?? "engineer",
|
|
status: "active",
|
|
adapterType: "codex_local",
|
|
adapterConfig: {},
|
|
runtimeConfig: {
|
|
heartbeat: {
|
|
wakeOnDemand: true,
|
|
maxConcurrentRuns: opts.maxConcurrentRuns ?? 1,
|
|
...(opts.heartbeatConfig ?? {}),
|
|
},
|
|
},
|
|
permissions: {},
|
|
});
|
|
return { companyId, agentId };
|
|
}
|
|
|
|
it.each(["legacy", "native"])("delivers an accepted non-assignee queue after its %s owner finished", async runtimeMode => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const [recipient] = await db.insert(agents).values({ companyId, name: "Saved feedback recipient", role: "engineer",
|
|
status: "idle", adapterType: "codex_local", adapterConfig: {}, runtimeConfig: { heartbeat: { wakeOnDemand: true } }, permissions: {},
|
|
}).returning();
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Existing accepted feedback", status: "in_progress", assigneeAgentId: agentId }).returning();
|
|
const [owner] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "assignment", status: "succeeded",
|
|
runtimeMode, nativeIssueId: runtimeMode === "native" ? issue.id : null, startedAt: new Date(), finishedAt: new Date(), contextSnapshot: { issueId: issue.id },
|
|
}).returning();
|
|
if (runtimeMode === "native") await db.insert(nativeRunFinalizations).values({ runId: owner.id, companyId, issueId: issue.id, phase: "committed" });
|
|
const [comment] = await db.insert(issueComments).values({ companyId, issueId: issue.id, body: "Saved feedback for the recipient",
|
|
authorAgentId: agentId, createdByRunId: owner.id,
|
|
}).returning();
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId, agentId: recipient.id, source: "automation",
|
|
reason: "issue_execution_deferred", status: "deferred_issue_execution", requestedByActorType: "agent", requestedByActorId: agentId,
|
|
payload: withQueuedCommentIdsInWakePayload({ issueId: issue.id, _paperclipWakeContext: {
|
|
issueId: issue.id, wakeReason: "issue_comment_mentioned", wakeSource: "automation",
|
|
} }, [comment.id]),
|
|
}).returning();
|
|
await heartbeat.resumeQueuedRuns();
|
|
const [delivered] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id));
|
|
expect(delivered.runId).toBeTruthy();
|
|
// The receipt can commit before execution enters liveRunExecutions.
|
|
// Observe actual adapter admission before awaiting its drain.
|
|
await vi.waitFor(async () => {
|
|
const current = await heartbeat.getRun(delivered.runId!);
|
|
expect(countExecuteCallsForRun(delivered.runId!), JSON.stringify({
|
|
status: current?.status, errorCode: current?.errorCode,
|
|
})).toBe(1);
|
|
}, { timeout: 5_000 });
|
|
await heartbeat.waitForRunExecutionDrain(delivered.runId!);
|
|
expect(countExecuteCallsForRun(delivered.runId!)).toBe(1);
|
|
expect((await heartbeat.getRun(delivered.runId!))?.contextSnapshot).toMatchObject({ wakeCommentIds: [comment.id] });
|
|
await heartbeat.resumeQueuedRuns();
|
|
expect(countExecuteCallsForRun(delivered.runId!)).toBe(1);
|
|
expect((await db.select().from(issueComments).where(eq(issueComments.id, comment.id)))[0]?.body).toBe(comment.body);
|
|
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0]?.assigneeAgentId).toBe(agentId);
|
|
});
|
|
|
|
it.each(["preparing", "verifying", "resuming"] as const)("keeps assigned work queued through %s, including recovery with timers disabled", async lifecycleState => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({ heartbeatConfig: { enabled: false } });
|
|
await db.update(agents).set({ status: "paused", lifecycleState }).where(eq(agents.id, agentId));
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Work accepted during setup", status: "todo", assigneeAgentId: agentId }).returning();
|
|
const run = await heartbeat.wakeup(agentId, { source: "assignment", triggerDetail: "system", reason: "issue_assigned",
|
|
requestedByActorType: "system", payload: { issueId: issue.id } });
|
|
expect(run?.status).toBe("queued");
|
|
await heartbeat.resumeQueuedRuns();
|
|
expect((await heartbeat.getRun(run!.id))?.status).toBe("queued");
|
|
expect(countExecuteCallsForRun(run!.id)).toBe(0);
|
|
expect((await heartbeat.reconcileStrandedAssignedIssues()).escalated).toBe(0);
|
|
expect((await db.select().from(issues).where(eq(issues.id, issue.id)))[0]).toMatchObject({ status: "todo", assigneeAgentId: agentId });
|
|
await db.update(agents).set({ status: "idle", lifecycleState: "ready" }).where(eq(agents.id, agentId));
|
|
const recovered = heartbeatService(db, { runtimeEnv: { ...process.env, PAPERCLIP_IN_WORKTREE: "false" } });
|
|
await recovered.resumeQueuedRuns();
|
|
await recovered.drainActiveRunExecutions();
|
|
expect(countExecuteCallsForRun(run!.id)).toBe(1);
|
|
});
|
|
|
|
async function seedQueuedRun(input: {
|
|
companyId: string;
|
|
agentId: string;
|
|
issueId: string;
|
|
wakeReason: string;
|
|
contextExtras?: Record<string, unknown>;
|
|
invocationSource?: "assignment" | "automation";
|
|
scheduledRetryReason?: string | null;
|
|
}) {
|
|
const wakeupRequestId = randomUUID();
|
|
const runId = randomUUID();
|
|
await db.insert(agentWakeupRequests).values({
|
|
id: wakeupRequestId,
|
|
companyId: input.companyId,
|
|
agentId: input.agentId,
|
|
source: input.invocationSource ?? "assignment",
|
|
triggerDetail: "system",
|
|
reason: input.wakeReason,
|
|
payload: { issueId: input.issueId },
|
|
status: "queued",
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId: input.companyId,
|
|
agentId: input.agentId,
|
|
invocationSource: input.invocationSource ?? "assignment",
|
|
triggerDetail: "system",
|
|
status: "queued",
|
|
wakeupRequestId,
|
|
scheduledRetryReason: input.scheduledRetryReason ?? null,
|
|
contextSnapshot: {
|
|
issueId: input.issueId,
|
|
wakeReason: input.wakeReason,
|
|
...(input.contextExtras ?? {}),
|
|
},
|
|
});
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId })
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
return { runId, wakeupRequestId };
|
|
}
|
|
|
|
async function seedContinuationSummary(input: {
|
|
companyId: string;
|
|
issueId: string;
|
|
agentId: string;
|
|
body: string;
|
|
}) {
|
|
const documentId = randomUUID();
|
|
const revisionId = randomUUID();
|
|
await db.insert(documents).values({
|
|
id: documentId,
|
|
companyId: input.companyId,
|
|
title: "Continuation Summary",
|
|
format: "markdown",
|
|
latestBody: input.body,
|
|
latestRevisionId: revisionId,
|
|
latestRevisionNumber: 1,
|
|
createdByAgentId: input.agentId,
|
|
updatedByAgentId: input.agentId,
|
|
});
|
|
await db.insert(documentRevisions).values({
|
|
id: revisionId,
|
|
companyId: input.companyId,
|
|
documentId,
|
|
revisionNumber: 1,
|
|
title: "Continuation Summary",
|
|
format: "markdown",
|
|
body: input.body,
|
|
createdByAgentId: input.agentId,
|
|
});
|
|
await db.insert(issueDocuments).values({
|
|
companyId: input.companyId,
|
|
issueId: input.issueId,
|
|
documentId,
|
|
key: ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY,
|
|
});
|
|
}
|
|
|
|
it("joins a claimed execution whose initial run read has not reached live registration", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Held preparation", status: "todo", assigneeAgentId: agentId }).returning();
|
|
const queued = await seedQueuedRun({ companyId, agentId, issueId: issue.id, wakeReason: "issue_assigned" });
|
|
const entered = Promise.withResolvers<void>(), release = Promise.withResolvers<void>();
|
|
const factory = runState.createHeartbeatRunState;
|
|
let held = false;
|
|
const spy = vi.spyOn(runState, "createHeartbeatRunState").mockImplementation((...args) => {
|
|
const owner = factory(...args);
|
|
const read = owner.getRun;
|
|
return { ...owner, getRun: async (...readArgs: Parameters<typeof read>) => {
|
|
const row = await read(...readArgs);
|
|
if (!held && row?.id === queued.runId && row.status === "running") {
|
|
held = true; entered.resolve(); await release.promise;
|
|
}
|
|
return row;
|
|
} };
|
|
});
|
|
const recovered = heartbeatService(db, { runtimeEnv: { ...process.env, PAPERCLIP_IN_WORKTREE: "false" } });
|
|
try {
|
|
await recovered.resumeQueuedRuns();
|
|
await entered.promise;
|
|
// Receipt admission is earlier than live-run registration. The legacy
|
|
// run-specific drain can return here; the tracked execution must not.
|
|
await recovered.waitForRunExecutionDrain(queued.runId);
|
|
expect(countExecuteCallsForRun(queued.runId)).toBe(0);
|
|
let settled = false;
|
|
const draining = recovered.drainActiveRunExecutions().then(() => { settled = true; });
|
|
await immediate();
|
|
expect(settled).toBe(false);
|
|
release.resolve();
|
|
await draining;
|
|
expect(countExecuteCallsForRun(queued.runId)).toBe(1);
|
|
expect(await recovered.getRun(queued.runId)).toMatchObject({ status: "succeeded", runtimeMode: "legacy" });
|
|
} finally {
|
|
release.resolve();
|
|
await recovered.drainActiveRunExecutions();
|
|
spy.mockRestore();
|
|
}
|
|
});
|
|
|
|
it.each([
|
|
{ name: "assignments", sameIssue: true, first: "assignment", second: "assignment" },
|
|
{ name: "unrelated tasks", sameIssue: false, first: "assignment", second: "assignment" },
|
|
{ name: "assignment then direct comment", sameIssue: true, first: "assignment", second: "direct" },
|
|
{ name: "direct comment then assignment", sameIssue: true, first: "direct", second: "assignment" },
|
|
{ name: "assignment then queued comment", sameIssue: true, first: "assignment", second: "queued" },
|
|
{ name: "queued comment then assignment", sameIssue: true, first: "queued", second: "assignment" },
|
|
{ name: "queued comments", sameIssue: true, first: "queued", second: "queued" },
|
|
])("serializes issue claims without serializing unrelated work: $name", async ({ sameIssue, first, second }) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({ maxConcurrentRuns: 2 });
|
|
const firstIssueId = randomUUID();
|
|
const secondIssueId = sameIssue ? firstIssueId : randomUUID();
|
|
for (const id of new Set([firstIssueId, secondIssueId])) {
|
|
await db.insert(issues).values({ id, companyId, title: "Concurrent assignment", status: "todo", assigneeAgentId: agentId });
|
|
}
|
|
async function queue(issueId: string, kind: string) {
|
|
const commentId = kind === "assignment" ? null : randomUUID();
|
|
if (commentId) await db.insert(issueComments).values({ id: commentId, companyId, issueId, authorUserId: "local-board", body: "Continue this task." });
|
|
const queued = await seedQueuedRun({ companyId, agentId, issueId,
|
|
wakeReason: commentId ? "issue_commented" : "issue_assigned",
|
|
invocationSource: commentId ? "automation" : "assignment",
|
|
contextExtras: commentId ? { wakeCommentIds: [commentId], wakeCommentId: commentId } : {},
|
|
});
|
|
if (kind === "queued") await db.update(agentWakeupRequests).set({
|
|
payload: withQueuedCommentIdsInWakePayload({ issueId }, [commentId!]),
|
|
}).where(eq(agentWakeupRequests.id, queued.wakeupRequestId));
|
|
}
|
|
await queue(firstIssueId, first);
|
|
await queue(secondIssueId, second);
|
|
let release!: () => void;
|
|
const gate = new Promise<void>((resolve) => { release = resolve; });
|
|
const entered = Promise.withResolvers<void>();
|
|
const expectedExecutions = sameIssue ? 1 : 2;
|
|
let executions = 0;
|
|
mockAdapterExecute.mockImplementation(async () => {
|
|
if (++executions === expectedExecutions) entered.resolve();
|
|
await gate;
|
|
return { exitCode: 0, signal: null, timedOut: false, errorMessage: null, summary: "Assignment finished.", provider: "test", model: "test-model" };
|
|
});
|
|
try {
|
|
await heartbeat.resumeQueuedRuns();
|
|
await entered.promise;
|
|
expect(mockAdapterExecute).toHaveBeenCalledTimes(expectedExecutions);
|
|
const runs = await db.select().from(heartbeatRuns);
|
|
const running = runs.filter((run) => run.status === "running");
|
|
expect(running).toHaveLength(sameIssue ? 1 : 2);
|
|
expect(runs.filter((run) => run.status === "queued")).toHaveLength(sameIssue ? 1 : 0);
|
|
for (const run of running) {
|
|
const [issue] = await db.select().from(issues).where(eq(issues.id, (run.contextSnapshot as { issueId: string }).issueId));
|
|
expect(issue.executionRunId).toBe(run.id);
|
|
}
|
|
} finally {
|
|
release();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
}
|
|
});
|
|
|
|
it.each(["active_lease", "pending_cleanup", "failed_cleanup", "finalizer_lease", "workspace_finalization",
|
|
"retained_ready", "retained_missing_receipt", "retained_failed", "retained_wrong_policy"])(
|
|
"waits for durable cleanup from another controller: %s", async (pending) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID(), previousId = randomUUID();
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Remote cleanup", status: "todo", assigneeAgentId: agentId });
|
|
await db.insert(heartbeatRuns).values({ id: previousId, companyId, agentId,
|
|
status: "succeeded", runtimeMode: "native", nativeIssueId: issueId,
|
|
invocationSource: "assignment", contextSnapshot: { issueId }, finishedAt: new Date() });
|
|
await db.update(issues).set({ executionRunId: previousId }).where(eq(issues.id, issueId));
|
|
const leaseId = randomUUID();
|
|
const retained = pending.startsWith("retained_");
|
|
const hasLease = retained || ["active_lease", "pending_cleanup", "failed_cleanup"].includes(pending);
|
|
if (hasLease) await db.insert(environmentLeases).values({ id: leaseId, companyId, issueId,
|
|
heartbeatRunId: previousId, provider: "daytona",
|
|
status: retained ? "retained" : pending === "pending_cleanup" ? "pending_cleanup" : "released",
|
|
leasePolicy: pending === "retained_wrong_policy" ? "retain_on_failure" : retained ? "reuse_by_environment" : "ephemeral",
|
|
releasedAt: retained || pending === "active_lease" ? null : new Date(),
|
|
cleanupStatus: ["retained_ready", "retained_wrong_policy"].includes(pending) ? "success"
|
|
: ["failed_cleanup", "retained_failed"].includes(pending) ? "failed" : null });
|
|
await db.insert(nativeRunFinalizations).values({ runId: previousId, companyId, issueId,
|
|
phase: pending === "workspace_finalization" ? "workspace_finalizing" : "committed",
|
|
leaseOwner: pending === "finalizer_lease" ? "another-controller" : null });
|
|
const next = await seedQueuedRun({ companyId, agentId, issueId, wakeReason: "issue_assigned", invocationSource: "assignment" });
|
|
mockAdapterExecute.mockImplementation(async () => {
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, issueId));
|
|
return { exitCode: 0, signal: null, timedOut: false, errorMessage: null,
|
|
summary: "Task completed", provider: "test", model: "test-model" };
|
|
});
|
|
// No in-memory executor exists in this service. Durable ownership alone
|
|
// must fence a second controller until cleanup settles. A verified warm
|
|
// retention receipt is already a settled boundary and allows continuation.
|
|
await heartbeat.resumeQueuedRuns();
|
|
if (pending !== "retained_ready") {
|
|
expect((await heartbeat.getRun(next.runId))?.status).toBe("queued");
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0].executionRunId).toBe(previousId);
|
|
if (hasLease) await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success" }).where(eq(environmentLeases.id, leaseId));
|
|
await db.update(nativeRunFinalizations).set({ phase: "committed", leaseOwner: null }).where(eq(nativeRunFinalizations.runId, previousId));
|
|
await heartbeat.resumeQueuedRuns();
|
|
}
|
|
await heartbeat.drainActiveRunExecutions();
|
|
expect((await heartbeat.getRun(next.runId))?.status).toBe("succeeded");
|
|
expect(mockAdapterExecute).toHaveBeenCalledTimes(1);
|
|
},
|
|
);
|
|
|
|
it("skips generic timer wakes with no actionable assigned work before adapter execution", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
enabled: true,
|
|
skipTimerWhenNoActionableWork: true,
|
|
},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "timer",
|
|
triggerDetail: "system",
|
|
});
|
|
|
|
expect(run).toBeNull();
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
const runRows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns);
|
|
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "heartbeat.timer.no_actionable_work",
|
|
});
|
|
expect(wakeup?.payload).toMatchObject({
|
|
heartbeatSkip: {
|
|
reason: expect.stringContaining("No assigned todo or in_progress issue"),
|
|
},
|
|
});
|
|
expect(runRows).toHaveLength(0);
|
|
});
|
|
|
|
it("checks guarded issue status and assignee under the enqueue lock", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const parkedIssueId = randomUUID();
|
|
const reassignedIssueId = randomUUID();
|
|
await db.insert(issues).values([
|
|
{
|
|
id: parkedIssueId,
|
|
companyId,
|
|
title: "Parked connection intent",
|
|
status: "backlog" as const,
|
|
priority: "medium" as const,
|
|
assigneeAgentId: agentId,
|
|
},
|
|
{
|
|
id: reassignedIssueId,
|
|
companyId,
|
|
title: "Reassigned connection intent",
|
|
status: "in_progress" as const,
|
|
priority: "medium" as const,
|
|
assigneeAgentId: null,
|
|
},
|
|
]);
|
|
|
|
for (const issueId of [parkedIssueId, reassignedIssueId]) {
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "automation",
|
|
triggerDetail: "system",
|
|
reason: "issue_commented",
|
|
payload: { issueId, interactionId: randomUUID() },
|
|
contextSnapshot: { issueId, wakeReason: "issue_commented" },
|
|
requestedByActorType: "user",
|
|
requestedByActorId: "responsible-user",
|
|
issueStateGuard: {
|
|
statuses: ["in_progress"],
|
|
assigneeAgentId: agentId,
|
|
},
|
|
});
|
|
expect(run).toBeNull();
|
|
}
|
|
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
expect(await db.select().from(heartbeatRuns)).toHaveLength(0);
|
|
expect(await db.select({ status: agentWakeupRequests.status, reason: agentWakeupRequests.reason })
|
|
.from(agentWakeupRequests)).toEqual([
|
|
{ status: "skipped", reason: "issue_state_guard_mismatch" },
|
|
{ status: "skipped", reason: "issue_state_guard_mismatch" },
|
|
]);
|
|
});
|
|
|
|
it.each([
|
|
{ runtimeMode: "native", status: "done", reassigned: false },
|
|
{ runtimeMode: "legacy", status: "done", reassigned: false },
|
|
{ runtimeMode: "native", status: "cancelled", reassigned: false },
|
|
{ runtimeMode: "native", status: "backlog", reassigned: false },
|
|
{ runtimeMode: "native", status: "in_review", reassigned: false },
|
|
{ runtimeMode: "native", status: "blocked", reassigned: false },
|
|
{ runtimeMode: "native", status: "in_progress", reassigned: true },
|
|
] as const)("skips stale $runtimeMode productive recovery after status=$status reassigned=$reassigned commits under the enqueue lock", async ({ runtimeMode, status, reassigned }) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
const runId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Completion racing with productive recovery",
|
|
status: "in_progress",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "assignment",
|
|
runtimeMode,
|
|
status: "succeeded",
|
|
livenessState: "completed",
|
|
contextSnapshot: { issueId, wakeReason: "issue_assigned" },
|
|
startedAt: new Date(),
|
|
finishedAt: new Date(),
|
|
});
|
|
|
|
// Let the real sweep select its in-progress snapshot, then hold the issue
|
|
// lock until the real enqueue transaction is waiting on the newer state.
|
|
const enqueueWakeup = vi.fn(async (...[targetAgentId, options]: Parameters<typeof heartbeat.wakeup>) => {
|
|
let pendingWake!: ReturnType<typeof heartbeat.wakeup>;
|
|
await db.transaction(async (tx) => {
|
|
await tx.update(issues).set({
|
|
status,
|
|
...(reassigned ? { assigneeAgentId: null, assigneeUserId: "responsible-user" } : {}),
|
|
}).where(eq(issues.id, issueId));
|
|
const [{ pid }] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
|
|
pendingWake = heartbeat.wakeup(targetAgentId, options);
|
|
try {
|
|
expect(await waitForCondition(async () => {
|
|
const [{ waiting }] = await db.execute<{ waiting: boolean }>(sql`
|
|
select exists (
|
|
select 1 from pg_stat_activity
|
|
where ${pid} = any(pg_blocking_pids(pid))
|
|
) as waiting
|
|
`);
|
|
return waiting;
|
|
})).toBe(true);
|
|
} catch (error) {
|
|
// Observe a pending rejection even if the lock assertion fails.
|
|
void pendingWake.catch(() => {});
|
|
throw error;
|
|
}
|
|
});
|
|
return pendingWake;
|
|
});
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(enqueueWakeup).toHaveBeenCalledOnce();
|
|
expect(result).toMatchObject({ continuationRequeued: 0, escalated: 0, skipped: 1, issueIds: [] });
|
|
expect(await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)).toEqual([{ id: runId }]);
|
|
expect(await db.select().from(issueComments)).toHaveLength(0);
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
const [wakeup] = await db.select().from(agentWakeupRequests);
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "issue_state_guard_mismatch",
|
|
runId: null,
|
|
payload: {
|
|
heartbeatSkip: {
|
|
expectedStatuses: ["in_progress"],
|
|
actualStatus: status,
|
|
expectedAssigneeAgentId: agentId,
|
|
actualAssigneeAgentId: reassigned ? null : agentId,
|
|
},
|
|
},
|
|
});
|
|
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
|
expect(issue).toMatchObject({ status, assigneeAgentId: reassigned ? null : agentId });
|
|
});
|
|
|
|
it("cancels a resolved connection-intent wake parked before queued-run claim", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Connection intent parked after enqueue",
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
const { runId, wakeupRequestId } = await seedQueuedRun({
|
|
companyId,
|
|
agentId,
|
|
issueId,
|
|
wakeReason: "issue_commented",
|
|
invocationSource: "automation",
|
|
contextExtras: {
|
|
interactionId: randomUUID(),
|
|
interactionKind: "connection_intent",
|
|
interactionStatus: "accepted",
|
|
interactionResolvedAt: "2026-08-28T13:30:00.000Z",
|
|
mutation: "interaction",
|
|
source: "connection_intent.resolved",
|
|
},
|
|
});
|
|
|
|
await db.update(issues).set({ status: "backlog" }).where(eq(issues.id, issueId));
|
|
await heartbeat.resumeQueuedRuns();
|
|
|
|
const [run, wakeup, issue] = await Promise.all([
|
|
db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: agentWakeupRequests.status, error: agentWakeupRequests.error })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: issues.status }).from(issues).where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null),
|
|
]);
|
|
expect(run).toMatchObject({ status: "cancelled", errorCode: "issue_not_in_progress" });
|
|
expect(wakeup).toMatchObject({ status: "skipped", error: expect.stringContaining("no longer in_progress") });
|
|
expect(issue?.status).toBe("backlog");
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
});
|
|
|
|
it("does not re-open a resolved connection-intent issue parked after claim", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Connection intent parked between claim and checkout",
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
const { runId, wakeupRequestId } = await seedQueuedRun({
|
|
companyId,
|
|
agentId,
|
|
issueId,
|
|
wakeReason: "issue_commented",
|
|
invocationSource: "automation",
|
|
contextExtras: {
|
|
interactionId: randomUUID(),
|
|
interactionKind: "connection_intent",
|
|
interactionStatus: "accepted",
|
|
interactionResolvedAt: "2026-08-28T13:30:00.000Z",
|
|
mutation: "interaction",
|
|
source: "connection_intent.resolved",
|
|
},
|
|
});
|
|
|
|
await db.execute(sql.raw(`
|
|
CREATE OR REPLACE FUNCTION park_connection_intent_after_claim()
|
|
RETURNS trigger AS $trigger$
|
|
BEGIN
|
|
IF NEW.id = '${runId}'::uuid AND NEW.status = 'running' THEN
|
|
UPDATE issues SET status = 'backlog' WHERE id = '${issueId}'::uuid;
|
|
END IF;
|
|
RETURN NEW;
|
|
END;
|
|
$trigger$ LANGUAGE plpgsql;
|
|
|
|
CREATE TRIGGER park_connection_intent_after_claim
|
|
AFTER UPDATE OF status ON heartbeat_runs
|
|
FOR EACH ROW EXECUTE FUNCTION park_connection_intent_after_claim();
|
|
`));
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
const [run, wakeup, issue] = await Promise.all([
|
|
db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: agentWakeupRequests.status, error: agentWakeupRequests.error })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: issues.status }).from(issues).where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null),
|
|
]);
|
|
expect(run).toMatchObject({ status: "cancelled", errorCode: "issue_not_in_progress" });
|
|
expect(wakeup).toMatchObject({ status: "skipped", error: expect.stringContaining("no longer in_progress") });
|
|
expect(issue?.status).toBe("backlog");
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
});
|
|
|
|
it.each([
|
|
{
|
|
mutation: "parked",
|
|
expectedErrorCode: "issue_not_in_progress",
|
|
expectedError: "no longer in_progress",
|
|
},
|
|
{
|
|
mutation: "reassigned",
|
|
expectedErrorCode: "issue_assignee_changed",
|
|
expectedError: "changed assignee",
|
|
},
|
|
])(
|
|
"cancels a resolved connection-intent wake $mutation after checkout but before adapter dispatch",
|
|
async ({ mutation, expectedErrorCode, expectedError }) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const replacementAgentId = randomUUID();
|
|
if (mutation === "reassigned") {
|
|
await db.insert(agents).values({
|
|
id: replacementAgentId,
|
|
companyId,
|
|
name: "ReplacementCoder",
|
|
role: "engineer",
|
|
status: "active",
|
|
adapterType: "codex_local",
|
|
adapterConfig: {},
|
|
runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } },
|
|
permissions: {},
|
|
});
|
|
}
|
|
const issueId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: `Connection intent ${mutation} at final dispatch`,
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
const { runId, wakeupRequestId } = await seedQueuedRun({
|
|
companyId,
|
|
agentId,
|
|
issueId,
|
|
wakeReason: "issue_commented",
|
|
invocationSource: "automation",
|
|
contextExtras: {
|
|
interactionId: randomUUID(),
|
|
interactionKind: "connection_intent",
|
|
interactionStatus: "accepted",
|
|
interactionResolvedAt: "2026-08-28T13:30:00.000Z",
|
|
mutation: "interaction",
|
|
source: "connection_intent.resolved",
|
|
},
|
|
});
|
|
beforeContinuationDispatchCheck = async ({ runId: guardedRunId, issueId: guardedIssueId }) => {
|
|
expect(guardedRunId).toBe(runId);
|
|
expect(guardedIssueId).toBe(issueId);
|
|
await db
|
|
.update(issues)
|
|
.set(mutation === "parked"
|
|
? { status: "backlog", updatedAt: new Date() }
|
|
: { assigneeAgentId: replacementAgentId, updatedAt: new Date() })
|
|
.where(eq(issues.id, issueId));
|
|
};
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
const [run, wakeup, issue] = await Promise.all([
|
|
db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: agentWakeupRequests.status, error: agentWakeupRequests.error })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: issues.status, assigneeAgentId: issues.assigneeAgentId })
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null),
|
|
]);
|
|
expect(run).toMatchObject({ status: "cancelled", errorCode: expectedErrorCode });
|
|
expect(wakeup).toMatchObject({ status: "skipped", error: expect.stringContaining(expectedError) });
|
|
expect(issue).toMatchObject(mutation === "parked"
|
|
? { status: "backlog", assigneeAgentId: agentId }
|
|
: { status: "in_progress", assigneeAgentId: replacementAgentId });
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
},
|
|
);
|
|
|
|
it("rejects ownership changes immediately before the final continuation handoff", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Connection intent parked at the atomic dispatch gate",
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
const { runId } = await seedQueuedRun({
|
|
companyId,
|
|
agentId,
|
|
issueId,
|
|
wakeReason: "issue_commented",
|
|
invocationSource: "automation",
|
|
contextExtras: {
|
|
interactionId: randomUUID(),
|
|
interactionKind: "connection_intent",
|
|
interactionStatus: "accepted",
|
|
interactionResolvedAt: "2026-08-28T13:30:00.000Z",
|
|
mutation: "interaction",
|
|
source: "connection_intent.resolved",
|
|
},
|
|
});
|
|
|
|
const ordering: string[] = [];
|
|
let parkPromise: Promise<unknown> | null = null;
|
|
afterContinuationDispatchCheck = async ({ runId: guardedRunId, issueId: guardedIssueId }) => {
|
|
expect(guardedRunId).toBe(runId);
|
|
expect(guardedIssueId).toBe(issueId);
|
|
ordering.push("validated");
|
|
parkPromise = Promise.resolve(
|
|
db
|
|
.update(issues)
|
|
.set({
|
|
status: "backlog",
|
|
checkoutRunId: null,
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(issues.id, issueId))
|
|
.returning({ id: issues.id }),
|
|
).then((rows) => {
|
|
expect(rows).toHaveLength(1);
|
|
ordering.push("parked");
|
|
});
|
|
// Admission is committed before adapter-owned setup. This concurrent
|
|
// update must not wait on a lock held by the adapter callback.
|
|
await new Promise((resolve) => setTimeout(resolve, 25));
|
|
await parkPromise;
|
|
expect(ordering).toEqual(["validated", "parked"]);
|
|
};
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await parkPromise;
|
|
|
|
expect((await heartbeat.getRun(runId))?.status).toBe("cancelled");
|
|
const issue = await db
|
|
.select({ status: issues.status })
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null);
|
|
expect(issue?.status).toBe("backlog");
|
|
expect(ordering).toEqual(["validated", "parked"]);
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
});
|
|
|
|
it("leaves ineligible timers disarmed without writing skipped wakes or moving their baseline", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
enabled: true,
|
|
intervalSec: 60,
|
|
skipTimerWhenNoActionableWork: true,
|
|
},
|
|
});
|
|
const now = new Date();
|
|
await db
|
|
.update(agents)
|
|
.set({ lastHeartbeatAt: new Date(now.getTime() - 120_000) })
|
|
.where(eq(agents.id, agentId));
|
|
|
|
const firstTick = await heartbeat.tickTimers(now);
|
|
const secondTick = await heartbeat.tickTimers(now);
|
|
|
|
expect(firstTick.skipped).toBe(0);
|
|
expect(await heartbeat.nextAgentTimerAt()).toBeNull();
|
|
expect(secondTick.skipped).toBe(0);
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const wakeups = await db
|
|
.select({ reason: agentWakeupRequests.reason })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
const [agent] = await db
|
|
.select({ lastHeartbeatAt: agents.lastHeartbeatAt })
|
|
.from(agents)
|
|
.where(eq(agents.id, agentId));
|
|
|
|
expect(wakeups).toEqual([]);
|
|
expect(agent?.lastHeartbeatAt).toBeInstanceOf(Date);
|
|
expect(agent?.lastHeartbeatAt?.getTime()).toBe(now.getTime() - 120_000);
|
|
});
|
|
|
|
it("atomically claims a due timer interval across overlapping scheduler ticks", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
enabled: true,
|
|
intervalSec: 60,
|
|
},
|
|
});
|
|
const now = new Date();
|
|
await db
|
|
.update(agents)
|
|
.set({
|
|
createdAt: new Date(now.getTime() - 120_000),
|
|
lastHeartbeatAt: null,
|
|
})
|
|
.where(eq(agents.id, agentId));
|
|
|
|
const results = await Promise.all([
|
|
heartbeat.tickTimers(now),
|
|
heartbeat.tickTimers(now),
|
|
]);
|
|
|
|
expect(results.reduce((total, result) => total + result.enqueued, 0)).toBe(1);
|
|
|
|
const runs = await db
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
contextSnapshot: heartbeatRuns.contextSnapshot,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.agentId, agentId));
|
|
const [agent] = await db
|
|
.select({ lastHeartbeatAt: agents.lastHeartbeatAt })
|
|
.from(agents)
|
|
.where(eq(agents.id, agentId));
|
|
|
|
expect(runs).toHaveLength(1);
|
|
expect(runs[0]?.contextSnapshot).toMatchObject({
|
|
timerClaimWasFirstHeartbeat: true,
|
|
});
|
|
expect(agent?.lastHeartbeatAt?.getTime()).toBeGreaterThanOrEqual(now.getTime());
|
|
});
|
|
|
|
it("allows generic timer wakes when the agent has assigned todo work", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
enabled: true,
|
|
skipTimerWhenNoActionableWork: true,
|
|
},
|
|
});
|
|
await db.insert(issues).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
title: "Assigned work",
|
|
status: "todo",
|
|
priority: "high",
|
|
assigneeAgentId: agentId,
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "timer",
|
|
triggerDetail: "system",
|
|
});
|
|
|
|
expect(run).not.toBeNull();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
expect(countExecuteCallsForRun(run!.id)).toBe(1);
|
|
});
|
|
|
|
it("allows legacy generic timer wakes by default when no skip policy is set", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
enabled: true,
|
|
},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "timer",
|
|
triggerDetail: "system",
|
|
});
|
|
|
|
expect(run).not.toBeNull();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
expect(countExecuteCallsForRun(run!.id)).toBe(1);
|
|
});
|
|
|
|
it("allows explicit proactive generic timer wakes without assigned issue work", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
enabled: true,
|
|
skipTimerWhenNoActionableWork: false,
|
|
},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "timer",
|
|
triggerDetail: "system",
|
|
});
|
|
|
|
expect(run).not.toBeNull();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
expect(countExecuteCallsForRun(run!.id)).toBe(1);
|
|
});
|
|
|
|
it("skips wakes before queueing when per-agent daily run cap is reached", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyRuns: 1,
|
|
},
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "succeeded",
|
|
createdAt: new Date(),
|
|
startedAt: new Date(),
|
|
finishedAt: new Date(),
|
|
contextSnapshot: {},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
});
|
|
|
|
expect(run).toBeNull();
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "heartbeat.daily_run_limit",
|
|
});
|
|
expect(wakeup?.payload).toMatchObject({
|
|
heartbeatSkip: {
|
|
observed: 1,
|
|
limit: 1,
|
|
},
|
|
});
|
|
});
|
|
|
|
it("treats zero daily run cap as a hard stop", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyRuns: 0,
|
|
},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
});
|
|
|
|
expect(run).toBeNull();
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "heartbeat.daily_run_limit",
|
|
});
|
|
expect(wakeup?.payload).toMatchObject({
|
|
heartbeatSkip: {
|
|
observed: 0,
|
|
limit: 0,
|
|
},
|
|
});
|
|
});
|
|
|
|
it("counts started cancelled runs toward the per-agent daily run cap", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyRuns: 1,
|
|
},
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "cancelled",
|
|
createdAt: new Date(),
|
|
startedAt: new Date(),
|
|
finishedAt: new Date(),
|
|
contextSnapshot: {},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
});
|
|
|
|
expect(run).toBeNull();
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "heartbeat.daily_run_limit",
|
|
});
|
|
expect(wakeup?.payload).toMatchObject({
|
|
heartbeatSkip: {
|
|
observed: 1,
|
|
limit: 1,
|
|
},
|
|
});
|
|
});
|
|
|
|
it("coalesces same-issue wakes before enforcing the daily run cap", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyRuns: 1,
|
|
},
|
|
});
|
|
const issueId = randomUUID();
|
|
const wakeupRequestId = randomUUID();
|
|
const queuedRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "succeeded",
|
|
createdAt: new Date(),
|
|
startedAt: new Date(),
|
|
finishedAt: new Date(),
|
|
contextSnapshot: {},
|
|
});
|
|
await db.insert(agentWakeupRequests).values({
|
|
id: wakeupRequestId,
|
|
companyId,
|
|
agentId,
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
reason: "manual",
|
|
payload: { issueId },
|
|
status: "queued",
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: queuedRunId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "queued",
|
|
wakeupRequestId,
|
|
contextSnapshot: { issueId },
|
|
});
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Queued issue work",
|
|
status: "in_progress",
|
|
priority: "high",
|
|
assigneeAgentId: agentId,
|
|
executionRunId: queuedRunId,
|
|
});
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId: queuedRunId })
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
payload: { issueId },
|
|
});
|
|
|
|
expect(run?.id).toBe(queuedRunId);
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const wakeups = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
runId: agentWakeupRequests.runId,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
|
|
expect(wakeups).toEqual(
|
|
expect.arrayContaining([
|
|
expect.objectContaining({
|
|
status: "coalesced",
|
|
reason: "issue_execution_same_name",
|
|
runId: queuedRunId,
|
|
}),
|
|
]),
|
|
);
|
|
});
|
|
|
|
it.each(["codex_local", "gemini_local"].flatMap(adapterType => [
|
|
{ adapterType, inputTokens: 50_000, cachedInputTokens: 550_000, rotate: true, rawInputIncludesCached: false },
|
|
{ adapterType, inputTokens: 50_000, cachedInputTokens: 449_999, rotate: false, rawInputIncludesCached: false },
|
|
{ adapterType, inputTokens: 0, cachedInputTokens: 500_000, rotate: true, rawInputIncludesCached: false },
|
|
{ adapterType, inputTokens: 400_000, cachedInputTokens: 350_000, rotate: false, rawInputIncludesCached: undefined },
|
|
{ adapterType, inputTokens: 500_000, cachedInputTokens: 350_000, rotate: true, rawInputIncludesCached: undefined },
|
|
]))("counts cached prompt input for $adapterType session rotation ($inputTokens + $cachedInputTokens; raw includes cache: $rawInputIncludesCached)", async ({ adapterType, inputTokens, cachedInputTokens, rotate, rawInputIncludesCached }) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({ heartbeatConfig: {
|
|
sessionCompaction: { enabled: true, maxSessionRuns: 0, maxRawInputTokens: 500_000, maxSessionAgeHours: 0 },
|
|
} });
|
|
const sessionId = randomUUID();
|
|
await db.update(agents).set({ adapterType }).where(eq(agents.id, agentId));
|
|
await db.insert(agentRuntimeState).values({ companyId, agentId, adapterType, sessionId });
|
|
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", sessionIdAfter: sessionId,
|
|
usageJson: { ...(rawInputIncludesCached === false ? { rawInputIncludesCached } : {}), inputTokens, cachedInputTokens, outputTokens: 10, rawInputTokens: inputTokens, rawCachedInputTokens: cachedInputTokens },
|
|
resultJson: { summary: "Continue from this completed turn" }, startedAt: new Date(), finishedAt: new Date(),
|
|
});
|
|
mockAdapterExecute.mockResolvedValueOnce({ exitCode: 0, signal: null, timedOut: false, errorMessage: null,
|
|
provider: "test", model: "test-model", costUsd: 0, usageComplete: true, usageBasis: "per_run",
|
|
usage: { inputTokens: 1, cachedInputTokens: 2, outputTokens: 3 },
|
|
});
|
|
const run = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual" });
|
|
expect(run).not.toBeNull();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
expect(countExecuteCallsForRun(run!.id)).toBe(1);
|
|
const [context] = mockAdapterExecute.mock.calls.find(([context]) => context?.runId === run!.id)!;
|
|
expect(context.runtime.sessionId).toBe(rotate ? null : sessionId);
|
|
if (rotate) {
|
|
expect(context.context.paperclipSessionRotationReason).toContain((inputTokens + (rawInputIncludesCached === false ? cachedInputTokens : 0)).toLocaleString("en-US"));
|
|
expect(context.context.paperclipSessionHandoffMarkdown).toContain("Continue from this completed turn");
|
|
} else expect(context.context.paperclipSessionRotationReason).toBeUndefined();
|
|
const [finished] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id));
|
|
expect(finished.usageJson).toMatchObject({ rawInputTokens: 1, rawCachedInputTokens: 2, rawInputIncludesCached: false });
|
|
});
|
|
|
|
it("skips wakes before queueing when per-agent daily cost cap is reached", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyCostCents: 75,
|
|
},
|
|
});
|
|
await db.insert(costEvents).values({
|
|
companyId,
|
|
agentId,
|
|
provider: "test",
|
|
biller: "test",
|
|
billingType: "metered_api",
|
|
model: "test-model",
|
|
inputTokens: 100,
|
|
outputTokens: 50,
|
|
costCents: 75,
|
|
occurredAt: new Date(),
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
});
|
|
|
|
expect(run).toBeNull();
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "heartbeat.daily_cost_limit",
|
|
});
|
|
expect(wakeup?.payload).toMatchObject({
|
|
heartbeatSkip: {
|
|
observed: 75,
|
|
limit: 75,
|
|
},
|
|
});
|
|
});
|
|
|
|
it("treats zero daily cost cap as a hard stop", async () => {
|
|
const { agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyCostCents: 0,
|
|
},
|
|
});
|
|
|
|
const run = await heartbeat.wakeup(agentId, {
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
});
|
|
|
|
expect(run).toBeNull();
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.agentId, agentId));
|
|
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
reason: "heartbeat.daily_cost_limit",
|
|
});
|
|
expect(wakeup?.payload).toMatchObject({
|
|
heartbeatSkip: {
|
|
observed: 0,
|
|
limit: 0,
|
|
},
|
|
});
|
|
});
|
|
|
|
it("skips already queued runs before adapter execution when the daily cost cap is reached", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyCostCents: 75,
|
|
},
|
|
});
|
|
const wakeupRequestId = randomUUID();
|
|
const runId = randomUUID();
|
|
await db.insert(agentWakeupRequests).values({
|
|
id: wakeupRequestId,
|
|
companyId,
|
|
agentId,
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
reason: "manual",
|
|
payload: {},
|
|
status: "queued",
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "queued",
|
|
wakeupRequestId,
|
|
contextSnapshot: {},
|
|
});
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId })
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
await db.insert(costEvents).values({
|
|
companyId,
|
|
agentId,
|
|
provider: "test",
|
|
biller: "test",
|
|
billingType: "metered_api",
|
|
model: "test-model",
|
|
inputTokens: 100,
|
|
outputTokens: 50,
|
|
costCents: 75,
|
|
occurredAt: new Date(),
|
|
});
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [run] = await db
|
|
.select({
|
|
status: heartbeatRuns.status,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
resultJson: heartbeatRuns.resultJson,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
const [wakeup] = await db
|
|
.select({
|
|
status: agentWakeupRequests.status,
|
|
error: agentWakeupRequests.error,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
|
|
expect(run).toMatchObject({
|
|
status: "cancelled",
|
|
errorCode: "heartbeat.daily_cost_limit",
|
|
});
|
|
expect(run?.resultJson).toMatchObject({
|
|
stopReason: "heartbeat.daily_cost_limit",
|
|
observed: 75,
|
|
limit: 75,
|
|
});
|
|
expect(wakeup).toMatchObject({
|
|
status: "skipped",
|
|
error: expect.stringContaining("per-day heartbeat budget cap"),
|
|
});
|
|
});
|
|
|
|
it("skips already queued issue runs at the daily run cap and releases the execution lock", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyRuns: 1,
|
|
},
|
|
});
|
|
const issueId = randomUUID();
|
|
const wakeupRequestId = randomUUID();
|
|
const queuedRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "succeeded",
|
|
createdAt: new Date(),
|
|
startedAt: new Date(),
|
|
finishedAt: new Date(),
|
|
contextSnapshot: {},
|
|
});
|
|
await db.insert(agentWakeupRequests).values({
|
|
id: wakeupRequestId,
|
|
companyId,
|
|
agentId,
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
reason: "manual",
|
|
payload: { issueId },
|
|
status: "queued",
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: queuedRunId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "queued",
|
|
wakeupRequestId,
|
|
contextSnapshot: { issueId },
|
|
});
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Queued issue work",
|
|
status: "in_progress",
|
|
priority: "high",
|
|
assigneeAgentId: agentId,
|
|
executionRunId: queuedRunId,
|
|
});
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId: queuedRunId })
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
|
|
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
|
|
|
const [run] = await db
|
|
.select({
|
|
status: heartbeatRuns.status,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, queuedRunId));
|
|
const [wakeup] = await db
|
|
.select({ status: agentWakeupRequests.status })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
const [issue] = await db
|
|
.select({ executionRunId: issues.executionRunId })
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId));
|
|
|
|
expect(run).toMatchObject({
|
|
status: "cancelled",
|
|
errorCode: "heartbeat.daily_run_limit",
|
|
});
|
|
expect(wakeup).toMatchObject({ status: "skipped" });
|
|
expect(issue?.executionRunId).toBeNull();
|
|
});
|
|
|
|
it("promotes deferred issue wakes when a queued holder is cancelled by the daily run cap", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent({
|
|
heartbeatConfig: {
|
|
maxDailyRuns: 1,
|
|
},
|
|
});
|
|
const peerAgentId = randomUUID();
|
|
const issueId = randomUUID();
|
|
const wakeupRequestId = randomUUID();
|
|
const queuedRunId = randomUUID();
|
|
const deferredWakeupId = randomUUID();
|
|
await db.insert(agents).values({
|
|
id: peerAgentId,
|
|
companyId,
|
|
name: "PeerAgent",
|
|
role: "engineer",
|
|
status: "active",
|
|
adapterType: "codex_local",
|
|
adapterConfig: {},
|
|
runtimeConfig: {
|
|
heartbeat: {
|
|
wakeOnDemand: true,
|
|
maxConcurrentRuns: 1,
|
|
},
|
|
},
|
|
permissions: {},
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "succeeded",
|
|
createdAt: new Date(),
|
|
startedAt: new Date(),
|
|
finishedAt: new Date(),
|
|
contextSnapshot: {},
|
|
});
|
|
await db.insert(agentWakeupRequests).values({
|
|
id: wakeupRequestId,
|
|
companyId,
|
|
agentId,
|
|
source: "on_demand",
|
|
triggerDetail: "manual",
|
|
reason: "manual",
|
|
payload: { issueId },
|
|
status: "queued",
|
|
});
|
|
await db.insert(heartbeatRuns).values({
|
|
id: queuedRunId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "on_demand",
|
|
triggerDetail: "manual",
|
|
status: "queued",
|
|
wakeupRequestId,
|
|
contextSnapshot: { issueId },
|
|
});
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Queued issue work",
|
|
status: "in_progress",
|
|
priority: "high",
|
|
assigneeAgentId: agentId,
|
|
executionRunId: queuedRunId,
|
|
});
|
|
await db.insert(agentWakeupRequests).values({
|
|
id: deferredWakeupId,
|
|
companyId,
|
|
agentId: peerAgentId,
|
|
source: "comment",
|
|
triggerDetail: "mention",
|
|
reason: "issue_execution_deferred",
|
|
payload: {
|
|
issueId,
|
|
_paperclipWakeContext: {
|
|
issueId,
|
|
wakeReason: "issue_mention",
|
|
},
|
|
},
|
|
status: "deferred_issue_execution",
|
|
});
|
|
await db
|
|
.update(agentWakeupRequests)
|
|
.set({ runId: queuedRunId })
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
const [deferred] = await db
|
|
.select({ status: agentWakeupRequests.status, runId: agentWakeupRequests.runId })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, deferredWakeupId));
|
|
const [promotedRun] = deferred?.runId
|
|
? await db
|
|
.select({ agentId: heartbeatRuns.agentId })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, deferred.runId))
|
|
: [];
|
|
|
|
expect(deferred?.status).not.toBe("deferred_issue_execution");
|
|
expect(promotedRun?.agentId).toBe(peerAgentId);
|
|
});
|
|
|
|
it("cancels queued max-turn continuations when another continuation owns the issue lock", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
const lockOwnerRunId = randomUUID();
|
|
|
|
await db.insert(heartbeatRuns).values({
|
|
id: lockOwnerRunId,
|
|
companyId,
|
|
agentId,
|
|
invocationSource: "automation",
|
|
triggerDetail: "system",
|
|
status: "scheduled_retry",
|
|
scheduledRetryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
|
scheduledRetryAttempt: 1,
|
|
scheduledRetryAt: new Date("2026-04-20T12:00:00.000Z"),
|
|
contextSnapshot: {
|
|
issueId,
|
|
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
|
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
|
},
|
|
});
|
|
|
|
await db.insert(issues).values({
|
|
id: issueId,
|
|
companyId,
|
|
title: "Duplicate max-turn continuation",
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: agentId,
|
|
executionRunId: lockOwnerRunId,
|
|
executionAgentNameKey: "claudecoder",
|
|
executionLockedAt: new Date("2026-04-20T11:59:00.000Z"),
|
|
});
|
|
|
|
const { runId, wakeupRequestId } = await seedQueuedRun({
|
|
companyId,
|
|
agentId,
|
|
issueId,
|
|
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
|
invocationSource: "automation",
|
|
scheduledRetryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
|
contextExtras: {
|
|
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
|
},
|
|
});
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
const [run, wakeup, issue] = await Promise.all([
|
|
db
|
|
.select({
|
|
status: heartbeatRuns.status,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
resultJson: heartbeatRuns.resultJson,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db
|
|
.select({ status: agentWakeupRequests.status, error: agentWakeupRequests.error })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db
|
|
.select({ executionRunId: issues.executionRunId })
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null),
|
|
]);
|
|
|
|
expect(run?.status).toBe("cancelled");
|
|
expect(run?.errorCode).toBe("issue_execution_lock_changed");
|
|
expect(run?.resultJson).toMatchObject({ stopReason: "issue_execution_lock_changed" });
|
|
expect(wakeup?.status).toBe("skipped");
|
|
expect(wakeup?.error).toContain("execution lock");
|
|
expect(issue?.executionRunId).toBe(lockOwnerRunId);
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
});
|
|
|
|
it.each([
|
|
["accepted", "connection_intent"],
|
|
["rejected", "connection_intent"],
|
|
["rejected", "request_confirmation"],
|
|
])("resumes a %s %s outcome and promotes the claimed review task", async (interactionStatus, interactionKind) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
const interactionId = randomUUID();
|
|
const wakeCommentId = randomUUID();
|
|
let claimedIssue: { status: string; executionRunId: string | null } | null = null;
|
|
let resumedRunId: string | undefined;
|
|
let signalAdapterStarted!: () => void;
|
|
const adapterStarted = new Promise<void>(resolve => { signalAdapterStarted = resolve; });
|
|
const execute = mockAdapterExecute.getMockImplementation()!;
|
|
mockAdapterExecute.mockImplementation(async (...args: Parameters<typeof execute>) => {
|
|
if ((args[0] as { runId: string }).runId === resumedRunId) {
|
|
signalAdapterStarted();
|
|
// A claimed continuation can finish after the old terminal poll.
|
|
if (interactionStatus === "rejected" && interactionKind === "connection_intent") {
|
|
await new Promise(resolve => setTimeout(resolve, 3100));
|
|
}
|
|
}
|
|
return execute(...args);
|
|
});
|
|
afterContinuationDispatchCheck = async ({ runId: checkedRunId, issueId: checkedIssueId }) => {
|
|
if (checkedIssueId !== issueId || checkedRunId !== resumedRunId) return;
|
|
claimedIssue = await db
|
|
.select({ status: issues.status, executionRunId: issues.executionRunId })
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null);
|
|
expect(claimedIssue).toEqual({ status: "in_progress", executionRunId: checkedRunId });
|
|
};
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Waiting for connection", status: "in_review", priority: "medium", assigneeAgentId: agentId });
|
|
await db.insert(issueComments).values({
|
|
id: wakeCommentId,
|
|
companyId,
|
|
issueId,
|
|
authorUserId: "local-board",
|
|
body: "Continue after the interaction result.",
|
|
});
|
|
if (interactionKind === "request_confirmation") {
|
|
await db.insert(issueThreadInteractions).values({
|
|
id: interactionId,
|
|
companyId,
|
|
issueId,
|
|
kind: "request_confirmation",
|
|
status: "rejected",
|
|
continuationPolicy: "wake_assignee",
|
|
payload: { version: 1, prompt: "Approve the plan?" },
|
|
result: { version: 1, outcome: "rejected", reason: "Needs more work" },
|
|
createdByAgentId: agentId,
|
|
resolvedAt: new Date(),
|
|
});
|
|
}
|
|
const { runId } = await seedQueuedRun({ companyId, agentId, issueId, wakeReason: "issue_commented",
|
|
contextExtras: { interactionId, interactionKind, interactionStatus, wakeCommentId,
|
|
originCommentIds: [wakeCommentId],
|
|
interactionResolvedAt: new Date().toISOString(), mutation: "interaction", source: `${interactionKind}.resolved`, forceFreshSession: true } });
|
|
resumedRunId = runId;
|
|
await heartbeat.resumeQueuedRuns();
|
|
// Adapter entry guarantees this exact execution is registered. Await its
|
|
// owning completion barrier before terminal assertions and fixture cleanup.
|
|
await adapterStarted;
|
|
await heartbeat.waitForRunExecutionDrain(runId);
|
|
expect((await db.select({ status: heartbeatRuns.status }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]?.status).toBe("succeeded");
|
|
expect(countExecuteCallsForRun(runId)).toBe(1);
|
|
expect(claimedIssue).toEqual({ status: "in_progress", executionRunId: runId });
|
|
});
|
|
|
|
async function seedAnsweredQuestion(input: {
|
|
companyId: string;
|
|
issueId: string;
|
|
agentId: string;
|
|
sourceRunId: string | null;
|
|
rejectedConfirmation?: boolean;
|
|
}) {
|
|
const interactionId = randomUUID();
|
|
const resolved = {
|
|
id: interactionId,
|
|
companyId: input.companyId,
|
|
issueId: input.issueId,
|
|
continuationPolicy: "wake_assignee",
|
|
createdByAgentId: input.agentId,
|
|
sourceRunId: input.sourceRunId,
|
|
resolvedByUserId: "responsible-user",
|
|
resolvedAt: new Date(),
|
|
};
|
|
await db.insert(issueThreadInteractions).values(input.rejectedConfirmation
|
|
? {
|
|
...resolved,
|
|
kind: "request_confirmation",
|
|
status: "rejected",
|
|
payload: { version: 1, prompt: "Approve the plan?" },
|
|
result: { version: 1, outcome: "rejected", reason: "Needs more work" },
|
|
}
|
|
: {
|
|
...resolved,
|
|
kind: "ask_user_questions",
|
|
status: "answered",
|
|
payload: { version: 1, questions: [{ id: "first-task-opening", prompt: "What would you like to do?", selectionMode: "single",
|
|
required: true, options: [{ id: "interview", label: "Interview me" }] }] },
|
|
result: { version: 1, answers: [{ questionId: "first-task-opening", optionIds: ["interview"] }] },
|
|
});
|
|
// Mirrors the context question-response delivery queues for an answer.
|
|
return {
|
|
interactionId,
|
|
interactionKind: input.rejectedConfirmation ? "request_confirmation" : "ask_user_questions",
|
|
interactionStatus: input.rejectedConfirmation ? "rejected" : "answered",
|
|
mutation: "interaction",
|
|
sourceRunId: input.sourceRunId,
|
|
externalChatContinuation: false,
|
|
source: "issue.interaction.respond",
|
|
};
|
|
}
|
|
|
|
it("starts the assignee when a question no run asked is answered on a todo task that has never started", async () => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
let answeredRunId: string | undefined;
|
|
let claimedIssue: { status: string; executionRunId: string | null } | null = null;
|
|
afterContinuationDispatchCheck = async ({ runId: checkedRunId, issueId: checkedIssueId }) => {
|
|
if (checkedIssueId !== issueId || checkedRunId !== answeredRunId) return;
|
|
claimedIssue = await db
|
|
.select({ status: issues.status, executionRunId: issues.executionRunId })
|
|
.from(issues)
|
|
.where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null);
|
|
};
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Onboarding first task", status: "todo", priority: "medium", assigneeAgentId: agentId });
|
|
const contextExtras = await seedAnsweredQuestion({ companyId, issueId, agentId, sourceRunId: null });
|
|
const { runId } = await seedQueuedRun({ companyId, agentId, issueId, wakeReason: "issue_commented",
|
|
invocationSource: "automation", contextExtras });
|
|
answeredRunId = runId;
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await vi.waitFor(async () => {
|
|
const current = await heartbeat.getRun(runId);
|
|
expect(countExecuteCallsForRun(runId), JSON.stringify({
|
|
status: current?.status, errorCode: current?.errorCode,
|
|
})).toBe(1);
|
|
}, { timeout: 5_000 });
|
|
await heartbeat.waitForRunExecutionDrain(runId);
|
|
|
|
expect((await heartbeat.getRun(runId))?.status).toBe("succeeded");
|
|
expect(claimedIssue).toEqual({ status: "in_progress", executionRunId: runId });
|
|
expect((await db.select({ startedAt: issues.startedAt }).from(issues).where(eq(issues.id, issueId)))[0]?.startedAt)
|
|
.toBeInstanceOf(Date);
|
|
});
|
|
|
|
it.each([
|
|
{ name: "a started task parked back in todo", startedAt: new Date("2026-01-01T00:00:00.000Z"), askedByRun: false, rejectedConfirmation: false },
|
|
{ name: "a question an agent run asked", startedAt: null, askedByRun: true, rejectedConfirmation: false },
|
|
{ name: "a rejected confirmation no run asked", startedAt: null, askedByRun: false, rejectedConfirmation: true },
|
|
])("keeps cancelling an answer continuation for $name", async ({ startedAt, askedByRun, rejectedConfirmation }) => {
|
|
const { companyId, agentId } = await seedCompanyAndAgent();
|
|
const issueId = randomUUID();
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Parked or agent-asked", status: "todo", priority: "medium",
|
|
assigneeAgentId: agentId, startedAt });
|
|
const [sourceRun] = askedByRun
|
|
? await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "assignment", status: "succeeded",
|
|
startedAt: new Date(), finishedAt: new Date(), contextSnapshot: { issueId } }).returning()
|
|
: [null];
|
|
const contextExtras = await seedAnsweredQuestion({ companyId, issueId, agentId, sourceRunId: sourceRun?.id ?? null,
|
|
rejectedConfirmation });
|
|
const { runId, wakeupRequestId } = await seedQueuedRun({ companyId, agentId, issueId, wakeReason: "issue_commented",
|
|
invocationSource: "automation", contextExtras });
|
|
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
|
|
const [run, wakeup, issue] = await Promise.all([
|
|
db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: agentWakeupRequests.status, error: agentWakeupRequests.error })
|
|
.from(agentWakeupRequests)
|
|
.where(eq(agentWakeupRequests.id, wakeupRequestId))
|
|
.then((rows) => rows[0] ?? null),
|
|
db.select({ status: issues.status }).from(issues).where(eq(issues.id, issueId))
|
|
.then((rows) => rows[0] ?? null),
|
|
]);
|
|
expect(run).toMatchObject({ status: "cancelled", errorCode: "issue_not_in_progress" });
|
|
expect(wakeup).toMatchObject({ status: "skipped", error: expect.stringContaining("no longer in_progress") });
|
|
expect(issue?.status).toBe("todo");
|
|
expect(countExecuteCallsForRun(runId)).toBe(0);
|
|
});
|
|
});
|