mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-22 16:51:28 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task recovery preserves work when execution or tool outcomes are uncertain. > - Automatic recovery can settle its record while keeping replay blocked. > - The recovery-actions API omitted those records unless they held workspace files. > - Operators and agent tools could therefore see an empty recovery history for a task that still had an execution hold. > - This pull request lists those holds and shows their matching decision in the existing stopped-run notice. > - Existing stop verification, continuation, and file repair gates remain in force. ## Linked Issues or Issue Description **What happened?** A cancelled or failed task can retain a settled recovery record with `automaticRecovery.replay: blocked`. The task remains held, but the recovery-actions API returns no action. The board notice also lacks that record's decision. **Expected behavior** List settled execution holds that still block replay. Let an authorized board user inspect the current action and run in Recovery details. Preserve the existing guarded continuation and required-file repair paths. **Steps to reproduce** 1. Stop a run whose outcome requires inspection. 2. Let automatic recovery settle its record with replay blocked. 3. Read the task's recovery actions and open its stopped-run notice. 4. Before this change, the action list is empty unless workspace recovery is required. **Paperclip version or commit** This PR includes current master through `66424362a85`. **Deployment mode** Authenticated and local deployments with persisted execution recovery holds. The original API correction and route regression are by Tao Xie (@TaoXieSZ). This update preserves those commits and adds the board inspection path, current documentation, and regression coverage. Related: #14840 describes controller lease loss. This change exposes recovery records; it does not change controller lease renewal. ## What Changed - Include settled execution reconciliation holds with replay blocked in the company-scoped recovery-actions response. - Read the matching action from the existing stopped-run notice. Show its decision under Recovery details, including when active recovery is empty. - Reject mismatched historical records in the notice. Bind history to the current action/run and refresh visible decisions every five seconds. Keep history read errors visible beside the current guidance. - Preserve existing run inspection, new-message continuation, and required-file repair controls. No new reconciliation form is added. - Update the execution documentation and both recovery capability maps. ## Verification - Recovery route suite with real embedded PostgreSQL: 80 passed, none skipped. - Notice and retry-client suites: 36 passed. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - `pnpm check:token-gates`: passed. - `node scripts/check-maps.mjs`: passed. This checks structure only. - `git diff --check`: passed. - Full `pnpm test:run` was started. It is not yet a full local pass. Initial database tests were affected by missing packaged macOS library aliases; the aliases were restored in this worktree's dependencies, and the complete recovery route suite then passed. - All CI checks passed on `09e4461e4d3a0960a4dc8bc270a2a10c628bf524`, including the full test matrix, typecheck, build and browser checks. Optional Storybook jobs were skipped. - Apex: 5/5 on the same head. Both review threads are resolved; the reviewer confirmed the existing guarded continuation design. ## Risks - The API response now includes settled records that still block replay. Consumers must distinguish a settled record from permission to continue. - Recovery history is supplemental. A failed history read keeps the current blocker and guidance visible. - The UI binds the decision to both the current action and run. It does not certify tool outcomes or grant execution authority. - No schema, dependency, telemetry, or runtime admission changes. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution, assisted with the board inspection and verification follow-up. The exact serving snapshot ID and context window size are not exposed in this session. The original API work is retained from Tao Xie's contribution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted suites; full-suite limits above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Devin Foley <devin@paperclip.ing> Co-authored-by: Paperclip <noreply@paperclip.ing>
3013 lines
118 KiB
TypeScript
3013 lines
118 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import fs from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import express from "express";
|
|
import request from "supertest";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { renderPaperclipWakePrompt } from "@paperclipai/adapter-utils/server-utils";
|
|
import {
|
|
costEvents,
|
|
agents,
|
|
agentRuntimeState,
|
|
authUsers,
|
|
agentWakeupRequests,
|
|
activityLog,
|
|
companies,
|
|
createDb,
|
|
environmentLeases,
|
|
environments,
|
|
heartbeatRuns,
|
|
issueComments,
|
|
issueInboxArchives,
|
|
issueRecoveryActions,
|
|
issueRelations,
|
|
issueThreadInteractions,
|
|
workspaceOperations,
|
|
issues,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
getEmbeddedPostgresTestSupport,
|
|
startEmbeddedPostgresTestDatabase,
|
|
} from "./helpers/embedded-postgres.js";
|
|
import { errorHandler } from "../middleware/index.js";
|
|
import { issueRoutes } from "../routes/issues.js";
|
|
import { buildPaperclipWakePayload, heartbeatService } from "../services/heartbeat.js";
|
|
import { deliverReconciledExecutions } from "../services/execution-recovery-resolution.js";
|
|
import { workspaceOperationService, isNativeWorkspaceFinalizationOperationActive } from "../services/workspace-operations.js";
|
|
import { issueRecoveryActionService } from "../services/issue-recovery-actions.js";
|
|
import { recoveryService } from "../services/recovery/service.js";
|
|
import { issueService } from "../services/issues.js";
|
|
import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js";
|
|
import { noticeMetadataReferencesRecoveryAction } from "../services/recovery/successful-run-handoff.js";
|
|
|
|
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
|
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
|
|
|
function makeRecoveryActionRow(overrides: Record<string, unknown> = {}) {
|
|
const now = new Date("2026-05-09T19:30:00.000Z");
|
|
return {
|
|
id: randomUUID(),
|
|
companyId: "company-1",
|
|
sourceIssueId: "source-1",
|
|
recoveryIssueId: null,
|
|
kind: "missing_disposition",
|
|
status: "active",
|
|
ownerType: "agent",
|
|
ownerAgentId: "agent-1",
|
|
ownerUserId: null,
|
|
previousOwnerAgentId: null,
|
|
returnOwnerAgentId: null,
|
|
cause: "successful_run_missing_issue_disposition",
|
|
fingerprint: "missing-disposition:fingerprint",
|
|
evidence: {},
|
|
nextAction: "Choose a valid issue disposition.",
|
|
wakePolicy: null,
|
|
monitorPolicy: null,
|
|
attemptCount: 1,
|
|
maxAttempts: null,
|
|
timeoutAt: null,
|
|
lastAttemptAt: now,
|
|
outcome: null,
|
|
resolutionNote: null,
|
|
resolvedAt: null,
|
|
createdAt: now,
|
|
updatedAt: now,
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
describe("issueRecoveryActionService", () => {
|
|
it("does not reactivate an action resolved between the active read and update", async () => {
|
|
const existingRow = makeRecoveryActionRow({ id: "existing-action", attemptCount: 1 });
|
|
const createdRow = makeRecoveryActionRow({ id: "new-action", attemptCount: 1 });
|
|
const selectResults = [[existingRow], []];
|
|
|
|
const makeSelectQuery = (rows: unknown[]) => ({
|
|
from() {
|
|
return this;
|
|
},
|
|
where() {
|
|
return this;
|
|
},
|
|
orderBy() {
|
|
return this;
|
|
},
|
|
limit() {
|
|
return Promise.resolve(rows);
|
|
},
|
|
});
|
|
|
|
const fakeDb = {
|
|
select: vi.fn(() => makeSelectQuery(selectResults.shift() ?? [])),
|
|
update: vi.fn(() => ({
|
|
set: vi.fn(() => ({
|
|
where: vi.fn(() => ({
|
|
returning: vi.fn(async () => []),
|
|
})),
|
|
})),
|
|
})),
|
|
insert: vi.fn(() => ({
|
|
values: vi.fn(() => ({
|
|
returning: vi.fn(async () => [createdRow]),
|
|
})),
|
|
})),
|
|
};
|
|
|
|
const result = await issueRecoveryActionService(fakeDb as never).upsertSourceScoped({
|
|
companyId: "company-1",
|
|
sourceIssueId: "source-1",
|
|
kind: "missing_disposition",
|
|
ownerType: "agent",
|
|
ownerAgentId: "agent-1",
|
|
cause: "successful_run_missing_issue_disposition",
|
|
fingerprint: "missing-disposition:fingerprint",
|
|
nextAction: "Choose a valid issue disposition.",
|
|
});
|
|
|
|
expect(result).toMatchObject({ id: "new-action", status: "active" });
|
|
expect(fakeDb.update).toHaveBeenCalledTimes(1);
|
|
expect(fakeDb.insert).toHaveBeenCalledTimes(1);
|
|
});
|
|
});
|
|
|
|
if (!embeddedPostgresSupport.supported) {
|
|
console.warn(
|
|
`Skipping embedded Postgres issue recovery action tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
|
);
|
|
}
|
|
|
|
describeEmbeddedPostgres("issue recovery actions", () => {
|
|
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
let db: ReturnType<typeof createDb>;
|
|
let operationLogRoot: string;
|
|
const priorOperationLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
|
|
|
beforeAll(async () => {
|
|
operationLogRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-recovery-operation-logs-"));
|
|
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = operationLogRoot;
|
|
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-recovery-actions-");
|
|
db = createDb(tempDb.connectionString);
|
|
}, 30_000);
|
|
|
|
afterEach(async () => {
|
|
await db.delete(workspaceOperations);
|
|
await db.delete(issueThreadInteractions);
|
|
await db.delete(issueRecoveryActions);
|
|
await db.delete(issueComments);
|
|
await db.delete(environmentLeases);
|
|
await db.delete(activityLog);
|
|
await db.delete(costEvents);
|
|
await db.delete(heartbeatRuns);
|
|
await db.delete(agentWakeupRequests);
|
|
await db.delete(environments);
|
|
await db.delete(issueInboxArchives);
|
|
await db.delete(issues);
|
|
await db.delete(agentRuntimeState);
|
|
await db.delete(agents);
|
|
await db.delete(companies);
|
|
await db.delete(authUsers);
|
|
});
|
|
|
|
afterAll(async () => {
|
|
await tempDb?.cleanup();
|
|
if (priorOperationLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
|
else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = priorOperationLogRoot;
|
|
await fs.rm(operationLogRoot, { recursive: true, force: true });
|
|
});
|
|
|
|
async function seedCompany() {
|
|
const companyId = randomUUID();
|
|
const managerId = randomUUID();
|
|
const coderId = randomUUID();
|
|
const sourceIssueId = randomUUID();
|
|
const prefix = `RA${companyId.replaceAll("-", "").slice(0, 6).toUpperCase()}`;
|
|
await db.insert(companies).values({
|
|
id: companyId,
|
|
name: "Recovery Co",
|
|
issuePrefix: prefix,
|
|
requireBoardApprovalForNewAgents: false,
|
|
});
|
|
await db.insert(agents).values([
|
|
{
|
|
id: managerId,
|
|
companyId,
|
|
name: "CTO",
|
|
role: "cto",
|
|
status: "idle",
|
|
adapterType: "codex_local",
|
|
adapterConfig: {},
|
|
runtimeConfig: {},
|
|
permissions: {},
|
|
},
|
|
{
|
|
id: coderId,
|
|
companyId,
|
|
name: "Coder",
|
|
role: "engineer",
|
|
status: "idle",
|
|
reportsTo: managerId,
|
|
adapterType: "codex_local",
|
|
adapterConfig: {},
|
|
runtimeConfig: {},
|
|
permissions: {},
|
|
},
|
|
]);
|
|
await db.insert(issues).values({
|
|
id: sourceIssueId,
|
|
companyId,
|
|
title: "Implement backend recovery",
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: coderId,
|
|
issueNumber: 1,
|
|
identifier: `${prefix}-1`,
|
|
});
|
|
const [sourceIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
return { companyId, managerId, coderId, sourceIssueId, prefix, sourceIssue: sourceIssue! };
|
|
}
|
|
|
|
async function seedHeartbeatRun(input: {
|
|
companyId: string;
|
|
agentId: string;
|
|
runId: string;
|
|
issueId?: string;
|
|
status?: string;
|
|
}) {
|
|
await db.insert(heartbeatRuns).values({
|
|
id: input.runId,
|
|
companyId: input.companyId,
|
|
agentId: input.agentId,
|
|
invocationSource: "manual",
|
|
status: input.status ?? "running",
|
|
startedAt: new Date("2026-05-13T18:00:00.000Z"),
|
|
contextSnapshot: input.issueId ? { issueId: input.issueId } : undefined,
|
|
});
|
|
}
|
|
|
|
function createApp(
|
|
actor: any = { type: "board", source: "local_implicit" },
|
|
opts: Parameters<typeof issueRoutes>[2] = {},
|
|
) {
|
|
const app = express();
|
|
app.use(express.json());
|
|
app.use((req, _res, next) => {
|
|
(req as any).actor = actor;
|
|
next();
|
|
});
|
|
app.use("/api", issueRoutes(db, {} as any, opts));
|
|
app.use(errorHandler);
|
|
return app;
|
|
}
|
|
|
|
async function seedNativeFinalizationRecovery(status: string) {
|
|
const fixture = await seedCompany();
|
|
const runId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId: fixture.companyId,
|
|
agentId: fixture.coderId,
|
|
status,
|
|
runtimeMode: "native",
|
|
nativeIssueId: fixture.sourceIssueId,
|
|
nativePhase: "finalizing",
|
|
// Native coordinator retries resume this original run, without creating
|
|
// the legacy scheduled_retry_reason projection.
|
|
scheduledRetryReason: null,
|
|
contextSnapshot: { issueId: fixture.sourceIssueId },
|
|
});
|
|
const svc = issueRecoveryActionService(db);
|
|
await svc.upsertSourceScoped({
|
|
companyId: fixture.companyId,
|
|
sourceIssueId: fixture.sourceIssueId,
|
|
kind: "active_run_watchdog",
|
|
ownerType: "agent",
|
|
ownerAgentId: fixture.coderId,
|
|
cause: "native_finalization_invalid",
|
|
fingerprint: `native:${runId}`,
|
|
nextAction: "Finish the existing run.",
|
|
wakePolicy: { kind: "resume_native_run", runId, notBefore: "2020-01-01T00:00:00Z" },
|
|
maxAttempts: 3,
|
|
});
|
|
return { ...fixture, runId, svc };
|
|
}
|
|
|
|
it.each(["queued", "running"])("projects the exact %s native run without a legacy scheduled retry", async (status) => {
|
|
const { companyId, sourceIssueId, runId, svc } = await seedNativeFinalizationRecovery(status);
|
|
const expected = { runId, status, workspaceOperationId: null };
|
|
expect((await svc.getActiveForIssue(companyId, sourceIssueId))?.nativeRunActivity).toEqual(expected);
|
|
expect((await svc.listActiveForIssues(companyId, [sourceIssueId])).get(sourceIssueId)?.nativeRunActivity).toEqual(expected);
|
|
const response = await request(createApp()).get(`/api/issues/${sourceIssueId}`).expect(200);
|
|
expect(response.body.scheduledRetry).toBeNull();
|
|
expect(response.body.activeRecoveryAction.nativeRunActivity).toEqual(expected);
|
|
});
|
|
|
|
it("projects an actual long-running export while its original heartbeat remains failed, then clears it", async () => {
|
|
const { companyId, sourceIssueId, runId, svc } = await seedNativeFinalizationRecovery("failed");
|
|
let release!: () => void;
|
|
let started!: () => void;
|
|
const held = new Promise<void>((resolve) => { release = resolve; });
|
|
const entered = new Promise<void>((resolve) => { started = resolve; });
|
|
const work = workspaceOperationService(db).createRecorder({ companyId, issueId: sourceIssueId, heartbeatRunId: runId }).recordOperation({
|
|
phase: "workspace_finalize", metadata: { owningService: "native_workspace_finalizer" },
|
|
run: async () => { started(); await held; return { status: "succeeded" }; },
|
|
});
|
|
await entered;
|
|
let operationId: string;
|
|
try {
|
|
const [operation] = await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, runId));
|
|
operationId = operation!.id;
|
|
// A slow callback remains live independently of elapsed wall-clock time.
|
|
await db.update(workspaceOperations).set({ startedAt: new Date("2000-01-01T00:00:00Z") }).where(eq(workspaceOperations.id, operationId));
|
|
const scope = { companyId, issueId: sourceIssueId, runId, operationId };
|
|
expect(isNativeWorkspaceFinalizationOperationActive(scope)).toBe(true);
|
|
expect(isNativeWorkspaceFinalizationOperationActive({ ...scope, companyId: randomUUID() })).toBe(false);
|
|
expect((await svc.getActiveForIssue(companyId, sourceIssueId))?.nativeRunActivity).toEqual({
|
|
runId, status: "running", workspaceOperationId: operationId,
|
|
});
|
|
} finally { release(); await work; }
|
|
expect((await svc.getActiveForIssue(companyId, sourceIssueId))?.nativeRunActivity).toBeNull();
|
|
// Even a stranded DB row cannot turn a joined callback back into activity.
|
|
await db.update(workspaceOperations).set({ status: "running", finishedAt: null }).where(eq(workspaceOperations.id, operationId!));
|
|
expect((await svc.getActiveForIssue(companyId, sourceIssueId))?.nativeRunActivity).toBeNull();
|
|
});
|
|
|
|
it("keeps the activity claim until a timed-out export callback actually joins", async () => {
|
|
const { companyId, sourceIssueId, runId } = await seedNativeFinalizationRecovery("failed");
|
|
let release!: () => void;
|
|
const held = new Promise<void>((resolve) => { release = resolve; });
|
|
const work = workspaceOperationService(db).createRecorder({ companyId, issueId: sourceIssueId, heartbeatRunId: runId }).recordOperation({
|
|
phase: "workspace_finalize", metadata: { owningService: "native_workspace_finalizer" }, timeoutMs: 10,
|
|
run: async () => { await held; return { status: "succeeded" }; },
|
|
});
|
|
try {
|
|
await expect(work).rejects.toThrow("exceeded");
|
|
const [operation] = await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, runId));
|
|
const scope = { companyId, issueId: sourceIssueId, runId, operationId: operation!.id };
|
|
expect(isNativeWorkspaceFinalizationOperationActive(scope)).toBe(true);
|
|
release();
|
|
await held;
|
|
await new Promise((resolve) => setImmediate(resolve));
|
|
expect(isNativeWorkspaceFinalizationOperationActive(scope)).toBe(false);
|
|
} finally { release(); }
|
|
});
|
|
|
|
it("does not treat an orphaned running export record as live after controller loss", async () => {
|
|
const { companyId, sourceIssueId, runId, svc } = await seedNativeFinalizationRecovery("failed");
|
|
await db.insert(workspaceOperations).values({
|
|
companyId, issueId: sourceIssueId, heartbeatRunId: runId,
|
|
phase: "workspace_finalize", status: "running",
|
|
metadata: { owningService: "native_workspace_finalizer" },
|
|
});
|
|
expect((await svc.getActiveForIssue(companyId, sourceIssueId))?.nativeRunActivity).toBeNull();
|
|
});
|
|
|
|
it.each(["company", "issue", "legacy", "finished"])("rejects an apparently live run with the wrong %s authority", async (mismatch) => {
|
|
const { companyId, sourceIssueId, runId, svc } = await seedNativeFinalizationRecovery("running");
|
|
const other = await seedCompany();
|
|
// Exercise service rejection of historical corruption behind the immutable binding trigger.
|
|
await db.transaction(async (tx) => {
|
|
await tx.execute(sql`set local session_replication_role = replica`);
|
|
await tx.update(heartbeatRuns).set({
|
|
...(mismatch === "company" ? { companyId: other.companyId, agentId: other.coderId } : {}),
|
|
...(mismatch === "issue" ? { nativeIssueId: other.sourceIssueId } : {}),
|
|
...(mismatch === "legacy" ? { runtimeMode: "legacy" } : {}),
|
|
...(mismatch === "finished" ? { finishedAt: new Date() } : {}),
|
|
}).where(eq(heartbeatRuns.id, runId));
|
|
});
|
|
expect((await svc.getActiveForIssue(companyId, sourceIssueId))?.nativeRunActivity).toBeNull();
|
|
});
|
|
|
|
it.each(["company", "issue", "run", "phase", "owner", "finished"])(
|
|
"does not borrow native finalization activity with a different %s binding", async (mismatch) => {
|
|
const { companyId, sourceIssueId, runId, svc } = await seedNativeFinalizationRecovery("failed");
|
|
const other = await seedNativeFinalizationRecovery("running");
|
|
await db.insert(workspaceOperations).values({
|
|
companyId: mismatch === "company" ? other.companyId : companyId,
|
|
issueId: mismatch === "issue" ? other.sourceIssueId : sourceIssueId,
|
|
heartbeatRunId: mismatch === "run" ? other.runId : runId,
|
|
phase: mismatch === "phase" ? "workspace_prepare" : "workspace_finalize",
|
|
status: "running", finishedAt: mismatch === "finished" ? new Date() : null,
|
|
metadata: { owningService: mismatch === "owner" ? "different_service" : "native_workspace_finalizer" },
|
|
});
|
|
expect((await svc.getActiveForIssue(companyId, sourceIssueId))?.nativeRunActivity).toBeNull();
|
|
expect((await svc.listActiveForIssues(companyId, [sourceIssueId, other.sourceIssueId])).get(sourceIssueId)?.nativeRunActivity).toBeNull();
|
|
},
|
|
);
|
|
|
|
it("upserts one active source-scoped action per issue and keeps company scoping explicit", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
const svc = issueRecoveryActionService(db);
|
|
|
|
const first = await svc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "stranded_assigned_issue",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "stranded_assigned_issue",
|
|
fingerprint: "recovery:fingerprint",
|
|
evidence: { latestRunId: "run-1" },
|
|
nextAction: "Restore a live execution path.",
|
|
wakePolicy: { type: "wake_owner" },
|
|
});
|
|
const second = await svc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "stranded_assigned_issue",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "stranded_assigned_issue",
|
|
fingerprint: "recovery:fingerprint",
|
|
evidence: { latestRunId: "run-2" },
|
|
nextAction: "Restore a live execution path.",
|
|
wakePolicy: { type: "wake_owner" },
|
|
});
|
|
|
|
expect(second.id).toBe(first.id);
|
|
expect(second.attemptCount).toBe(2);
|
|
expect(second.evidence).toMatchObject({ latestRunId: "run-2" });
|
|
expect(await svc.getActiveForIssue(companyId, sourceIssueId)).toMatchObject({ id: first.id });
|
|
expect(await svc.getActiveForIssue(randomUUID(), sourceIssueId)).toBeNull();
|
|
});
|
|
|
|
it("enforces maxAttempts once and removes every automatic recovery path", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
const svc = issueRecoveryActionService(db);
|
|
const base = {
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "active_run_watchdog" as const,
|
|
ownerType: "agent" as const,
|
|
ownerAgentId: managerId,
|
|
returnOwnerAgentId: managerId,
|
|
cause: "process_lost",
|
|
fingerprint: "run-process-lost",
|
|
nextAction: "Resume the same run.",
|
|
wakePolicy: { kind: "resume_native_run", runId: "run-1" },
|
|
monitorPolicy: { kind: "watch_run", runId: "run-1" },
|
|
maxAttempts: 3,
|
|
};
|
|
|
|
const first = await svc.upsertSourceScoped(base);
|
|
const second = await svc.upsertSourceScoped(base);
|
|
const exhausted = await svc.upsertSourceScoped(base);
|
|
const replay = await svc.upsertSourceScoped(base);
|
|
|
|
expect(first.attemptCount).toBe(1);
|
|
expect(second.attemptCount).toBe(2);
|
|
expect(exhausted).toMatchObject({
|
|
id: first.id,
|
|
status: "escalated",
|
|
ownerType: "board",
|
|
ownerAgentId: null,
|
|
returnOwnerAgentId: managerId,
|
|
attemptCount: 3,
|
|
maxAttempts: 3,
|
|
wakePolicy: null,
|
|
monitorPolicy: null,
|
|
outcome: "escalated",
|
|
evidence: {
|
|
recoveryBudget: {
|
|
state: "exhausted",
|
|
attemptsUsed: 3,
|
|
maxAttempts: 3,
|
|
},
|
|
},
|
|
});
|
|
expect(replay).toEqual(exhausted);
|
|
});
|
|
|
|
it("preserves legacy recovery ownership when new evidence is folded into an active action", async () => {
|
|
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
|
const svc = issueRecoveryActionService(db);
|
|
const legacy = await svc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "stranded_assigned_issue",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
previousOwnerAgentId: coderId,
|
|
returnOwnerAgentId: coderId,
|
|
cause: "process_lost",
|
|
fingerprint: "legacy-recovery",
|
|
evidence: { latestRunId: "run-1" },
|
|
nextAction: "Repair the execution path.",
|
|
wakePolicy: { type: "bounded_recovery_owner", ownerAgentId: managerId, attempt: 1, maxAttempts: 5 },
|
|
attemptCount: 1,
|
|
maxAttempts: 5,
|
|
});
|
|
|
|
const updated = await svc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "stranded_assigned_issue",
|
|
ownerType: "board",
|
|
ownerAgentId: null,
|
|
previousOwnerAgentId: coderId,
|
|
returnOwnerAgentId: coderId,
|
|
cause: "process_lost",
|
|
fingerprint: "legacy-recovery",
|
|
evidence: { latestRunId: "run-2" },
|
|
evidenceOnCreate: { routingPolicy: "board_escalation_no_takeover_v1" },
|
|
nextAction: "Board decision required.",
|
|
wakePolicy: { type: "board_escalation" },
|
|
preserveExistingOwner: true,
|
|
});
|
|
|
|
expect(updated).toMatchObject({
|
|
id: legacy.id,
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
previousOwnerAgentId: coderId,
|
|
returnOwnerAgentId: coderId,
|
|
attemptCount: 2,
|
|
maxAttempts: 5,
|
|
nextAction: "Repair the execution path.",
|
|
evidence: expect.objectContaining({ latestRunId: "run-2" }),
|
|
wakePolicy: expect.objectContaining({ type: "bounded_recovery_owner" }),
|
|
});
|
|
expect(updated.evidence).not.toHaveProperty("routingPolicy");
|
|
});
|
|
|
|
it("escalates stranded assigned work into a source action instead of a recovery issue", async () => {
|
|
const { companyId, coderId, sourceIssue } = await seedCompany();
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
const latestRun = {
|
|
id: randomUUID(),
|
|
agentId: coderId,
|
|
status: "failed",
|
|
error: "adapter failed",
|
|
errorCode: "adapter_failed",
|
|
contextSnapshot: { retryReason: "issue_continuation_needed" },
|
|
livenessState: "needs_followup",
|
|
} as const;
|
|
|
|
await Promise.all([
|
|
recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun,
|
|
comment: "Automatic continuation recovery failed.",
|
|
}),
|
|
recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun,
|
|
comment: "Automatic continuation recovery failed.",
|
|
}),
|
|
]);
|
|
|
|
const actionRows = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.sourceIssueId, sourceIssue.id));
|
|
expect(actionRows).toHaveLength(1);
|
|
expect(actionRows[0]).toMatchObject({
|
|
companyId,
|
|
kind: "stranded_assigned_issue",
|
|
status: "active",
|
|
ownerType: "board",
|
|
ownerAgentId: null,
|
|
previousOwnerAgentId: coderId,
|
|
returnOwnerAgentId: coderId,
|
|
cause: "stranded_assigned_issue",
|
|
attemptCount: 2,
|
|
evidence: expect.objectContaining({
|
|
routingPolicy: "board_escalation_no_takeover_v1",
|
|
}),
|
|
});
|
|
|
|
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssue.id));
|
|
expect(updatedIssue).toMatchObject({
|
|
status: "blocked",
|
|
});
|
|
const recoveryIssues = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stranded_issue_recovery")));
|
|
expect(recoveryIssues).toHaveLength(0);
|
|
expect(updatedIssue?.assigneeAgentId).toBe(coderId);
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
// Model the production payload: `requestedRef` keeps the operator spelling,
|
|
// and the fingerprint carries the canonical remote ref. Two equivalent
|
|
// spellings of one remote branch share `identityRef`, so they share one
|
|
// fingerprint. A different branch gets a different `identityRef`.
|
|
const makeUnresolvedBaseRefRun = (agentId: string, issueId: string) =>
|
|
(requestedRef: string, identityRef: string) =>
|
|
({
|
|
id: randomUUID(),
|
|
agentId,
|
|
status: "failed",
|
|
error: `Configured workspace base ref "${requestedRef}" did not resolve to a commit on origin after an authenticated fetch.`,
|
|
errorCode: "configuration_incomplete",
|
|
contextSnapshot: { issueId },
|
|
livenessState: "needs_followup",
|
|
resultJson: {
|
|
configurationIncomplete: {
|
|
reason: "workspace_base_ref_unresolved",
|
|
requestedRef,
|
|
attemptedRefs: [identityRef],
|
|
fingerprint: `workspace_base_ref:${identityRef}`,
|
|
},
|
|
},
|
|
}) as const;
|
|
|
|
it("bounds configuration-incomplete recovery by the unresolved base ref fingerprint", async () => {
|
|
const { coderId, sourceIssue } = await seedCompany();
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
const makeRun = makeUnresolvedBaseRefRun(coderId, sourceIssue.id);
|
|
|
|
// Two reconciliations with the same unresolved ref reuse one active action.
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun: makeRun("fix/foo", "origin/fix/foo"),
|
|
recoveryCause: "configuration_incomplete",
|
|
});
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun: makeRun("fix/foo", "origin/fix/foo"),
|
|
recoveryCause: "configuration_incomplete",
|
|
});
|
|
|
|
const actions = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.sourceIssueId, sourceIssue.id));
|
|
expect(actions).toHaveLength(1);
|
|
expect(actions[0]).toMatchObject({
|
|
cause: "configuration_incomplete",
|
|
status: "active",
|
|
attemptCount: 2,
|
|
});
|
|
// The fingerprint carries the canonical remote ref, so the same branch stays
|
|
// one action and a different branch would make a distinct fingerprint.
|
|
expect(actions[0]?.fingerprint).toBe(
|
|
`source_scoped_recovery:${sourceIssue.companyId}:${sourceIssue.id}:configuration_incomplete:workspace_base_ref:origin/fix/foo`,
|
|
);
|
|
});
|
|
|
|
it("keeps equivalent spellings of one unresolved base ref under one recovery identity", async () => {
|
|
const { coderId, sourceIssue } = await seedCompany();
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
const makeRun = makeUnresolvedBaseRefRun(coderId, sourceIssue.id);
|
|
|
|
// The operator retries the same remote branch under two spellings. Both map
|
|
// to the canonical `origin/fix/foo` identity, so recovery must not reset the
|
|
// attempt count. Each failed run still needs its own actionable notice.
|
|
const firstRun = makeRun("fix/foo", "origin/fix/foo");
|
|
const secondRun = makeRun("origin/fix/foo", "origin/fix/foo");
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun: firstRun,
|
|
recoveryCause: "configuration_incomplete",
|
|
});
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun: secondRun,
|
|
recoveryCause: "configuration_incomplete",
|
|
});
|
|
|
|
const actions = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.sourceIssueId, sourceIssue.id));
|
|
// One identity, one active action, the attempt count advances.
|
|
expect(actions).toHaveLength(1);
|
|
expect(actions[0]).toMatchObject({
|
|
cause: "configuration_incomplete",
|
|
status: "active",
|
|
attemptCount: 2,
|
|
});
|
|
expect(actions[0]?.fingerprint).toBe(
|
|
`source_scoped_recovery:${sourceIssue.companyId}:${sourceIssue.id}:configuration_incomplete:workspace_base_ref:origin/fix/foo`,
|
|
);
|
|
|
|
// Each run gets one notice bound to the same action; reconciling the
|
|
// latest run again must not duplicate its notice.
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue, previousStatus: "in_progress", latestRun: secondRun,
|
|
recoveryCause: "configuration_incomplete",
|
|
});
|
|
const notices = await db
|
|
.select({ metadata: issueComments.metadata })
|
|
.from(issueComments)
|
|
.where(
|
|
and(
|
|
eq(issueComments.issueId, sourceIssue.id),
|
|
eq(issueComments.authorType, "system"),
|
|
),
|
|
);
|
|
expect(
|
|
notices.filter((row) =>
|
|
noticeMetadataReferencesRecoveryAction(row.metadata, actions[0]!.id),
|
|
),
|
|
).toHaveLength(2);
|
|
expect(notices.map((row) => row.metadata?.sourceRunId).sort()).toEqual([firstRun.id, secondRun.id].sort());
|
|
});
|
|
|
|
it("gives a distinct recovery identity and a new operator notice when the unresolved base ref changes", async () => {
|
|
const { coderId, sourceIssue } = await seedCompany();
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
const makeRun = makeUnresolvedBaseRefRun(coderId, sourceIssue.id);
|
|
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun: makeRun("fix/foo", "origin/fix/foo"),
|
|
recoveryCause: "configuration_incomplete",
|
|
});
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun: makeRun("fix/bar", "origin/fix/bar"),
|
|
recoveryCause: "configuration_incomplete",
|
|
});
|
|
|
|
const actions = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.sourceIssueId, sourceIssue.id));
|
|
// The prior ref keeps its own record and the new ref gets a fresh identity.
|
|
expect(actions).toHaveLength(2);
|
|
const priorAction = actions.find((row) =>
|
|
row.fingerprint.endsWith("workspace_base_ref:origin/fix/foo"),
|
|
);
|
|
const newAction = actions.find((row) =>
|
|
row.fingerprint.endsWith("workspace_base_ref:origin/fix/bar"),
|
|
);
|
|
expect(priorAction?.status).toBe("cancelled");
|
|
expect(priorAction?.outcome).toBe("cancelled");
|
|
expect(newAction?.status).toBe("active");
|
|
expect(newAction?.attemptCount).toBe(1);
|
|
expect(newAction?.id).not.toBe(priorAction?.id);
|
|
|
|
// The operator gets one notice per distinct ref, each bound to its action.
|
|
const systemComments = await db
|
|
.select({ metadata: issueComments.metadata })
|
|
.from(issueComments)
|
|
.where(
|
|
and(
|
|
eq(issueComments.issueId, sourceIssue.id),
|
|
eq(issueComments.authorType, "system"),
|
|
),
|
|
);
|
|
expect(
|
|
systemComments.some((row) =>
|
|
noticeMetadataReferencesRecoveryAction(row.metadata, priorAction!.id),
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
systemComments.some((row) =>
|
|
noticeMetadataReferencesRecoveryAction(row.metadata, newAction!.id),
|
|
),
|
|
).toBe(true);
|
|
});
|
|
|
|
it.each([
|
|
["process_lost", undefined],
|
|
["adapter_failed", "successful_run_missing_state"],
|
|
["codex_output_inactivity_monitor", undefined],
|
|
["workspace_validation_failed", "workspace_validation_failed"],
|
|
["adapter_failed", undefined],
|
|
] as const)(
|
|
"routes %s recovery through the cause-keyed playbook",
|
|
async (errorCode, explicitCause) => {
|
|
const { coderId, sourceIssue } = await seedCompany();
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
const latestRun = {
|
|
id: randomUUID(),
|
|
agentId: coderId,
|
|
status: errorCode === "adapter_failed" && explicitCause === "successful_run_missing_state"
|
|
? "succeeded"
|
|
: "failed",
|
|
error: `${errorCode} failure`,
|
|
errorCode,
|
|
contextSnapshot: { retryReason: "issue_continuation_needed" },
|
|
livenessState: "needs_followup",
|
|
resultJson: errorCode === "workspace_validation_failed"
|
|
? { workspaceValidation: { reason: "missing_workspace", fingerprint: "workspace:test" } }
|
|
: null,
|
|
} as const;
|
|
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun,
|
|
...(explicitCause ? { recoveryCause: explicitCause } : {}),
|
|
});
|
|
|
|
const [action] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.sourceIssueId, sourceIssue.id));
|
|
expect(action).toMatchObject({
|
|
ownerType: "board",
|
|
ownerAgentId: null,
|
|
previousOwnerAgentId: coderId,
|
|
returnOwnerAgentId: coderId,
|
|
evidence: expect.objectContaining({
|
|
routingPolicy: "board_escalation_no_takeover_v1",
|
|
}),
|
|
wakePolicy: expect.objectContaining({
|
|
type: "board_escalation",
|
|
preservesSourceAssignee: true,
|
|
}),
|
|
});
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
},
|
|
);
|
|
|
|
it("stands down while the latest run was cancelled by a board operator", async () => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId: coderId,
|
|
invocationSource: "manual",
|
|
status: "cancelled",
|
|
error: "Cancelled by a board operator",
|
|
errorCode: "cancelled",
|
|
resultJson: { cancelledByActorType: "user", cancelledByUserId: "board-user" },
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result.operatorCancelExempted).toBe(1);
|
|
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("stands down after an operator interrupt cancellation", async () => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId: coderId,
|
|
invocationSource: "manual",
|
|
status: "cancelled",
|
|
error: "Interrupted by board comment",
|
|
errorCode: "operator_interrupted",
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result.operatorCancelExempted).toBe(1);
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("still recovers system-cancelled runs with no operator attribution", async () => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId: coderId,
|
|
invocationSource: "manual",
|
|
status: "cancelled",
|
|
error: "Cancelled because the workspace lease expired",
|
|
errorCode: "cancelled",
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result.operatorCancelExempted).toBe(0);
|
|
expect(result.escalated).toBe(1);
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
expect(await db.select().from(issueRecoveryActions)).toEqual([expect.objectContaining({
|
|
cause: "legacy_execution_requires_reconciliation", ownerType: "board", returnOwnerAgentId: coderId,
|
|
})]);
|
|
});
|
|
|
|
it("schedules a provider-quota monitor for the original assignee without creating recovery work", async () => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
const runId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId,
|
|
agentId: coderId,
|
|
invocationSource: "manual",
|
|
status: "failed",
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
error: "You've hit your usage limit for GPT-5. Try again at 12:00 AM (UTC).",
|
|
errorCode: "adapter_failed",
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result.providerQuotaMonitored).toBe(1);
|
|
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(updatedIssue).toMatchObject({
|
|
status: "in_progress",
|
|
assigneeAgentId: coderId,
|
|
monitorScheduledBy: "assignee",
|
|
monitorNotes: "Provider usage quota reached; retry the original assignee at the provider reset time.",
|
|
});
|
|
expect(updatedIssue?.monitorNextCheckAt).toBeInstanceOf(Date);
|
|
expect(updatedIssue?.executionPolicy).toMatchObject({
|
|
monitor: {
|
|
serviceName: "AI provider quota",
|
|
externalRef: runId,
|
|
maxAttempts: null,
|
|
recoveryPolicy: "wake_owner",
|
|
},
|
|
});
|
|
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(updatedRun).toMatchObject({ errorCode: "provider_quota" });
|
|
expect(updatedRun?.resultJson).toMatchObject({ errorFamily: "provider_quota" });
|
|
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
|
|
const secondResult = await recovery.reconcileStrandedAssignedIssues();
|
|
expect(secondResult).toMatchObject({ providerQuotaMonitored: 0, skipped: 1 });
|
|
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
|
});
|
|
|
|
it("schedules another provider-quota monitor after a prior quota monitor fired", async () => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
await db.update(issues).set({ monitorAttemptCount: 1 }).where(eq(issues.id, sourceIssueId));
|
|
const runId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId,
|
|
agentId: coderId,
|
|
invocationSource: "manual",
|
|
status: "failed",
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
error: "Provider quota exceeded for this model.",
|
|
errorCode: "adapter_failed",
|
|
startedAt: new Date("2026-07-15T21:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T21:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const recovery = recoveryService(db, { enqueueWakeup: vi.fn(async () => null) });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result.providerQuotaMonitored).toBe(1);
|
|
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(updatedIssue?.executionPolicy).toMatchObject({
|
|
monitor: {
|
|
maxAttempts: null,
|
|
externalRef: runId,
|
|
},
|
|
});
|
|
});
|
|
|
|
it("skips provider-quota monitor scheduling for todo issues without aborting reconciliation", async () => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
await db.update(issues).set({ status: "todo" }).where(eq(issues.id, sourceIssueId));
|
|
const runId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId,
|
|
agentId: coderId,
|
|
invocationSource: "manual",
|
|
status: "failed",
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
error: "Provider quota exceeded for this model.",
|
|
errorCode: "adapter_failed",
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result).toMatchObject({ providerQuotaMonitored: 0, skipped: 1 });
|
|
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(updatedIssue).toMatchObject({
|
|
status: "todo",
|
|
assigneeAgentId: coderId,
|
|
monitorNextCheckAt: null,
|
|
});
|
|
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(updatedRun?.errorCode).toBe("adapter_failed");
|
|
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("does not create takeover recovery when a quota monitor cannot be scheduled", async () => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, sourceIssueId));
|
|
const runId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId,
|
|
agentId: coderId,
|
|
invocationSource: "manual",
|
|
status: "failed",
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
error: "Provider quota exceeded for this model.",
|
|
errorCode: "adapter_failed",
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result).toMatchObject({ providerQuotaMonitored: 0, skipped: 1 });
|
|
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(updatedIssue).toMatchObject({
|
|
status: "in_review",
|
|
assigneeAgentId: coderId,
|
|
monitorNextCheckAt: null,
|
|
});
|
|
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(updatedRun?.errorCode).toBe("adapter_failed");
|
|
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("schedules a quota monitor for a cross-agent active review participant", async () => {
|
|
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
|
const stageId = randomUUID();
|
|
await db.update(issues).set({
|
|
status: "in_review",
|
|
assigneeAgentId: coderId,
|
|
executionPolicy: {
|
|
mode: "normal",
|
|
commentRequired: true,
|
|
stages: [{
|
|
id: stageId,
|
|
type: "review",
|
|
approvalsNeeded: 1,
|
|
participants: [{ id: randomUUID(), type: "agent", agentId: managerId, userId: null }],
|
|
}],
|
|
},
|
|
executionState: {
|
|
status: "pending",
|
|
currentStageId: stageId,
|
|
currentStageIndex: 0,
|
|
currentStageType: "review",
|
|
currentParticipant: { type: "agent", agentId: managerId, userId: null },
|
|
returnAssignee: { type: "agent", agentId: coderId, userId: null },
|
|
reviewRequest: null,
|
|
completedStageIds: [],
|
|
lastDecisionId: null,
|
|
lastDecisionOutcome: null,
|
|
},
|
|
}).where(eq(issues.id, sourceIssueId));
|
|
const [reviewIssueBeforeRecovery] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(reviewIssueBeforeRecovery).toMatchObject({
|
|
assigneeAgentId: coderId,
|
|
executionState: {
|
|
currentParticipant: { type: "agent", agentId: managerId },
|
|
returnAssignee: { type: "agent", agentId: coderId },
|
|
},
|
|
});
|
|
const runId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId,
|
|
agentId: managerId,
|
|
invocationSource: "automation",
|
|
status: "failed",
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
error: "Provider quota exceeded for this model.",
|
|
errorCode: "adapter_failed",
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result).toMatchObject({ providerQuotaMonitored: 1, reviewParticipantRequeued: 0 });
|
|
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(updatedIssue).toMatchObject({
|
|
status: "in_review",
|
|
assigneeAgentId: coderId,
|
|
monitorNextCheckAt: expect.any(Date),
|
|
monitorNotes: "Provider usage quota reached; retry the active review participant after the default recovery backoff.",
|
|
});
|
|
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(updatedRun?.errorCode).toBe("provider_quota");
|
|
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("does not restamp an in_review quota monitor when the assignee has a newer terminal run", async () => {
|
|
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
|
const stageId = randomUUID();
|
|
await db.update(issues).set({
|
|
status: "in_review",
|
|
assigneeAgentId: coderId,
|
|
executionPolicy: {
|
|
mode: "normal",
|
|
commentRequired: true,
|
|
stages: [{
|
|
id: stageId,
|
|
type: "review",
|
|
approvalsNeeded: 1,
|
|
participants: [{ id: randomUUID(), type: "agent", agentId: managerId, userId: null }],
|
|
}],
|
|
},
|
|
executionState: {
|
|
status: "pending",
|
|
currentStageId: stageId,
|
|
currentStageIndex: 0,
|
|
currentStageType: "review",
|
|
currentParticipant: { type: "agent", agentId: managerId, userId: null },
|
|
returnAssignee: { type: "agent", agentId: coderId, userId: null },
|
|
reviewRequest: null,
|
|
completedStageIds: [],
|
|
lastDecisionId: null,
|
|
lastDecisionOutcome: null,
|
|
},
|
|
}).where(eq(issues.id, sourceIssueId));
|
|
const participantRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: participantRunId,
|
|
companyId,
|
|
agentId: managerId,
|
|
invocationSource: "automation",
|
|
status: "failed",
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
error: "Provider quota exceeded for this model.",
|
|
errorCode: "adapter_failed",
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
|
|
const firstResult = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(firstResult).toMatchObject({ providerQuotaMonitored: 1 });
|
|
const [monitoredIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
const firstNextCheckAt = monitoredIssue?.monitorNextCheckAt;
|
|
expect(firstNextCheckAt).toBeInstanceOf(Date);
|
|
expect(monitoredIssue?.executionPolicy).toMatchObject({
|
|
monitor: {
|
|
serviceName: "AI provider quota",
|
|
externalRef: participantRunId,
|
|
},
|
|
});
|
|
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId: coderId,
|
|
invocationSource: "automation",
|
|
status: "failed",
|
|
error: "Stale assignee wake fired after the issue entered review.",
|
|
errorCode: "issue_assignee_changed",
|
|
startedAt: new Date("2026-07-15T20:02:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:03:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
|
|
const secondResult = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(secondResult).toMatchObject({ providerQuotaMonitored: 0, skipped: 1 });
|
|
const [unchangedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(unchangedIssue?.monitorNextCheckAt?.getTime()).toBe(firstNextCheckAt?.getTime());
|
|
expect(unchangedIssue?.executionPolicy).toMatchObject({
|
|
monitor: {
|
|
serviceName: "AI provider quota",
|
|
externalRef: participantRunId,
|
|
},
|
|
});
|
|
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("classifies review recovery from the active participant run instead of a newer assignee run", async () => {
|
|
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
|
const stageId = randomUUID();
|
|
await db.update(issues).set({
|
|
status: "in_review",
|
|
assigneeAgentId: coderId,
|
|
executionPolicy: {
|
|
mode: "normal",
|
|
commentRequired: true,
|
|
stages: [{
|
|
id: stageId,
|
|
type: "review",
|
|
approvalsNeeded: 1,
|
|
participants: [{ id: randomUUID(), type: "agent", agentId: managerId, userId: null }],
|
|
}],
|
|
},
|
|
executionState: {
|
|
status: "pending",
|
|
currentStageId: stageId,
|
|
currentStageIndex: 0,
|
|
currentStageType: "review",
|
|
currentParticipant: { type: "agent", agentId: managerId, userId: null },
|
|
returnAssignee: { type: "agent", agentId: coderId, userId: null },
|
|
reviewRequest: null,
|
|
completedStageIds: [],
|
|
lastDecisionId: null,
|
|
lastDecisionOutcome: null,
|
|
},
|
|
}).where(eq(issues.id, sourceIssueId));
|
|
const participantRunId = randomUUID();
|
|
const assigneeRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values([{
|
|
id: participantRunId,
|
|
companyId,
|
|
agentId: managerId,
|
|
invocationSource: "automation",
|
|
status: "failed",
|
|
error: "review process exited unexpectedly",
|
|
errorCode: "adapter_failed",
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
}, {
|
|
id: assigneeRunId,
|
|
companyId,
|
|
agentId: coderId,
|
|
invocationSource: "automation",
|
|
status: "failed",
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
error: "You've hit your usage limit. Try again at 11:00 PM (UTC)",
|
|
errorCode: "adapter_failed",
|
|
startedAt: new Date("2026-07-15T20:02:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:03:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
}]);
|
|
const enqueueWakeup = vi.fn(async () => ({ id: randomUUID() } as never));
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result).toMatchObject({ providerQuotaMonitored: 0, reviewParticipantRequeued: 0, escalated: 1 });
|
|
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(updatedIssue).toMatchObject({
|
|
status: "in_review",
|
|
assigneeAgentId: coderId,
|
|
monitorNextCheckAt: null,
|
|
});
|
|
const [assigneeRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, assigneeRunId));
|
|
expect(assigneeRun?.errorCode).toBe("adapter_failed");
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
expect(await db.select().from(issueRecoveryActions)).toEqual([expect.objectContaining({
|
|
cause: "legacy_execution_requires_reconciliation", ownerType: "board", returnOwnerAgentId: coderId,
|
|
evidence: expect.objectContaining({ runId: participantRunId }),
|
|
})]);
|
|
});
|
|
|
|
it("blocks a cross-agent review participant with incomplete configuration", async () => {
|
|
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
|
const stageId = randomUUID();
|
|
await db.update(issues).set({
|
|
status: "in_review",
|
|
executionPolicy: {
|
|
mode: "normal",
|
|
commentRequired: true,
|
|
stages: [{
|
|
id: stageId,
|
|
type: "review",
|
|
approvalsNeeded: 1,
|
|
participants: [{ id: randomUUID(), type: "agent", agentId: managerId, userId: null }],
|
|
}],
|
|
},
|
|
executionState: {
|
|
status: "pending",
|
|
currentStageId: stageId,
|
|
currentStageIndex: 0,
|
|
currentStageType: "review",
|
|
currentParticipant: { type: "agent", agentId: managerId, userId: null },
|
|
returnAssignee: { type: "agent", agentId: coderId, userId: null },
|
|
reviewRequest: null,
|
|
completedStageIds: [],
|
|
lastDecisionId: null,
|
|
lastDecisionOutcome: null,
|
|
},
|
|
}).where(eq(issues.id, sourceIssueId));
|
|
const runId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId,
|
|
agentId: managerId,
|
|
invocationSource: "automation",
|
|
status: "failed",
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
error: "model_not_found: requested review model does not exist",
|
|
errorCode: "adapter_failed",
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const enqueueWakeup = vi.fn(async () => ({ id: randomUUID() } as never));
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result).toMatchObject({ escalated: 1, reviewParticipantRequeued: 0 });
|
|
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(updatedIssue).toMatchObject({
|
|
status: "blocked",
|
|
assigneeAgentId: coderId,
|
|
});
|
|
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(updatedRun?.errorCode).toBe("configuration_incomplete");
|
|
const [action] = await db.select().from(issueRecoveryActions);
|
|
expect(action).toMatchObject({
|
|
sourceIssueId,
|
|
ownerType: "board",
|
|
ownerAgentId: null,
|
|
previousOwnerAgentId: coderId,
|
|
cause: "configuration_incomplete",
|
|
recoveryIssueId: null,
|
|
});
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("uses the default quota backoff when the provider does not state a reset time", async () => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: randomUUID(),
|
|
companyId,
|
|
agentId: coderId,
|
|
invocationSource: "manual",
|
|
status: "failed",
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
error: "Provider quota exceeded for this model.",
|
|
errorCode: "adapter_failed",
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const recovery = recoveryService(db, { enqueueWakeup: vi.fn(async () => null) });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result.providerQuotaMonitored).toBe(1);
|
|
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(updatedIssue).toMatchObject({
|
|
status: "in_progress",
|
|
assigneeAgentId: coderId,
|
|
monitorNotes: "Provider usage quota reached; retry the original assignee after the default recovery backoff.",
|
|
});
|
|
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
|
});
|
|
|
|
it("classifies model lookup failures as configuration incomplete without waking a recovery owner", async () => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
const runId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId,
|
|
agentId: coderId,
|
|
invocationSource: "manual",
|
|
status: "failed",
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
|
error: "model_not_found: requested model does not exist",
|
|
errorCode: "adapter_failed",
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result).toMatchObject({ escalated: 1, skipped: 0 });
|
|
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(updatedIssue?.status).toBe("blocked");
|
|
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(updatedRun?.errorCode).toBe("configuration_incomplete");
|
|
const [action] = await db.select().from(issueRecoveryActions);
|
|
expect(action).toMatchObject({
|
|
sourceIssueId,
|
|
cause: "configuration_incomplete",
|
|
recoveryIssueId: null,
|
|
});
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("does not classify stale configuration failures from a non-assignee run", async () => {
|
|
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
|
const runId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId,
|
|
companyId,
|
|
agentId: managerId,
|
|
invocationSource: "manual",
|
|
status: "failed",
|
|
error: "model_not_found: previous assignee model does not exist",
|
|
errorCode: "adapter_failed",
|
|
startedAt: new Date("2026-07-15T20:00:00.000Z"),
|
|
finishedAt: new Date("2026-07-15T20:01:00.000Z"),
|
|
contextSnapshot: { issueId: sourceIssueId },
|
|
});
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
|
|
const result = await recovery.reconcileStrandedAssignedIssues();
|
|
|
|
expect(result).toMatchObject({ escalated: 0, skipped: 1 });
|
|
const [updatedIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(updatedIssue).toMatchObject({
|
|
status: "in_progress",
|
|
assigneeAgentId: coderId,
|
|
});
|
|
const [updatedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
expect(updatedRun?.errorCode).toBe("adapter_failed");
|
|
expect(await db.select().from(issueRecoveryActions)).toHaveLength(0);
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("reuses the same source-scoped action when latest run IDs change while the cause stays the same", async () => {
|
|
const { companyId, managerId, coderId, sourceIssue } = await seedCompany();
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
const firstLatestRun = {
|
|
id: randomUUID(),
|
|
agentId: coderId,
|
|
status: "failed",
|
|
error: "adapter failed",
|
|
errorCode: "adapter_failed",
|
|
contextSnapshot: { retryReason: "issue_continuation_needed" },
|
|
livenessState: "needs_followup",
|
|
} as const;
|
|
const secondLatestRun = {
|
|
...firstLatestRun,
|
|
id: randomUUID(),
|
|
};
|
|
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun: firstLatestRun,
|
|
comment: "Automatic continuation recovery failed.",
|
|
});
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun: secondLatestRun,
|
|
comment: "Automatic continuation recovery failed.",
|
|
});
|
|
|
|
const actionRows = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.sourceIssueId, sourceIssue.id));
|
|
expect(actionRows).toHaveLength(1);
|
|
expect(actionRows[0]).toMatchObject({
|
|
companyId,
|
|
kind: "stranded_assigned_issue",
|
|
status: "active",
|
|
previousOwnerAgentId: coderId,
|
|
returnOwnerAgentId: coderId,
|
|
cause: "stranded_assigned_issue",
|
|
attemptCount: 2,
|
|
});
|
|
expect(actionRows[0]?.evidence).toMatchObject({ latestRunId: secondLatestRun.id });
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("deduplicates workspace-incoherence recovery actions by the typed workspace fingerprint", async () => {
|
|
const { companyId, coderId, sourceIssue } = await seedCompany();
|
|
const enqueueWakeup = vi.fn(async () => null);
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
const workspaceFingerprint = `workspace_incoherence:v1:sha256:${"a".repeat(64)}`;
|
|
const workspaceValidation = {
|
|
reason: "git_worktree_branch_incoherence",
|
|
fingerprint: workspaceFingerprint,
|
|
sourceIssueId: sourceIssue.id,
|
|
sourceIdentifier: sourceIssue.identifier,
|
|
executionWorkspaceId: "execution-workspace-1",
|
|
expectedBranch: "PAP-1-expected",
|
|
actualBranch: "PAP-1-publish",
|
|
cleanliness: "dirty",
|
|
provenance: {
|
|
expectedBranchExists: true,
|
|
actualBranchExists: true,
|
|
expectedHeadSha: "1111111111111111111111111111111111111111",
|
|
actualHeadSha: "2222222222222222222222222222222222222222",
|
|
sameHead: false,
|
|
},
|
|
safeRepair: {
|
|
eligible: false,
|
|
attempted: false,
|
|
succeeded: false,
|
|
reason: "worktree is not clean",
|
|
},
|
|
};
|
|
const firstLatestRun = {
|
|
id: randomUUID(),
|
|
agentId: coderId,
|
|
status: "failed",
|
|
error: "workspace branch mismatch",
|
|
errorCode: "workspace_validation_failed",
|
|
contextSnapshot: {},
|
|
livenessState: "failed",
|
|
resultJson: { workspaceValidation },
|
|
} as const;
|
|
const secondLatestRun = {
|
|
...firstLatestRun,
|
|
id: randomUUID(),
|
|
};
|
|
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun: firstLatestRun,
|
|
comment: "Workspace failed validation.",
|
|
recoveryCause: "workspace_validation_failed",
|
|
});
|
|
// Prove dedupe uses the structured recovery-action reference rather than
|
|
// depending only on the legacy body marker.
|
|
await db
|
|
.update(issueComments)
|
|
.set({ body: "Workspace recovery was already escalated." })
|
|
.where(eq(issueComments.issueId, sourceIssue.id));
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun: secondLatestRun,
|
|
comment: "Workspace failed validation.",
|
|
recoveryCause: "workspace_validation_failed",
|
|
});
|
|
|
|
const actionRows = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.sourceIssueId, sourceIssue.id));
|
|
expect(actionRows).toHaveLength(1);
|
|
expect(actionRows[0]).toMatchObject({
|
|
companyId,
|
|
kind: "workspace_validation",
|
|
cause: "workspace_validation_failed",
|
|
status: "active",
|
|
attemptCount: 2,
|
|
fingerprint: expect.stringContaining(workspaceFingerprint),
|
|
evidence: expect.objectContaining({
|
|
latestRunId: secondLatestRun.id,
|
|
latestRunErrorCode: "workspace_validation_failed",
|
|
workspaceValidation: expect.objectContaining({
|
|
reason: "git_worktree_branch_incoherence",
|
|
fingerprint: workspaceFingerprint,
|
|
sourceIssueId: sourceIssue.id,
|
|
executionWorkspaceId: "execution-workspace-1",
|
|
expectedBranch: "PAP-1-expected",
|
|
actualBranch: "PAP-1-publish",
|
|
cleanliness: "dirty",
|
|
}),
|
|
}),
|
|
nextAction: expect.stringContaining("git worktree branch incoherence"),
|
|
wakePolicy: expect.objectContaining({
|
|
type: "board_escalation",
|
|
reason: "workspace_validation_failed",
|
|
preservesSourceAssignee: true,
|
|
}),
|
|
});
|
|
|
|
const comments = await db.select().from(issueComments).where(eq(issueComments.issueId, sourceIssue.id));
|
|
const escalationComments = comments.filter((comment) =>
|
|
noticeMetadataReferencesRecoveryAction(comment.metadata, actionRows[0]!.id),
|
|
);
|
|
expect(escalationComments).toHaveLength(1);
|
|
expect(escalationComments[0]?.presentation).toMatchObject({
|
|
kind: "system_notice",
|
|
tone: "danger",
|
|
title: "Workspace validation failed",
|
|
});
|
|
expect(enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("keeps the source issue blocked when source-scoped wakeup is claimed synchronously", async () => {
|
|
const { companyId, managerId, coderId, sourceIssue } = await seedCompany();
|
|
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, managerId));
|
|
const enqueueWakeup = vi.fn(async () => {
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "in_progress" })
|
|
.where(eq(issues.id, sourceIssue.id));
|
|
return null;
|
|
});
|
|
const recovery = recoveryService(db, { enqueueWakeup });
|
|
const firstLatestRun = {
|
|
id: randomUUID(),
|
|
agentId: coderId,
|
|
status: "failed",
|
|
error: "adapter failed",
|
|
errorCode: "adapter_failed",
|
|
contextSnapshot: { retryReason: "issue_continuation_needed" },
|
|
livenessState: "needs_followup",
|
|
} as const;
|
|
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun: firstLatestRun,
|
|
comment: "Automatic continuation recovery failed.",
|
|
});
|
|
|
|
const [afterFirst] = await db.select().from(issues).where(eq(issues.id, sourceIssue.id));
|
|
expect(afterFirst?.status).toBe("blocked");
|
|
expect(afterFirst?.assigneeAgentId).toBe(coderId);
|
|
|
|
const secondLatestRun = {
|
|
...firstLatestRun,
|
|
id: randomUUID(),
|
|
};
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: sourceIssue,
|
|
previousStatus: "in_progress",
|
|
latestRun: secondLatestRun,
|
|
comment: "Automatic continuation recovery failed.",
|
|
});
|
|
|
|
const actionRows = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.sourceIssueId, sourceIssue.id));
|
|
expect(actionRows).toHaveLength(1);
|
|
expect(actionRows[0]).toMatchObject({
|
|
companyId,
|
|
kind: "stranded_assigned_issue",
|
|
status: "active",
|
|
previousOwnerAgentId: coderId,
|
|
returnOwnerAgentId: coderId,
|
|
cause: "stranded_assigned_issue",
|
|
attemptCount: 2,
|
|
});
|
|
const [afterSecond] = await db.select().from(issues).where(eq(issues.id, sourceIssue.id));
|
|
expect(afterSecond?.status).toBe("blocked");
|
|
|
|
const comments = await db.select().from(issueComments).where(eq(issueComments.issueId, sourceIssue.id));
|
|
expect(comments).toHaveLength(1);
|
|
// Dedupe for structured notices is metadata-based: the short body no longer
|
|
// carries the `Recovery action: \`id\`` marker line.
|
|
expect(comments[0]?.body).not.toContain("Recovery action:");
|
|
expect(noticeMetadataReferencesRecoveryAction(comments[0]?.metadata, actionRows[0]!.id)).toBe(true);
|
|
expect(comments[0]?.presentation).toMatchObject({ kind: "system_notice", tone: "danger" });
|
|
});
|
|
|
|
it("does not create nested recovery artifacts when issue-backed fallback work itself fails", async () => {
|
|
const { companyId, managerId, sourceIssueId, prefix } = await seedCompany();
|
|
const recoveryIssueId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: recoveryIssueId,
|
|
companyId,
|
|
title: "Recover stalled issue",
|
|
status: "in_progress",
|
|
priority: "medium",
|
|
assigneeAgentId: managerId,
|
|
parentId: sourceIssueId,
|
|
issueNumber: 2,
|
|
identifier: `${prefix}-2`,
|
|
originKind: "stranded_issue_recovery",
|
|
originId: sourceIssueId,
|
|
originFingerprint: `stranded_issue_recovery:${sourceIssueId}`,
|
|
});
|
|
const [recoveryIssue] = await db.select().from(issues).where(eq(issues.id, recoveryIssueId));
|
|
const recovery = recoveryService(db, { enqueueWakeup: vi.fn(async () => null) });
|
|
|
|
await recovery.escalateStrandedAssignedIssue({
|
|
issue: recoveryIssue!,
|
|
previousStatus: "in_progress",
|
|
latestRun: {
|
|
id: randomUUID(),
|
|
agentId: managerId,
|
|
status: "failed",
|
|
error: "adapter failed",
|
|
errorCode: "adapter_failed",
|
|
contextSnapshot: { retryReason: "issue_continuation_needed" },
|
|
livenessState: "needs_followup",
|
|
},
|
|
});
|
|
|
|
const actionRows = await db.select().from(issueRecoveryActions);
|
|
expect(actionRows).toHaveLength(0);
|
|
const recoveryIssues = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stranded_issue_recovery")));
|
|
expect(recoveryIssues).toHaveLength(1);
|
|
expect(recoveryIssues[0]?.status).toBe("blocked");
|
|
});
|
|
|
|
it("exposes active recovery actions on the issue read API", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "missing_disposition",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "successful_run_missing_issue_disposition",
|
|
fingerprint: "missing-disposition:fingerprint",
|
|
evidence: { sourceRunId: "run-1" },
|
|
nextAction: "Choose a valid issue disposition.",
|
|
wakePolicy: { type: "wake_owner" },
|
|
});
|
|
const app = createApp();
|
|
|
|
const detail = await request(app).get(`/api/issues/${sourceIssueId}`).expect(200);
|
|
expect(detail.body.activeRecoveryAction).toMatchObject({
|
|
id: action.id,
|
|
sourceIssueId,
|
|
kind: "missing_disposition",
|
|
ownerAgentId: managerId,
|
|
});
|
|
|
|
const list = await request(app).get(`/api/issues/${sourceIssueId}/recovery-actions`).expect(200);
|
|
expect(list.body.active).toMatchObject({ id: action.id });
|
|
expect(list.body.actions).toHaveLength(1);
|
|
});
|
|
|
|
it("lists a settled reconciliation hold that still blocks replay", async () => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
const runId = randomUUID();
|
|
const [action] = await db.insert(issueRecoveryActions).values({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "active_run_watchdog",
|
|
status: "resolved",
|
|
outcome: "blocked",
|
|
ownerType: "board",
|
|
returnOwnerAgentId: coderId,
|
|
cause: "legacy_execution_requires_reconciliation",
|
|
fingerprint: runId,
|
|
nextAction: "Preserve recorded work without replay.",
|
|
evidence: { runId, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } },
|
|
}).returning();
|
|
|
|
const list = await request(createApp()).get(`/api/issues/${sourceIssueId}/recovery-actions`).expect(200);
|
|
|
|
expect(list.body.active).toBeNull();
|
|
expect(list.body.actions).toEqual([
|
|
expect.objectContaining({
|
|
id: action!.id,
|
|
status: "resolved",
|
|
cause: "legacy_execution_requires_reconciliation",
|
|
evidence: { runId, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } },
|
|
}),
|
|
]);
|
|
});
|
|
|
|
it("projects recovery action metadata into the structured wake payload", async () => {
|
|
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
|
const action = await issueRecoveryActionService(db).upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "workspace_validation",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
previousOwnerAgentId: coderId,
|
|
returnOwnerAgentId: coderId,
|
|
cause: "workspace_validation_failed",
|
|
fingerprint: "workspace:wake-payload",
|
|
evidence: {
|
|
failureSummary: "Worktree branch does not match the pinned branch.",
|
|
routingFallbackReason: null,
|
|
},
|
|
nextAction: "Repair the worktree, then return the issue to the coder.",
|
|
wakePolicy: { type: "wake_owner" },
|
|
maxAttempts: 3,
|
|
});
|
|
|
|
const payload = await buildPaperclipWakePayload({
|
|
db,
|
|
companyId,
|
|
contextSnapshot: {
|
|
issueId: sourceIssueId,
|
|
wakeReason: "source_scoped_recovery_action",
|
|
recoveryActionId: action.id,
|
|
recoveryCause: action.cause,
|
|
},
|
|
});
|
|
|
|
expect(payload?.recovery).toEqual({
|
|
cause: "workspace_validation_failed",
|
|
failureSummary: "Worktree branch does not match the pinned branch.",
|
|
originalAssignee: { id: coderId, name: "Coder" },
|
|
attemptCount: 1,
|
|
maxAttempts: 3,
|
|
nextAction: "Repair the worktree, then return the issue to the coder.",
|
|
routingFallbackReason: null,
|
|
});
|
|
});
|
|
|
|
it.each(["resolved", "cancelled"])("omits %s recovery instructions when the original worker resumes", async (status) => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
const action = await issueRecoveryActionService(db).upsertSourceScoped({
|
|
companyId, sourceIssueId, kind: "stranded_assigned_issue", ownerType: "board",
|
|
returnOwnerAgentId: coderId, cause: "stranded_assigned_issue", fingerprint: "finished-recovery",
|
|
evidence: { failureSummary: "Retry budget exhausted during cleanup." },
|
|
nextAction: "Repair the runtime, then retry the original owner.", wakePolicy: null,
|
|
});
|
|
await db.update(issueRecoveryActions).set({ status, outcome: "handed_back", resolvedAt: new Date() })
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
const payload = await buildPaperclipWakePayload({ db, companyId, contextSnapshot: {
|
|
issueId: sourceIssueId, wakeReason: "issue_recovery_action_restored",
|
|
recoveryActionId: action.id, recoveryCause: action.cause,
|
|
} });
|
|
expect(payload?.issue?.id).toBe(sourceIssueId);
|
|
expect(payload?.recovery).toBeNull();
|
|
const prompt = renderPaperclipWakePrompt(payload);
|
|
expect(prompt).toContain("Implement backend recovery");
|
|
expect(prompt).not.toContain("Recovery contract:");
|
|
expect(prompt).not.toContain(action.nextAction);
|
|
expect(prompt).not.toContain("Do not produce the deliverable");
|
|
});
|
|
|
|
it("accepts new verified evidence after an automatic no-replay disposition without reopening on duplicate requests", async () => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
const runId = randomUUID();
|
|
await seedHeartbeatRun({ companyId, agentId: coderId, runId, issueId: sourceIssueId, status: "failed" });
|
|
await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, sourceIssueId));
|
|
const [action] = await db.insert(issueRecoveryActions).values({
|
|
companyId, sourceIssueId, kind: "active_run_watchdog", status: "resolved", outcome: "blocked",
|
|
ownerType: "board", returnOwnerAgentId: coderId, cause: "uncertain_external_action", fingerprint: runId,
|
|
nextAction: "Preserve recorded work without replay.",
|
|
evidence: { runId, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } },
|
|
}).returning();
|
|
const app = createApp();
|
|
const body = { actionId: action!.id, outcome: "restored", sourceIssueStatus: "todo",
|
|
executionReconciliation: { runId, providerStopped: true, actionOutcome: "not_performed",
|
|
outcomeEvidence: "Provider receipts confirm the action was never submitted; the stopped process has no remaining effects." } };
|
|
// A retry without new evidence cannot clear the hold or reopen the task.
|
|
await request(app).post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`).send({ ...body, executionReconciliation: undefined }).expect(200);
|
|
expect((await db.select().from(issues).where(eq(issues.id, sourceIssueId)))[0]!.status).toBe("blocked");
|
|
const resolved = await request(app).post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`).send(body).expect(200);
|
|
expect(resolved.body.issue.status).toBe("todo");
|
|
const [recorded] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action!.id));
|
|
expect(recorded!.evidence).not.toHaveProperty("automaticRecovery");
|
|
expect(recorded!.evidence).toMatchObject({ executionReconciliation: { runId }, continuationDelivery: "pending" });
|
|
await request(app).post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`).send(body).expect(200);
|
|
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action!.id)))[0]).toEqual(recorded);
|
|
});
|
|
|
|
async function seedReconciledDelivery() {
|
|
const fixture = await seedCompany();
|
|
const { companyId, coderId, sourceIssueId } = fixture;
|
|
const responsibleUserId = randomUUID();
|
|
await db.insert(authUsers).values({
|
|
id: responsibleUserId,
|
|
name: "Recovery operator",
|
|
email: `${responsibleUserId}@example.test`,
|
|
emailVerified: true,
|
|
createdAt: new Date(),
|
|
updatedAt: new Date(),
|
|
});
|
|
await db
|
|
.update(companies)
|
|
.set({ defaultResponsibleUserId: responsibleUserId })
|
|
.where(eq(companies.id, companyId));
|
|
await db
|
|
.update(agents)
|
|
.set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } })
|
|
.where(eq(agents.id, coderId));
|
|
const previousRunId = randomUUID();
|
|
await seedHeartbeatRun({
|
|
companyId,
|
|
agentId: coderId,
|
|
runId: previousRunId,
|
|
issueId: sourceIssueId,
|
|
status: "failed",
|
|
});
|
|
// Occupy the agent's only dispatch slot, independently of this issue. These
|
|
// tests exercise real wake admission, but cannot launch a provider process.
|
|
await seedHeartbeatRun({
|
|
companyId,
|
|
agentId: coderId,
|
|
runId: randomUUID(),
|
|
status: "running",
|
|
});
|
|
const [action] = await db
|
|
.insert(issueRecoveryActions)
|
|
.values({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "active_run_watchdog",
|
|
status: "resolved",
|
|
outcome: "restored",
|
|
ownerType: "board",
|
|
returnOwnerAgentId: coderId,
|
|
cause: "uncertain_external_action",
|
|
fingerprint: previousRunId,
|
|
nextAction: "Continue from the verified reconciliation.",
|
|
evidence: {
|
|
runId: previousRunId,
|
|
continuationDelivery: "pending",
|
|
executionReconciliation: {
|
|
runId: previousRunId,
|
|
providerStopped: true,
|
|
actionOutcome: "not_performed",
|
|
outcomeEvidence: "Verified absent provider effect.",
|
|
},
|
|
},
|
|
})
|
|
.returning();
|
|
return {
|
|
...fixture,
|
|
previousRunId,
|
|
action: action!,
|
|
heartbeat: heartbeatService(db, { runtimeEnv: {} }),
|
|
};
|
|
}
|
|
|
|
it("delivers a reconciled execution once across concurrent sweeps without a deferred duplicate", async () => {
|
|
const { action, heartbeat } = await seedReconciledDelivery();
|
|
let entered = 0;
|
|
let release!: () => void;
|
|
const bothEntered = new Promise<void>((resolve) => {
|
|
release = resolve;
|
|
});
|
|
const wake: typeof heartbeat.wakeup = async (...args) => {
|
|
entered += 1;
|
|
if (entered === 2) release();
|
|
await bothEntered;
|
|
return heartbeat.wakeup(...args);
|
|
};
|
|
await Promise.all([
|
|
deliverReconciledExecutions(db, wake),
|
|
deliverReconciledExecutions(db, wake),
|
|
]);
|
|
const wakes = await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
eq(
|
|
agentWakeupRequests.idempotencyKey,
|
|
`execution-reconciliation:${action.id}`,
|
|
),
|
|
);
|
|
expect(wakes).toHaveLength(1);
|
|
expect(wakes[0]).toMatchObject({ status: "queued" });
|
|
const [receipt] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
expect(receipt!.evidence).toMatchObject({
|
|
continuationDelivery: "delivered",
|
|
continuationRunId: wakes[0]!.runId,
|
|
});
|
|
});
|
|
|
|
it("reconciles a lost wake acknowledgement after the exact successor has already finished", async () => {
|
|
const { action, heartbeat, companyId, previousRunId } =
|
|
await seedReconciledDelivery();
|
|
let successorId: string | undefined;
|
|
await deliverReconciledExecutions(db, async (...args) => {
|
|
const run = await heartbeat.wakeup(...args);
|
|
expect(run).not.toBeNull();
|
|
successorId = run!.id;
|
|
expect(run!.retryOfRunId).toBe(previousRunId);
|
|
throw new Error("fixture lost post-commit wake acknowledgement");
|
|
});
|
|
expect(successorId).toBeDefined();
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "succeeded", finishedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, successorId!));
|
|
await deliverReconciledExecutions(db, heartbeat.wakeup);
|
|
const wakes = await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
eq(
|
|
agentWakeupRequests.idempotencyKey,
|
|
`execution-reconciliation:${action.id}`,
|
|
),
|
|
);
|
|
expect(wakes).toHaveLength(1);
|
|
const [successor] = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, companyId),
|
|
eq(heartbeatRuns.id, successorId!),
|
|
),
|
|
);
|
|
expect(successor).toMatchObject({
|
|
status: "succeeded",
|
|
retryOfRunId: previousRunId,
|
|
});
|
|
const [receipt] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
expect(receipt!.evidence).toMatchObject({
|
|
continuationDelivery: "delivered",
|
|
continuationRunId: successorId,
|
|
});
|
|
});
|
|
|
|
it.each(["owner", "status", "decision"] as const)(
|
|
"rechecks the current reconciliation %s after the sweep read",
|
|
async (changed) => {
|
|
const { action, heartbeat, sourceIssueId, managerId } =
|
|
await seedReconciledDelivery();
|
|
await deliverReconciledExecutions(db, async (...args) => {
|
|
if (changed === "owner")
|
|
await db
|
|
.update(issues)
|
|
.set({ assigneeAgentId: managerId })
|
|
.where(eq(issues.id, sourceIssueId));
|
|
if (changed === "status")
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "done" })
|
|
.where(eq(issues.id, sourceIssueId));
|
|
if (changed === "decision")
|
|
await db
|
|
.update(issueRecoveryActions)
|
|
.set({
|
|
evidence: {
|
|
...action.evidence,
|
|
executionReconciliation: {
|
|
...(action.evidence.executionReconciliation as object),
|
|
runId: randomUUID(),
|
|
},
|
|
},
|
|
})
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
return heartbeat.wakeup(...args);
|
|
});
|
|
expect(
|
|
await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
eq(
|
|
agentWakeupRequests.idempotencyKey,
|
|
`execution-reconciliation:${action.id}`,
|
|
),
|
|
),
|
|
).toHaveLength(0);
|
|
const [receipt] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
expect(receipt!.evidence.continuationDelivery).toBe("pending");
|
|
},
|
|
);
|
|
|
|
it("keeps reconciliation pending behind unrelated issue work without creating a second deferred outbox", async () => {
|
|
const { action, heartbeat, sourceIssueId, companyId, coderId } =
|
|
await seedReconciledDelivery();
|
|
const occupiedRunId = randomUUID();
|
|
await seedHeartbeatRun({
|
|
companyId,
|
|
agentId: coderId,
|
|
runId: occupiedRunId,
|
|
issueId: sourceIssueId,
|
|
status: "queued",
|
|
});
|
|
await deliverReconciledExecutions(db, heartbeat.wakeup);
|
|
await deliverReconciledExecutions(db, heartbeat.wakeup);
|
|
expect(
|
|
await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
eq(
|
|
agentWakeupRequests.idempotencyKey,
|
|
`execution-reconciliation:${action.id}`,
|
|
),
|
|
),
|
|
).toHaveLength(0);
|
|
const [occupied] = await db
|
|
.select()
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, occupiedRunId));
|
|
expect(occupied!.contextSnapshot).toEqual({ issueId: sourceIssueId });
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({ status: "succeeded", finishedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, occupiedRunId));
|
|
await deliverReconciledExecutions(db, heartbeat.wakeup);
|
|
const wakes = await db
|
|
.select()
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
eq(
|
|
agentWakeupRequests.idempotencyKey,
|
|
`execution-reconciliation:${action.id}`,
|
|
),
|
|
);
|
|
expect(wakes).toHaveLength(1);
|
|
expect(wakes[0]!.runId).not.toBe(occupiedRunId);
|
|
});
|
|
|
|
it("does not overwrite a newer reconciliation decision after a prior wake commits", async () => {
|
|
const { action, heartbeat } = await seedReconciledDelivery();
|
|
const newerEvidence = {
|
|
...action.evidence,
|
|
continuationDelivery: "invalidated",
|
|
operatorNote: "Do not continue after new evidence.",
|
|
};
|
|
await deliverReconciledExecutions(db, async (...args) => {
|
|
const run = await heartbeat.wakeup(...args);
|
|
expect(run).not.toBeNull();
|
|
await db
|
|
.update(issueRecoveryActions)
|
|
.set({ evidence: newerEvidence })
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
return run;
|
|
});
|
|
const [receipt] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
expect(receipt!.evidence).toEqual(newerEvidence);
|
|
});
|
|
|
|
it("resolves an active recovery action and removes it from active projections", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "missing_disposition",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "successful_run_missing_issue_disposition",
|
|
fingerprint: "missing-disposition:fingerprint",
|
|
evidence: { sourceRunId: "run-1" },
|
|
nextAction: "Choose a valid issue disposition.",
|
|
wakePolicy: { type: "wake_owner" },
|
|
});
|
|
const app = createApp();
|
|
|
|
const resolved = await request(app)
|
|
.post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`)
|
|
.send({
|
|
actionId: action.id,
|
|
outcome: "restored",
|
|
sourceIssueStatus: "done",
|
|
resolutionNote: "Operator confirmed the source issue is complete.",
|
|
})
|
|
.expect(200);
|
|
|
|
expect(resolved.body.issue).toMatchObject({
|
|
id: sourceIssueId,
|
|
status: "done",
|
|
activeRecoveryAction: null,
|
|
});
|
|
expect(resolved.body.recoveryAction).toMatchObject({
|
|
id: action.id,
|
|
status: "resolved",
|
|
outcome: "owner_completed",
|
|
resolutionNote: "Operator confirmed the source issue is complete.",
|
|
});
|
|
expect(resolved.body.recoveryAction.resolvedAt).toBeTruthy();
|
|
expect(await recoveryActionSvc.getActiveForIssue(companyId, sourceIssueId)).toBeNull();
|
|
expect(
|
|
await db
|
|
.select()
|
|
.from(issueInboxArchives)
|
|
.where(eq(issueInboxArchives.issueId, sourceIssueId)),
|
|
).toHaveLength(1);
|
|
|
|
const detail = await request(app).get(`/api/issues/${sourceIssueId}`).expect(200);
|
|
expect(detail.body.activeRecoveryAction).toBeNull();
|
|
|
|
const activityRows = await db
|
|
.select()
|
|
.from(activityLog)
|
|
.where(eq(activityLog.entityId, sourceIssueId));
|
|
expect(activityRows.map((row) => row.action)).toEqual(
|
|
expect.arrayContaining(["issue.updated", "issue.recovery_action_resolved"]),
|
|
);
|
|
});
|
|
|
|
it.each(["ask_user_questions", "request_confirmation"] as const)("does not retry past a new pending %s", async (kind) => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
await db.update(issues).set({ status: "blocked", assigneeAgentId: coderId }).where(eq(issues.id, sourceIssueId));
|
|
const action = await issueRecoveryActionService(db).upsertSourceScoped({
|
|
companyId, sourceIssueId, kind: "deliberate_wait_without_target", ownerType: "board",
|
|
returnOwnerAgentId: coderId, cause: "deliberate_wait_without_target", fingerprint: "disposition:pending-interaction",
|
|
nextAction: "Review the outcome.", wakePolicy: { type: "board_escalation" },
|
|
});
|
|
const [interaction] = await db.insert(issueThreadInteractions).values({
|
|
companyId, issueId: sourceIssueId, kind, status: "pending", createdByAgentId: coderId,
|
|
payload: kind === "request_confirmation"
|
|
? { version: 1, prompt: "Confirm the next step." }
|
|
: { version: 1, questions: [{ id: "next", prompt: "Which step?", selectionMode: "single", options: [{ id: "a", label: "A" }] }] },
|
|
}).returning();
|
|
const wake = vi.fn(async () => null);
|
|
const app = createApp(undefined, { recoveryActionEnqueueWakeup: wake });
|
|
const body = { actionId: action.id, outcome: "restored", sourceIssueStatus: "todo" };
|
|
const denied = await request(app).post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`).send(body).expect(409);
|
|
expect(denied.body.details?.code).toBe("disposition_recovery_interaction_pending");
|
|
expect(wake).not.toHaveBeenCalled();
|
|
expect(await db.select().from(issues).where(eq(issues.id, sourceIssueId))).toEqual([
|
|
expect.objectContaining({ status: "blocked", assigneeAgentId: coderId }),
|
|
]);
|
|
expect(await issueRecoveryActionService(db).getActiveForIssue(companyId, sourceIssueId)).toMatchObject({ id: action.id });
|
|
expect(await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction!.id))).toEqual([
|
|
expect.objectContaining({ status: "pending" }),
|
|
]);
|
|
expect(await db.select().from(activityLog).where(eq(activityLog.entityId, sourceIssueId))).toHaveLength(0);
|
|
// A settled interaction must not leave a permanent retry veto.
|
|
await db.update(issueThreadInteractions).set({ status: "resolved", resolvedAt: new Date() }).where(eq(issueThreadInteractions.id, interaction!.id));
|
|
await request(app).post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`).send(body).expect(200);
|
|
expect(wake).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("retries an exhausted disposition action once, preserving the owner and audit trail", async () => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
await db.update(issues).set({ status: "blocked", assigneeAgentId: coderId }).where(eq(issues.id, sourceIssueId));
|
|
const action = await issueRecoveryActionService(db).upsertSourceScoped({
|
|
companyId, sourceIssueId, kind: "deliberate_wait_without_target", ownerType: "board",
|
|
previousOwnerAgentId: coderId, returnOwnerAgentId: coderId, cause: "deliberate_wait_without_target",
|
|
fingerprint: "disposition:exhausted", evidence: { terminalReason: "unchanged_source_state_exhausted", sourceAttemptCount: 2, sourceMaxAttempts: 2 },
|
|
nextAction: "Review the outcome.", wakePolicy: { type: "board_escalation" },
|
|
});
|
|
const wake = vi.fn(async () => null);
|
|
const app = createApp(undefined, { recoveryActionEnqueueWakeup: wake });
|
|
const body = { actionId: action.id, outcome: "restored", sourceIssueStatus: "todo" };
|
|
const first = await request(app).post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`).send(body).expect(200);
|
|
expect(first.body.issue).toMatchObject({ status: "todo", assigneeAgentId: coderId, activeRecoveryAction: null });
|
|
expect(first.body.recoveryAction).toMatchObject({ id: action.id, status: "resolved", outcome: "handed_back" });
|
|
await request(app).post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`).send(body).expect(200);
|
|
expect(wake).toHaveBeenCalledTimes(1);
|
|
const logs = await db.select().from(activityLog).where(eq(activityLog.entityId, sourceIssueId));
|
|
expect(logs.filter(entry => entry.action === "issue.recovery_action_resolved")).toHaveLength(1);
|
|
expect((await db.select().from(issues).where(eq(issues.id, sourceIssueId)))[0]).toMatchObject({ status: "todo", assigneeAgentId: coderId });
|
|
});
|
|
|
|
it("hands restored work back to the recorded return owner and records the outcome", async () => {
|
|
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "blocked", assigneeAgentId: coderId })
|
|
.where(eq(issues.id, sourceIssueId));
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "workspace_validation",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
previousOwnerAgentId: coderId,
|
|
returnOwnerAgentId: coderId,
|
|
cause: "workspace_validation_failed",
|
|
fingerprint: "workspace:fingerprint",
|
|
evidence: { latestRunId: "run-1" },
|
|
nextAction: "Repair the workspace and hand the issue back.",
|
|
wakePolicy: { type: "wake_owner" },
|
|
});
|
|
|
|
const enqueueRecoveryActionWakeup = vi.fn(async () => null);
|
|
const resolved = await request(createApp(undefined, {
|
|
recoveryActionEnqueueWakeup: enqueueRecoveryActionWakeup,
|
|
}))
|
|
.post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`)
|
|
.send({
|
|
actionId: action.id,
|
|
outcome: "restored",
|
|
sourceIssueStatus: "todo",
|
|
resolutionNote: "Workspace repaired.",
|
|
})
|
|
.expect(200);
|
|
|
|
expect(resolved.body.issue).toMatchObject({
|
|
id: sourceIssueId,
|
|
status: "todo",
|
|
assigneeAgentId: coderId,
|
|
activeRecoveryAction: null,
|
|
});
|
|
expect(resolved.body.recoveryAction).toMatchObject({
|
|
id: action.id,
|
|
status: "resolved",
|
|
outcome: "handed_back",
|
|
});
|
|
expect(enqueueRecoveryActionWakeup).toHaveBeenCalledWith(
|
|
coderId,
|
|
expect.objectContaining({
|
|
reason: "issue_recovery_action_restored",
|
|
payload: expect.objectContaining({ issueId: sourceIssueId, recoveryActionId: action.id }),
|
|
}),
|
|
);
|
|
});
|
|
|
|
it("does not enqueue a restored wake when todo status and assignee are unchanged", async () => {
|
|
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "todo", assigneeAgentId: coderId })
|
|
.where(eq(issues.id, sourceIssueId));
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "workspace_validation",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
previousOwnerAgentId: coderId,
|
|
returnOwnerAgentId: coderId,
|
|
cause: "workspace_validation_failed",
|
|
fingerprint: "workspace:already-restored",
|
|
evidence: { latestRunId: "run-1" },
|
|
nextAction: "Confirm the workspace remains healthy.",
|
|
wakePolicy: { type: "wake_owner" },
|
|
});
|
|
|
|
const enqueueRecoveryActionWakeup = vi.fn(async () => null);
|
|
await request(createApp(undefined, {
|
|
recoveryActionEnqueueWakeup: enqueueRecoveryActionWakeup,
|
|
}))
|
|
.post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`)
|
|
.send({
|
|
actionId: action.id,
|
|
outcome: "restored",
|
|
sourceIssueStatus: "todo",
|
|
resolutionNote: "Workspace was already restored.",
|
|
})
|
|
.expect(200);
|
|
|
|
expect(enqueueRecoveryActionWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("resolves an active recovery action by returning the source issue to todo", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "blocked", assigneeAgentId: null, assigneeUserId: "board-user" })
|
|
.where(eq(issues.id, sourceIssueId));
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "issue_graph_liveness",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "issue_graph_liveness",
|
|
fingerprint: "graph-liveness:try-again",
|
|
evidence: { latestIssueStatus: "blocked" },
|
|
nextAction: "Restore a live execution path.",
|
|
wakePolicy: { type: "manual" },
|
|
});
|
|
const app = createApp();
|
|
|
|
const resolved = await request(app)
|
|
.post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`)
|
|
.send({
|
|
actionId: action.id,
|
|
outcome: "restored",
|
|
sourceIssueStatus: "todo",
|
|
resolutionNote: "Try the source issue again.",
|
|
})
|
|
.expect(200);
|
|
|
|
expect(resolved.body.issue).toMatchObject({
|
|
id: sourceIssueId,
|
|
status: "todo",
|
|
activeRecoveryAction: null,
|
|
});
|
|
expect(resolved.body.recoveryAction).toMatchObject({
|
|
id: action.id,
|
|
status: "resolved",
|
|
outcome: "restored",
|
|
resolutionNote: "Try the source issue again.",
|
|
});
|
|
expect(await recoveryActionSvc.getActiveForIssue(companyId, sourceIssueId)).toBeNull();
|
|
});
|
|
|
|
it("marks a recovery action stale when a blocked source issue is manually moved to todo", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "blocked", assigneeAgentId: null, assigneeUserId: "board-user" })
|
|
.where(eq(issues.id, sourceIssueId));
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "issue_graph_liveness",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "issue_graph_liveness",
|
|
fingerprint: "graph-liveness:manual-restore",
|
|
evidence: { latestIssueStatus: "blocked" },
|
|
nextAction: "Restore a live execution path.",
|
|
wakePolicy: { type: "manual" },
|
|
});
|
|
const app = createApp();
|
|
|
|
const patched = await request(app)
|
|
.patch(`/api/issues/${sourceIssueId}`)
|
|
.send({ status: "todo" })
|
|
.expect(200);
|
|
|
|
expect(patched.body).toMatchObject({
|
|
id: sourceIssueId,
|
|
status: "todo",
|
|
activeRecoveryAction: null,
|
|
});
|
|
|
|
const [actionRow] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
expect(actionRow).toMatchObject({
|
|
status: "cancelled",
|
|
outcome: "cancelled",
|
|
resolutionNote: "Recovery action became stale because the source issue was manually moved from blocked to todo.",
|
|
});
|
|
expect(actionRow?.resolvedAt).toBeTruthy();
|
|
expect(await recoveryActionSvc.getActiveForIssue(companyId, sourceIssueId)).toBeNull();
|
|
|
|
const detail = await request(app).get(`/api/issues/${sourceIssueId}`).expect(200);
|
|
expect(detail.body.activeRecoveryAction).toBeNull();
|
|
|
|
const activityRows = await db
|
|
.select()
|
|
.from(activityLog)
|
|
.where(eq(activityLog.entityId, sourceIssueId));
|
|
expect(activityRows.map((row) => row.action)).toEqual(
|
|
expect.arrayContaining(["issue.updated", "issue.recovery_action_resolved"]),
|
|
);
|
|
expect(activityRows.find((row) => row.action === "issue.recovery_action_resolved")?.details).toMatchObject({
|
|
source: "source_revalidation",
|
|
trigger: "issue_update",
|
|
});
|
|
});
|
|
|
|
it("folds stale recovery during read projection after the source issue reaches done", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "issue_graph_liveness",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "issue_graph_liveness",
|
|
fingerprint: "graph-liveness:done-projection",
|
|
evidence: { latestIssueStatus: "in_progress" },
|
|
nextAction: "Restore a live execution path.",
|
|
wakePolicy: { type: "manual" },
|
|
});
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, sourceIssueId));
|
|
const app = createApp();
|
|
|
|
const detail = await request(app).get(`/api/issues/${sourceIssueId}`).expect(200);
|
|
|
|
expect(detail.body).toMatchObject({
|
|
id: sourceIssueId,
|
|
status: "done",
|
|
activeRecoveryAction: null,
|
|
});
|
|
const [actionRow] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
expect(actionRow).toMatchObject({
|
|
status: "cancelled",
|
|
outcome: "cancelled",
|
|
resolutionNote: "Recovery action became stale because the source issue reached done.",
|
|
});
|
|
expect(actionRow?.resolvedAt).toBeTruthy();
|
|
|
|
const activityRows = await db
|
|
.select()
|
|
.from(activityLog)
|
|
.where(eq(activityLog.entityId, sourceIssueId));
|
|
expect(activityRows.find((row) => row.action === "issue.recovery_action_resolved")?.details).toMatchObject({
|
|
source: "source_revalidation",
|
|
trigger: "read_projection",
|
|
recoveryActionId: action.id,
|
|
});
|
|
expect(activityRows.find((row) => row.action === "issue.recovery_action_resolved")).toMatchObject({
|
|
actorType: "system", actorId: "recovery", agentId: null,
|
|
});
|
|
});
|
|
|
|
it.each(["done", "cancelled"])("keeps one workspace owner hold across reads and sweeps of a %s task", async (status) => {
|
|
const { companyId, coderId, sourceIssueId } = await seedCompany();
|
|
const runId = randomUUID();
|
|
const owner = { token: randomUUID(), hostname: "old-controller.invalid", pid: 42, processStartedAt: new Date(0).toISOString() };
|
|
await db.insert(heartbeatRuns).values({
|
|
id: runId, companyId, agentId: coderId, nativeIssueId: sourceIssueId,
|
|
runtimeMode: "native", status: "failed", finishedAt: new Date(),
|
|
runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner },
|
|
});
|
|
await db.update(issues).set({ status }).where(eq(issues.id, sourceIssueId));
|
|
const userId = "board-reader";
|
|
const app = createApp({ type: "board", source: "local_implicit", userId });
|
|
const copyback = vi.fn();
|
|
const recovery = recoveryService(db, { enqueueWakeup: vi.fn(async () => null) });
|
|
let firstAction: typeof issueRecoveryActions.$inferSelect | undefined;
|
|
for (let tick = 0; tick < 3; tick += 1) {
|
|
expect(await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId }, copyback)).toEqual({ acquired: false });
|
|
const detail = await request(app).get(`/api/issues/${sourceIssueId}`).expect(200);
|
|
expect(detail.body.activeRecoveryAction).toMatchObject({
|
|
cause: "native_workspace_finalization_owner_unverified", status: "active", attemptCount: 1,
|
|
});
|
|
await request(app).get(`/api/companies/${companyId}/issues?view=compact`).expect(200);
|
|
await request(app).get(`/api/issues/${sourceIssueId}/recovery-actions`).expect(200);
|
|
await recovery.reconcileStrandedAssignedIssues();
|
|
const actions = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, sourceIssueId));
|
|
expect(actions).toHaveLength(1);
|
|
firstAction ??= actions[0]!;
|
|
expect(actions[0]).toEqual(firstAction);
|
|
}
|
|
expect(copyback).not.toHaveBeenCalled();
|
|
expect(await issueService(db).list(companyId, { touchedByUserId: userId })).toEqual([]);
|
|
expect(await db.select().from(activityLog).where(eq(activityLog.entityId, sourceIssueId))).toHaveLength(0);
|
|
|
|
// Only clearing the exact physical owner receipt makes this hold stale.
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: {} }).where(eq(heartbeatRuns.id, runId));
|
|
const cleared = await request(app).get(`/api/issues/${sourceIssueId}`).expect(200);
|
|
expect(cleared.body.activeRecoveryAction).toBeNull();
|
|
const rows = await db.select().from(activityLog).where(eq(activityLog.entityId, sourceIssueId));
|
|
expect(rows).toEqual([expect.objectContaining({ actorType: "system", actorId: "recovery", action: "issue.recovery_action_resolved" })]);
|
|
expect(await issueService(db).list(companyId, { touchedByUserId: userId })).toEqual([]);
|
|
});
|
|
|
|
it("keeps active recovery visible when a plain comment does not create a live path", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
await db
|
|
.update(issues)
|
|
.set({ assigneeAgentId: null, assigneeUserId: "board-user" })
|
|
.where(eq(issues.id, sourceIssueId));
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "issue_graph_liveness",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "issue_graph_liveness",
|
|
fingerprint: "graph-liveness:plain-comment",
|
|
evidence: { latestIssueStatus: "in_progress" },
|
|
nextAction: "Restore a live execution path.",
|
|
wakePolicy: { type: "manual" },
|
|
});
|
|
const app = createApp();
|
|
|
|
await request(app)
|
|
.post(`/api/issues/${sourceIssueId}/comments`)
|
|
.send({ body: "I am looking at this, but not changing the disposition." })
|
|
.expect(201);
|
|
|
|
expect(await recoveryActionSvc.getActiveForIssue(companyId, sourceIssueId)).toMatchObject({
|
|
id: action.id,
|
|
status: "active",
|
|
});
|
|
const detail = await request(app).get(`/api/issues/${sourceIssueId}`).expect(200);
|
|
expect(detail.body.activeRecoveryAction).toMatchObject({ id: action.id });
|
|
});
|
|
|
|
it("folds stale recovery when a structured resume comment restores todo dispatch", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "blocked", assigneeAgentId: null, assigneeUserId: "board-user" })
|
|
.where(eq(issues.id, sourceIssueId));
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "issue_graph_liveness",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "issue_graph_liveness",
|
|
fingerprint: "graph-liveness:resume-comment",
|
|
evidence: { latestIssueStatus: "blocked" },
|
|
nextAction: "Restore a live execution path.",
|
|
wakePolicy: { type: "manual" },
|
|
});
|
|
const app = createApp();
|
|
|
|
await request(app)
|
|
.post(`/api/issues/${sourceIssueId}/comments`)
|
|
.send({ body: "Resume this now.", resume: true })
|
|
.expect(201);
|
|
|
|
const [sourceIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(sourceIssue?.status).toBe("todo");
|
|
const [actionRow] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
expect(actionRow).toMatchObject({
|
|
status: "cancelled",
|
|
outcome: "cancelled",
|
|
resolutionNote: "Recovery action became stale because the source issue was manually moved from blocked to todo.",
|
|
});
|
|
expect(await recoveryActionSvc.getActiveForIssue(companyId, sourceIssueId)).toBeNull();
|
|
|
|
const activityRows = await db
|
|
.select()
|
|
.from(activityLog)
|
|
.where(eq(activityLog.entityId, sourceIssueId));
|
|
expect(activityRows.find((row) => row.action === "issue.recovery_action_resolved")?.details).toMatchObject({
|
|
source: "source_revalidation",
|
|
trigger: "comment",
|
|
recoveryActionId: action.id,
|
|
});
|
|
});
|
|
|
|
it("rejects peer-agent source issue updates that would hide another owner's recovery action", async () => {
|
|
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "blocked", assigneeAgentId: null, assigneeUserId: "board-user" })
|
|
.where(eq(issues.id, sourceIssueId));
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "issue_graph_liveness",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "issue_graph_liveness",
|
|
fingerprint: "graph-liveness:peer-status-update",
|
|
evidence: { latestIssueStatus: "blocked" },
|
|
nextAction: "Restore a live execution path.",
|
|
wakePolicy: { type: "manual" },
|
|
});
|
|
const app = createApp({
|
|
type: "agent",
|
|
agentId: coderId,
|
|
companyId,
|
|
runId: randomUUID(),
|
|
source: "agent_jwt",
|
|
});
|
|
|
|
await request(app)
|
|
.patch(`/api/issues/${sourceIssueId}`)
|
|
.send({ status: "todo" })
|
|
.expect(403);
|
|
|
|
const [sourceIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(sourceIssue?.status).toBe("blocked");
|
|
const [actionRow] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
expect(actionRow).toMatchObject({
|
|
status: "active",
|
|
outcome: null,
|
|
resolvedAt: null,
|
|
});
|
|
});
|
|
|
|
it("rejects peer-agent recovery action resolution on a board-owned source issue", async () => {
|
|
const { companyId, managerId, coderId, sourceIssueId } = await seedCompany();
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "blocked", assigneeAgentId: null, assigneeUserId: "board-user" })
|
|
.where(eq(issues.id, sourceIssueId));
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "issue_graph_liveness",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "issue_graph_liveness",
|
|
fingerprint: "graph-liveness:peer-resolution",
|
|
evidence: { latestIssueStatus: "blocked" },
|
|
nextAction: "Restore a live execution path.",
|
|
wakePolicy: { type: "manual" },
|
|
});
|
|
const app = createApp({
|
|
type: "agent",
|
|
agentId: coderId,
|
|
companyId,
|
|
runId: randomUUID(),
|
|
source: "agent_jwt",
|
|
});
|
|
|
|
await request(app)
|
|
.post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`)
|
|
.send({
|
|
actionId: action.id,
|
|
outcome: "restored",
|
|
sourceIssueStatus: "done",
|
|
resolutionNote: "Peer agent should not be able to clear this recovery.",
|
|
})
|
|
.expect(403);
|
|
|
|
const [sourceIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(sourceIssue?.status).toBe("blocked");
|
|
const [actionRow] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
expect(actionRow).toMatchObject({
|
|
status: "active",
|
|
outcome: null,
|
|
resolvedAt: null,
|
|
});
|
|
});
|
|
|
|
it("keeps the named recovery owner from completing a board-owned source issue", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
await db
|
|
.update(issues)
|
|
.set({ status: "blocked", assigneeAgentId: null, assigneeUserId: "board-user" })
|
|
.where(eq(issues.id, sourceIssueId));
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "issue_graph_liveness",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "issue_graph_liveness",
|
|
fingerprint: "graph-liveness:owner-resolution",
|
|
evidence: { latestIssueStatus: "blocked" },
|
|
nextAction: "Restore a live execution path.",
|
|
wakePolicy: { type: "manual" },
|
|
});
|
|
const runId = randomUUID();
|
|
const app = createApp({
|
|
type: "agent",
|
|
agentId: managerId,
|
|
companyId,
|
|
runId,
|
|
source: "agent_jwt",
|
|
});
|
|
await seedHeartbeatRun({
|
|
companyId,
|
|
agentId: managerId,
|
|
runId,
|
|
issueId: sourceIssueId,
|
|
});
|
|
|
|
const resolved = await request(app)
|
|
.post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`)
|
|
.send({
|
|
actionId: action.id,
|
|
outcome: "restored",
|
|
sourceIssueStatus: "done",
|
|
resolutionNote: "Recovery owner verified the work was intentionally completed.",
|
|
})
|
|
.expect(403);
|
|
|
|
expect(resolved.body.details?.code).toBe("recovery_source_authority_required");
|
|
const [sourceAfter, actionAfter] = await Promise.all([
|
|
db.select().from(issues).where(eq(issues.id, sourceIssueId)).then((rows) => rows[0]),
|
|
db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)).then((rows) => rows[0]),
|
|
]);
|
|
expect(sourceAfter).toMatchObject({ status: "blocked", assigneeUserId: "board-user" });
|
|
expect(actionAfter).toMatchObject({ status: "active", outcome: null });
|
|
});
|
|
|
|
it("rejects blocked recovery resolution when the source issue has no first-class blockers", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "issue_graph_liveness",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "issue_graph_liveness",
|
|
fingerprint: "graph-liveness:blocked-without-blocker",
|
|
evidence: { latestIssueStatus: "in_progress" },
|
|
nextAction: "Choose a disposition with a live continuation path.",
|
|
wakePolicy: { type: "manual" },
|
|
});
|
|
const app = createApp();
|
|
|
|
const rejected = await request(app)
|
|
.post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`)
|
|
.send({
|
|
actionId: action.id,
|
|
outcome: "blocked",
|
|
sourceIssueStatus: "blocked",
|
|
})
|
|
.expect(422);
|
|
|
|
expect(rejected.body.error).toContain("requires an unresolved first-class blocker");
|
|
|
|
const [sourceIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(sourceIssue?.status).toBe("in_progress");
|
|
|
|
const [actionRow] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
expect(actionRow).toMatchObject({
|
|
status: "active",
|
|
outcome: null,
|
|
resolvedAt: null,
|
|
});
|
|
});
|
|
|
|
it("allows blocked recovery resolution when the source issue has an unresolved first-class blocker", async () => {
|
|
const { companyId, managerId, sourceIssueId, prefix } = await seedCompany();
|
|
const blockerIssueId = randomUUID();
|
|
await db.insert(issues).values({
|
|
id: blockerIssueId,
|
|
companyId,
|
|
title: "Unblock recovery disposition",
|
|
status: "todo",
|
|
priority: "medium",
|
|
assigneeAgentId: managerId,
|
|
issueNumber: 2,
|
|
identifier: `${prefix}-2`,
|
|
});
|
|
await db.insert(issueRelations).values({
|
|
companyId,
|
|
issueId: blockerIssueId,
|
|
relatedIssueId: sourceIssueId,
|
|
type: "blocks",
|
|
});
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "issue_graph_liveness",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "issue_graph_liveness",
|
|
fingerprint: "graph-liveness:blocked-with-blocker",
|
|
evidence: { latestIssueStatus: "in_progress" },
|
|
nextAction: "Wait for the blocker before continuing.",
|
|
wakePolicy: { type: "manual" },
|
|
});
|
|
const app = createApp();
|
|
|
|
const resolved = await request(app)
|
|
.post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`)
|
|
.send({
|
|
actionId: action.id,
|
|
outcome: "blocked",
|
|
sourceIssueStatus: "blocked",
|
|
resolutionNote: "The source issue is explicitly blocked by a follow-up.",
|
|
})
|
|
.expect(200);
|
|
|
|
expect(resolved.body.issue).toMatchObject({
|
|
id: sourceIssueId,
|
|
status: "blocked",
|
|
activeRecoveryAction: null,
|
|
});
|
|
expect(resolved.body.recoveryAction).toMatchObject({
|
|
id: action.id,
|
|
status: "resolved",
|
|
outcome: "blocked",
|
|
resolutionNote: "The source issue is explicitly blocked by a follow-up.",
|
|
});
|
|
expect(await recoveryActionSvc.getActiveForIssue(companyId, sourceIssueId)).toBeNull();
|
|
});
|
|
|
|
it("rejects false-positive recovery resolution without an explicit source issue status", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "issue_graph_liveness",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "issue_graph_liveness",
|
|
fingerprint: "graph-liveness:fingerprint",
|
|
evidence: { latestIssueStatus: "in_progress" },
|
|
nextAction: "Confirm whether the issue is actually stranded.",
|
|
wakePolicy: { type: "manual" },
|
|
});
|
|
const app = createApp();
|
|
|
|
await request(app)
|
|
.post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`)
|
|
.send({
|
|
actionId: action.id,
|
|
outcome: "false_positive",
|
|
resolutionNote: "The source issue still has a live execution path.",
|
|
})
|
|
.expect(400);
|
|
|
|
const [sourceIssue] = await db.select().from(issues).where(eq(issues.id, sourceIssueId));
|
|
expect(sourceIssue?.status).toBe("in_progress");
|
|
|
|
const [actionRow] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
expect(actionRow).toMatchObject({
|
|
status: "active",
|
|
outcome: null,
|
|
resolutionNote: null,
|
|
});
|
|
});
|
|
|
|
it("allows false-positive recovery resolution to restore a blocked source issue in the same request", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, sourceIssueId));
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "issue_graph_liveness",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "issue_graph_liveness",
|
|
fingerprint: "graph-liveness:false-positive-unblock",
|
|
evidence: { latestIssueStatus: "blocked" },
|
|
nextAction: "Confirm whether the issue is actually stranded.",
|
|
wakePolicy: { type: "manual" },
|
|
});
|
|
const app = createApp();
|
|
|
|
const resolved = await request(app)
|
|
.post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`)
|
|
.send({
|
|
actionId: action.id,
|
|
outcome: "false_positive",
|
|
sourceIssueStatus: "in_review",
|
|
resolutionNote: "Recovery signal was stale; return to review.",
|
|
})
|
|
.expect(200);
|
|
|
|
expect(resolved.body.issue).toMatchObject({
|
|
id: sourceIssueId,
|
|
status: "in_review",
|
|
activeRecoveryAction: null,
|
|
});
|
|
expect(resolved.body.recoveryAction).toMatchObject({
|
|
id: action.id,
|
|
status: "resolved",
|
|
outcome: "false_positive",
|
|
resolutionNote: "Recovery signal was stale; return to review.",
|
|
});
|
|
});
|
|
|
|
it("enforces company scope when resolving recovery actions", async () => {
|
|
const { companyId, managerId, sourceIssueId } = await seedCompany();
|
|
const recoveryActionSvc = issueRecoveryActionService(db);
|
|
const action = await recoveryActionSvc.upsertSourceScoped({
|
|
companyId,
|
|
sourceIssueId,
|
|
kind: "missing_disposition",
|
|
ownerType: "agent",
|
|
ownerAgentId: managerId,
|
|
cause: "successful_run_missing_issue_disposition",
|
|
fingerprint: "missing-disposition:fingerprint",
|
|
evidence: { sourceRunId: "run-1" },
|
|
nextAction: "Choose a valid issue disposition.",
|
|
wakePolicy: { type: "wake_owner" },
|
|
});
|
|
const app = createApp({
|
|
type: "agent",
|
|
agentId: randomUUID(),
|
|
companyId: randomUUID(),
|
|
runId: randomUUID(),
|
|
source: "agent_jwt",
|
|
});
|
|
|
|
await request(app)
|
|
.post(`/api/issues/${sourceIssueId}/recovery-actions/resolve`)
|
|
.send({
|
|
actionId: action.id,
|
|
outcome: "restored",
|
|
sourceIssueStatus: "done",
|
|
})
|
|
.expect(404);
|
|
|
|
const [actionRow] = await db
|
|
.select()
|
|
.from(issueRecoveryActions)
|
|
.where(eq(issueRecoveryActions.id, action.id));
|
|
expect(actionRow?.status).toBe("active");
|
|
});
|
|
});
|