mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run is cut off by a restart or a release swap, the fresh-continuation fallback continues the task in a new run. > - The continuation goes through normal admission, or through a person's saved follow-up when one is waiting. > - In release-swap waves, three gaps left tasks stopped with no run: admission refused an isolated continuation of a task with no project, a person's saved follow-up behind the replaced run was never promoted, and the fallback relied on a saved follow-up that the cut-off run had already read. > - This pull request closes those three gaps. > - The benefit is that a twice-interrupted task, and a task with a waiting follow-up, continue by themselves. ## Linked Issues or Issue Description Refs #21135, #21222 **What happened?** 1. A task with no project was interrupted twice. The second continuation used a fresh isolated workspace, which clears the task's workspace binding. The admission preflight still required that binding for an isolated task with no project, so it refused the continuation as `workspace_worktree_requires_project` about every 15 s. Each refusal re-blocked the task and added a comment (329 to 428 skipped wakes per wave). The runtime itself supports this case: it prepares a new workspace (`create_new`). 2. The fallback continued a task with delivery `saved_input` (a person's deferred follow-up is the next turn). The stranded-queue pass refused to promote that follow-up, because the replaced native run's finalization was `terminal_failure` (restart recovery ended it), not `committed`. The release path would also have re-blocked the task with a reconciliation incident. The follow-up never ran. 3. The fallback chose `saved_input` because a deferred user wake existed, but the cut-off run had already taken that message into its turn. Promotion settled the message as delivered, and no run followed. The task showed "working". **Expected behavior** An interrupted task continues by itself in each case, with no person action and no repeated comments. **Steps to reproduce** 1. Run a release-swap wave with projectless tasks in isolated mode and person follow-ups during runs. 2. Interrupt the same task twice, or interrupt a run that has a deferred follow-up behind it. 3. See the task stay blocked or "working" with no run. ## What Changed - Admission (`server/src/services/execution-workspace-policy.ts`, `server/src/services/heartbeat/queue.ts`, `server/src/services/fresh-continuation-fallback.ts`): `isUnrunnableWorktreeCombo` accepts `freshIsolatedContinuation`. The admission preflight sets it from the new `isFreshIsolatedContinuationWake`, which returns at once (no query) unless the wake context carries `freshContinuationIsolation`. This does not touch the dispatch ranking. - Follow-up promotion (`server/src/modules/wake-queue/adapters/postgres.ts`): the stranded-queue pass and the release path skip the native finalization gate and the reconciliation incident for a run that the fallback superseded (`agentRunSupersededByFreshContinuation`). - Saved-input choice (`server/src/services/fresh-continuation-fallback.ts`): a deferred user wake counts as saved input only when it has no comment ids, or still has a message that the agent has not received (`undeliveredLegacyUserCommentIds`, called, not changed). Otherwise the fallback records its own continuation. - Review round 1: (a) whenever the old writer is not proven stopped (`fresh_isolated_workspace`), the fallback records its own isolated continuation first, also when the task had a binding; saved input follows it on the new workspace. (b) Release skips the manual-admission hold and the pending-cleanup check for a run the fallback replaced with an isolated continuation (`fresh_isolated_workspace` or `remote_epoch`, via `freshContinuationIsolatesRun`); other replacements still wait. - Review round 2 (the isolated continuation must run before the waiting follow-up): while the fallback's `fresh_isolated_workspace` continuation is still pending delivery, release leaves the person's follow-up deferred. Otherwise the follow-up would become the task's next run without the isolation marker, and delivery would count it as the continuation and skip the isolated run. The continuation's own release promotes the follow-up behind it. `freshContinuationIsolatesRun` now also reports `continuationPending`. ## Verification Red-to-green tests (each failed before its fix and passes after it): - `server/src/__tests__/execution-workspace-policy.test.ts`: admission keeps a fresh isolated continuation of a projectless task runnable. - `server/src/modules/wake-queue/adapters/postgres.test.ts` "restart-ended native run": saved input behind a run the fallback replaced is promoted. - `server/src/services/fresh-continuation-fallback.test.ts` "relies on saved input only while it still holds an undelivered message" (taken: true gives `recorded`, taken: false gives `saved_input`). - `postgres.test.ts` "promotes saved input behind an exhausted restart hold ... (released, continuation pending)": the follow-up stays deferred while the isolated continuation is pending, then is promoted when the continuation's run is released. It was red before the fix (`expected 'promoted' to be 'released'`). - Local, round 2: `cd server && npx vitest run src/modules/wake-queue/adapters/postgres.test.ts src/services/fresh-continuation-fallback.test.ts`: 2 files, 177 tests passed. Local command, on this branch based on current master (includes #21222 and #21234): - `cd server && npx vitest run src/services/fresh-continuation-fallback.test.ts src/__tests__/execution-workspace-policy.test.ts src/modules/wake-queue/adapters/postgres.test.ts`: 3 files, 221 tests passed (re-run after rebasing on #21234). Local release-swap cells: - L1 (6 tasks, 3 swaps): 5 fresh isolated continuations delivered, including a second one on the same task; 0 `workspace_worktree_requires_project` rows (before: 329 to 428 skipped wakes per wave). - R2 (6 tasks, 3 swaps): a task interrupted three times had its second and third continuations delivered. The same cell found gap 3 (a follow-up already read by the cut-off run), which the last commit fixes. - Hour-long release-swap wave (w-20261112T233630-s7201) on the branch that holds these fixes and the related recovery fixes: 11/11 tasks finished hands-off (master: 9/11). The lost-continuation and follow-up-promotion fixes were visibly working in it. Review-round tests: "moves the task off the old writer's workspace and runs the isolated continuation before saved input" (changed expectation: `recorded` instead of `saved_input`; red on the previous head); "promotes saved input behind an exhausted restart hold once a %s continuation replaced the run" (isolated: promoted, red before; `writer_ended`: held). 3 files, 224 tests passed. Conflict note: the last commit was resolved onto current master. It no longer imports `taskHistoryReadsForRun`, which belongs to a separate opening-note change. ## Risks - An isolated projectless continuation now passes admission. It runs in a new workspace that the run prepares, so it never shares a workspace with the cut-off run. - A superseded run's follow-up is promoted without its native finalization committed. The superseded run is fenced (its writes are refused), so there is still one writer. - No migration. No schema change. The normal admission path adds no query. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
964 lines
71 KiB
TypeScript
964 lines
71 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
activityLog, agentWakeupRequests, agents, approvals, authUsers, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRuns, issueApprovals, issueRecoveryActions, issueThreadInteractions,
|
|
issueComments, issues, nativeRunFinalizations, toolInvocations,
|
|
} from "@paperclipai/db";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
|
import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./execution-recovery-resolution.js";
|
|
import { getExecutionBlocker } from "./execution-blocker.js";
|
|
import {
|
|
FRESH_CONTINUATIONS_PER_HOUR, FRESH_CONTINUATION_SCHEMA, PERSON_OWNED_FAILURE_CODES, continueHeldExecutions, ensureFreshContinuation,
|
|
resetBacklogContinuationPacingForTests,
|
|
freshContinuationNote, heldExecutionsAwaitingFreshContinuation, nextFreshContinuationCheckAt, recordFreshContinuationCheck, FRESH_CONTINUATION_PENDING_BOUND_MS, FRESH_CONTINUATION_PROGRESS_BOUND_MS, FRESH_CONTINUATION_SAVED_WORK_BOUND_MS,
|
|
} from "./fresh-continuation-fallback.js";
|
|
import { heartbeatService } from "./heartbeat.js";
|
|
import { buildExecutionContinuation } from "./execution-continuation.js";
|
|
import { currentProcessSpaceHash } from "./controller-process-identity.js";
|
|
import { executionControlSchedule } from "./execution-control-work.js";
|
|
import { readFreshWorkspaceReadiness } from "./fresh-workspace-continuation.js";
|
|
import { taskInterruptionPresentation } from "./task-interruption-presentation.js";
|
|
import { agentRunWritesRevoked, assertAgentRunWriteAllowed } from "../agent-run-cancellation.js";
|
|
import { loadAgentRunAuthentication } from "../middleware/agent-run-authentication.js";
|
|
import { SERVER_RESTART_RETRY_MAX_ATTEMPTS, SERVER_RESTART_RETRY_WINDOW_MS } from "./execution-recovery-attempt.js";
|
|
import { FRESH_CONTINUATION_WINDOW_MS } from "./fresh-continuation-fallback.js";
|
|
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
const suite = support.supported ? describe : describe.skip;
|
|
|
|
// Copy shown to people never asks them to inspect or repair a sandbox, workspace or run.
|
|
const adminText = /sandbox|inspect|repair|administrator|reconcile/i;
|
|
|
|
type Seed = {
|
|
runtime: "native" | "legacy"; status?: "failed" | "interrupted" | "timed_out" | "cancelled";
|
|
errorCode: string; error?: string;
|
|
/** How today's code leaves the run: an active hold (before disposition) or a settled-but-blocked hold (older releases). */
|
|
hold?: "active" | "settled_blocked" | "none";
|
|
coordinatorState?: string;
|
|
};
|
|
|
|
suite("fresh-continuation fallback", () => {
|
|
let db!: ReturnType<typeof createDb>;
|
|
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
let companyId = "", agentId = "", userId = "";
|
|
|
|
beforeAll(async () => {
|
|
temporary = await startEmbeddedPostgresTestDatabase("paperclip-fresh-continuation-");
|
|
db = createDb(temporary.connectionString);
|
|
companyId = randomUUID(); agentId = randomUUID(); userId = randomUUID();
|
|
await db.insert(authUsers).values({ id: userId, name: "Board", email: `${userId}@example.test`,
|
|
emailVerified: true, createdAt: new Date(), updatedAt: new Date() });
|
|
await db.insert(companies).values({ id: companyId, name: "Paperclip", issuePrefix: "FCF",
|
|
requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: userId });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "idle",
|
|
adapterType: "paperclip_runner", adapterConfig: {}, runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } }, permissions: {} });
|
|
}, 60_000);
|
|
|
|
afterEach(async () => {
|
|
resetBacklogContinuationPacingForTests(db);
|
|
await db.execute(sql`truncate table issue_recovery_actions, native_run_finalizations, tool_invocations,
|
|
issue_approvals, approvals, issue_thread_interactions, activity_log, agent_wakeup_requests, environment_leases, execution_workspaces, heartbeat_runs, issues cascade`);
|
|
});
|
|
|
|
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
|
|
|
|
/** A run that ended without finishing, in the shape today's release leaves it. */
|
|
async function unfinishedRun(seed: Seed) {
|
|
const issueId = randomUUID(), runId = randomUUID();
|
|
const finishedAt = new Date(Date.now() - 60_000);
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Ship the release notes", status: "blocked",
|
|
assigneeAgentId: agentId, statusVersion: 4, blockedTransitionAt: finishedAt });
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "assignment", issueId,
|
|
nativeIssueId: seed.runtime === "native" ? issueId : null, runtimeMode: seed.runtime,
|
|
status: seed.status ?? "failed", errorCode: seed.errorCode, error: seed.error ?? seed.errorCode,
|
|
startedAt: new Date(finishedAt.getTime() - 10 * 60_000), finishedAt, createdAt: new Date(finishedAt.getTime() - 11 * 60_000),
|
|
responsibleUserId: userId, contextSnapshot: { issueId, wakeReason: "issue_assigned" },
|
|
...(seed.runtime === "native" ? { nativePhase: "terminal_failure" }
|
|
: { runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } } }) });
|
|
if (seed.runtime === "native") {
|
|
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "terminal_failure",
|
|
recoveryState: seed.coordinatorState ?? "blocked", failureCode: "native_continuation_requires_reconciliation", leaseOwner: null });
|
|
}
|
|
let actionId: string | null = null;
|
|
if ((seed.hold ?? "active") !== "none") {
|
|
const native = seed.runtime === "native";
|
|
const settled = seed.hold === "settled_blocked";
|
|
const [action] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
|
status: settled ? "resolved" : "active", outcome: settled ? "blocked" : null, resolvedAt: settled ? finishedAt : null,
|
|
ownerType: "board", returnOwnerAgentId: agentId,
|
|
cause: native ? "native_continuation_requires_reconciliation" : "legacy_execution_requires_reconciliation",
|
|
fingerprint: native ? `native-continuation:${runId}` : `legacy-execution:${runId}`,
|
|
nextAction: "Inspect the original failure and reconcile the previous execution before continuing. Automatic recovery cannot start another incident.",
|
|
maxAttempts: 3, createdAt: finishedAt,
|
|
evidence: { runId, originalFailureCode: seed.errorCode, nativeFailureBlock: { runId, statusVersion: 4 },
|
|
...(settled ? { automaticRecovery: { policy: "preserve_without_replay_v1", runId, replay: "blocked",
|
|
actionOutcome: "unknown", recordedAt: finishedAt.toISOString() } } : {}) } }).returning();
|
|
actionId = action!.id;
|
|
}
|
|
return { issueId, runId, actionId };
|
|
}
|
|
|
|
const issue = async (id: string) => (await db.select().from(issues).where(eq(issues.id, id)))[0]!;
|
|
const successors = async (runId: string) => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId));
|
|
/** Keeps the agent's only slot busy, so a delivered continuation stays observable as queued. */
|
|
async function occupySlot() {
|
|
const [other] = await db.insert(issues).values({ companyId, title: "Other work", status: "in_progress", assigneeAgentId: agentId }).returning();
|
|
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: other!.id, status: "running", contextSnapshot: { issueId: other!.id } });
|
|
}
|
|
async function deliver() {
|
|
const heartbeat = heartbeatService(db);
|
|
await deliverReconciledExecutions(db, heartbeat.wakeup);
|
|
await deliverReconciledExecutions(db, heartbeat.wakeup);
|
|
}
|
|
|
|
async function expectOneFreshSuccessor(f: { issueId: string; runId: string }) {
|
|
expect(await getExecutionBlocker(db, companyId, f.issueId)).toBeNull();
|
|
expect(await issue(f.issueId)).toMatchObject({ status: "todo" });
|
|
await occupySlot();
|
|
await deliver();
|
|
const next = await successors(f.runId);
|
|
expect(next).toHaveLength(1);
|
|
expect(next[0]!.contextSnapshot).toMatchObject({ issueId: f.issueId, previousRunId: f.runId, forceFreshSession: true,
|
|
wakeReason: "issue_recovery_action_restored" });
|
|
const [record] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, f.issueId),
|
|
sql`${issueRecoveryActions.evidence}->'freshContinuation' is not null`));
|
|
expect(record!.evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: next[0]!.id,
|
|
freshContinuation: { schema: FRESH_CONTINUATION_SCHEMA, sourceRunId: f.runId },
|
|
executionReconciliation: { runId: f.runId, providerStopped: true, actionOutcome: "mixed" } });
|
|
expect(record!.nextAction).not.toMatch(adminText);
|
|
expect(record!.ownerType).toBe("system");
|
|
return next[0]!;
|
|
}
|
|
|
|
// The five most frequent endings on staging (2026-11-12): each left the task
|
|
// Blocked for a person with no successor run.
|
|
it.each([
|
|
{ errorCode: "native_continuation_requires_reconciliation", error: "runner_remote_process_identity_unavailable", runtime: "native" as const },
|
|
{ errorCode: "controller_lease_lost", error: "Legacy controller lease lost", runtime: "legacy" as const },
|
|
{ errorCode: "process_lost", error: "Process lost -- server may have restarted; retrying once", runtime: "legacy" as const },
|
|
{ errorCode: "server_shutdown_interrupted", error: "Interrupted by graceful server shutdown (SIGTERM)", runtime: "legacy" as const,
|
|
status: "interrupted" as const },
|
|
{ errorCode: "native_session_interrupted", error: "runner_ingress_auth_failed: Runner ingress did not complete PRP authentication",
|
|
runtime: "native" as const },
|
|
])("continues a task whose run ended with $errorCode in exactly one fresh run from saved work", async seed => {
|
|
const f = await unfinishedRun(seed);
|
|
await settleUnrecoverableExecutions(db);
|
|
const hold = (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!)))[0]!;
|
|
expect(hold).toMatchObject({ status: "resolved", outcome: "restored" });
|
|
expect(hold.evidence.automaticRecovery).toBeUndefined();
|
|
await expectOneFreshSuccessor(f);
|
|
}, 60_000);
|
|
|
|
// The only native swap failure left on a later staging swap: the Runner
|
|
// resumed a harness that was not suspended or was gone.
|
|
// Exact resume is fixed elsewhere; whatever still fails must fall back.
|
|
it.each([
|
|
{ errorCode: "native_session_interrupted",
|
|
error: "runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_not_suspended" },
|
|
{ errorCode: "native_session_interrupted",
|
|
error: "runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_unavailable" },
|
|
{ errorCode: "runner_harness_state_mismatch", error: "runner_harness_state_mismatch: resumed_unavailable" },
|
|
])("continues a task whose resumed Runner harness was in the wrong state ($error)", async seed => {
|
|
const f = await unfinishedRun({ runtime: "native", ...seed });
|
|
await settleUnrecoverableExecutions(db);
|
|
await expectOneFreshSuccessor(f);
|
|
}, 60_000);
|
|
|
|
// Staging: the person had already pressed Retry, and the automatic successor
|
|
// could not carry that authorization forward. That is the system's failure,
|
|
// not the person's: the task continues under its responsible user.
|
|
it("starts the continuation when the failed run's exhaustion hold lands after the continuation was recorded", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_runner_process_exited",
|
|
error: "native_session_recovery_failed: Error: runner_local_connect_failed: runnerd exited unexpectedly" });
|
|
await settleUnrecoverableExecutions(db);
|
|
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog", status: "active",
|
|
ownerType: "board", returnOwnerAgentId: agentId, cause: "native_session_retry_exhausted", fingerprint: `native-retry:${f.runId}`,
|
|
nextAction: "Automatic recovery stopped.", maxAttempts: 3,
|
|
evidence: { runId: f.runId, coordinatorAttempt: 3, recoveryDisposition: "native_session_retry_exhausted" } });
|
|
await deliver();
|
|
await continueHeldExecutions(db); await settleUnrecoverableExecutions(db); await deliver();
|
|
expect(await successors(f.runId)).toHaveLength(1);
|
|
}, 60_000);
|
|
|
|
it("keeps a person's newer block when the failed run's exhaustion hold lands after the continuation was recorded", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_runner_process_exited",
|
|
error: "native_session_recovery_failed: Error: runner_local_connect_failed: runnerd exited unexpectedly" });
|
|
await settleUnrecoverableExecutions(db);
|
|
const decisionId = randomUUID();
|
|
await db.update(issues).set({ status: "blocked", statusVersion: 9, lastStatusDecisionId: decisionId, blockedTransitionAt: new Date() })
|
|
.where(eq(issues.id, f.issueId));
|
|
const [late] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog", status: "active",
|
|
ownerType: "board", returnOwnerAgentId: agentId, cause: "native_session_retry_exhausted", fingerprint: `native-retry:${f.runId}`,
|
|
nextAction: "Automatic recovery stopped.", maxAttempts: 3,
|
|
evidence: { runId: f.runId, coordinatorAttempt: 3, recoveryDisposition: "native_session_retry_exhausted", nativeFailureBlock: { runId: f.runId, statusVersion: 4 } } }).returning();
|
|
await settleUnrecoverableExecutions(db);
|
|
expect(await issue(f.issueId)).toMatchObject({ status: "blocked", lastStatusDecisionId: decisionId });
|
|
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, late!.id)))[0]).toMatchObject({ status: "resolved", ownerType: "system" });
|
|
}, 60_000);
|
|
|
|
it("continues a task whose automatic retry lost the person's continuation authorization", async () => {
|
|
expect(PERSON_OWNED_FAILURE_CODES).not.toContain("continuation_user_authorization_missing");
|
|
const f = await unfinishedRun({ runtime: "legacy", errorCode: "continuation_user_authorization_missing",
|
|
error: "The automatic retry could not revalidate the original user continuation" });
|
|
await settleUnrecoverableExecutions(db);
|
|
const next = await expectOneFreshSuccessor(f);
|
|
expect(next.responsibleUserId).toBe(userId);
|
|
}, 60_000);
|
|
|
|
it("falls back for a failure code nobody has classified yet", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "runner_new_failure_added_next_week" });
|
|
await settleUnrecoverableExecutions(db);
|
|
await expectOneFreshSuccessor(f);
|
|
}, 60_000);
|
|
|
|
it("creates one successor when two servers call it at once", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation",
|
|
error: "runner_remote_process_identity_unavailable" });
|
|
const results = await Promise.all([
|
|
ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }),
|
|
ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }),
|
|
settleUnrecoverableExecutions(db).then(() => null),
|
|
]);
|
|
expect(results.filter(result => result?.kind === "continued")).toHaveLength(1);
|
|
expect(results.filter(result => result && result.kind !== "continued").map(result => result!.kind))
|
|
.toEqual([expect.stringMatching(/already_continued|successor_exists/)]);
|
|
await expectOneFreshSuccessor(f);
|
|
// A later call, from either server or a sweep, adds nothing.
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
|
|
.toMatchObject({ kind: expect.stringMatching(/already_continued|successor_exists/) });
|
|
await deliver();
|
|
expect(await successors(f.runId)).toHaveLength(1);
|
|
}, 60_000);
|
|
|
|
// Upgrade safety: rows saved by the previous release (a settled-but-blocked
|
|
// no-replay hold, its coordinator still awaiting a runner reattach) and a
|
|
// task already Blocked continue on the first pass after deploy, without Retry.
|
|
it("continues tasks that older releases left blocked by these codes", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation",
|
|
error: "runner_remote_process_identity_unavailable", hold: "settled_blocked", coordinatorState: "awaiting_runner_reattach" });
|
|
expect(await getExecutionBlocker(db, companyId, f.issueId)).toMatchObject({ recoveryActionId: f.actionId });
|
|
await settleUnrecoverableExecutions(db);
|
|
await expectOneFreshSuccessor(f);
|
|
}, 60_000);
|
|
|
|
/** A held record from an older release (settled blocked), its source run ended `ageMs` ago with `errorCode`. */
|
|
async function heldBacklog(errorCode: string, ageMs: number) {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode, hold: "settled_blocked" });
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - ageMs) }).where(eq(heartbeatRuns.id, f.runId));
|
|
return f;
|
|
}
|
|
const continuedFrom = async (runId: string) => db.select().from(issueRecoveryActions)
|
|
.where(sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,sourceRunId}' = ${runId}`);
|
|
const evidenceOf = async (actionId: string) =>
|
|
(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, actionId)))[0]!;
|
|
|
|
it("continues only recent interruptions from the held backlog, newest first, and leaves the rest untouched (staging herd)", async () => {
|
|
// paperclip.staging, 22:53Z: the first release with the fallback continued 176 held tasks in 30 minutes;
|
|
// 74 had a source run that ended more than 7 days ago, many with configuration or provider failures.
|
|
const HOUR = 60 * 60_000;
|
|
const recentOld = await heldBacklog("native_restart_recovery_blocked", 6 * HOUR);
|
|
const recentNew = await heldBacklog("native_continuation_requires_reconciliation", 1 * HOUR);
|
|
const weekOld = await heldBacklog("native_restart_recovery_blocked", 3 * 24 * HOUR);
|
|
const monthOld = await heldBacklog("native_session_interrupted", 28 * 24 * HOUR);
|
|
const providerFailure = await heldBacklog("runner_remote_provider_artifact_incompatible", 2 * HOUR);
|
|
const adapterFailure = await heldBacklog("adapter_failed", 2 * HOUR);
|
|
const untouched = [weekOld, monthOld, providerFailure, adapterFailure];
|
|
const before = await Promise.all(untouched.map(f => evidenceOf(f.actionId!)));
|
|
|
|
expect(await continueHeldExecutions(db, new Date(), 1)).toMatchObject({ continued: 1 });
|
|
expect(await continuedFrom(recentNew.runId)).toHaveLength(1);
|
|
expect(await continuedFrom(recentOld.runId)).toHaveLength(0);
|
|
for (let pass = 0; pass < 3; pass += 1) await continueHeldExecutions(db, new Date());
|
|
expect(await continuedFrom(recentOld.runId)).toHaveLength(1);
|
|
for (const [index, f] of untouched.entries()) {
|
|
expect(await continuedFrom(f.runId)).toHaveLength(0);
|
|
const after = await evidenceOf(f.actionId!);
|
|
expect({ status: after.status, evidence: after.evidence, updatedAt: after.updatedAt })
|
|
.toEqual({ status: before[index]!.status, evidence: before[index]!.evidence, updatedAt: before[index]!.updatedAt });
|
|
}
|
|
}, 120_000);
|
|
|
|
it("paces the held backlog to ten continuations a minute per server", async () => {
|
|
const records = [];
|
|
for (let index = 0; index < 12; index += 1) records.push(await heldBacklog("native_restart_recovery_blocked", (index + 1) * 60_000));
|
|
const now = new Date();
|
|
for (let pass = 0; pass < 4; pass += 1) await continueHeldExecutions(db, now);
|
|
const started = (await Promise.all(records.map(f => continuedFrom(f.runId)))).filter(rows => rows.length > 0).length;
|
|
expect(started).toBe(10);
|
|
// The next minute has room again.
|
|
await continueHeldExecutions(db, new Date(now.getTime() + 61_000));
|
|
expect((await Promise.all(records.map(f => continuedFrom(f.runId)))).filter(rows => rows.length > 0).length).toBe(12);
|
|
}, 180_000);
|
|
|
|
const sweepWake = (issueId: string) => heartbeatService(db).wakeup(agentId, { source: "automation", triggerDetail: "system",
|
|
reason: "issue_continuation_needed", payload: { issueId }, requestedByActorType: "system",
|
|
requestedByActorId: "idle-task-sweep", contextSnapshot: { issueId, taskId: issueId, wakeReason: "issue_continuation_needed" } });
|
|
const pendingRecord = async (issueId: string) => (await db.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.sourceIssueId, issueId), sql`${issueRecoveryActions.evidence}->'freshContinuation' is not null`)))[0];
|
|
|
|
// An ordinary system wake (the idle-task sweep's) reaches the fallback. The
|
|
// continuation is recorded pending and counts as delivered only once a run
|
|
// for it exists.
|
|
it("records a pending continuation for an ordinary continuation wake and delivers it once", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation",
|
|
error: "runner_remote_process_identity_unavailable", hold: "settled_blocked" });
|
|
await occupySlot();
|
|
expect(await sweepWake(f.issueId)).toBeNull();
|
|
expect(await getExecutionBlocker(db, companyId, f.issueId)).toBeNull();
|
|
expect((await pendingRecord(f.issueId))!.evidence).toMatchObject({ continuationDelivery: "pending",
|
|
freshContinuation: { sourceRunId: f.runId, trigger: "continuation_wake", delivery: "recorded" } });
|
|
// A second sweep wake adds nothing.
|
|
expect(await sweepWake(f.issueId)).toBeNull();
|
|
await deliver();
|
|
expect(await successors(f.runId)).toHaveLength(1);
|
|
expect((await pendingRecord(f.issueId))!.evidence.continuationDelivery).toBe("delivered");
|
|
}, 60_000);
|
|
|
|
// A gate after the hook must never consume the recovery: it stays pending
|
|
// and is delivered once the gate clears.
|
|
it("keeps the continuation pending while the agent's daily run limit declines it, then delivers it", async () => {
|
|
// The daily limit counts runs started in the current UTC day, and it reads
|
|
// the clock on every check. Freeze Date (timers stay real) so every check in
|
|
// this test sees the same day, even when UTC midnight passes mid-test.
|
|
const now = new Date();
|
|
vi.useFakeTimers({ toFake: ["Date"], now });
|
|
try {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted", hold: "settled_blocked" });
|
|
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1, maxDailyRuns: 1 } } }).where(eq(agents.id, agentId));
|
|
// Pin the runs to that day so the result does not depend on the hour:
|
|
// the source run started yesterday (as it does just after 00:00 UTC), and
|
|
// another run started today used the single run.
|
|
const todayStart = Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate());
|
|
await db.update(heartbeatRuns).set({ startedAt: new Date(todayStart - 60_000), createdAt: new Date(todayStart - 120_000) })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
const [other] = await db.insert(issues).values({ companyId, title: "Used today's run", status: "done", assigneeAgentId: agentId }).returning();
|
|
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: other!.id, status: "succeeded",
|
|
startedAt: now, finishedAt: now, contextSnapshot: { issueId: other!.id } });
|
|
expect(await sweepWake(f.issueId)).toBeNull();
|
|
await deliver();
|
|
expect(await successors(f.runId)).toHaveLength(0);
|
|
expect((await pendingRecord(f.issueId))!.evidence.continuationDelivery).toBe("pending");
|
|
// The limit clears: the next pass delivers it.
|
|
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, agentId));
|
|
await occupySlot();
|
|
await deliver();
|
|
expect(await successors(f.runId)).toHaveLength(1);
|
|
} finally {
|
|
vi.useRealTimers();
|
|
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, agentId));
|
|
}
|
|
}, 60_000);
|
|
|
|
it("leaves a paused agent's task waiting with that reason, not idle and not consumed", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted", hold: "settled_blocked" });
|
|
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, agentId));
|
|
try {
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "continuation_wake" }))
|
|
.toEqual({ kind: "person_owned", reason: "agent_not_available" });
|
|
expect(await pendingRecord(f.issueId)).toBeUndefined();
|
|
expect(await getExecutionBlocker(db, companyId, f.issueId)).toMatchObject({ recoveryActionId: f.actionId });
|
|
} finally {
|
|
await db.update(agents).set({ status: "idle" }).where(eq(agents.id, agentId));
|
|
}
|
|
}, 60_000);
|
|
|
|
// The chaos repro: the agent's only slot is held. The delivered continuation
|
|
// waits queued (never lost) and starts as soon as the slot frees.
|
|
it("leaves the delivered continuation queued while the agent's slot is full and starts it when the slot frees", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
|
|
const [other] = await db.insert(issues).values({ companyId, title: "Holds the slot", status: "in_progress", assigneeAgentId: agentId }).returning();
|
|
const [holder] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: other!.id, status: "running",
|
|
contextSnapshot: { issueId: other!.id } }).returning();
|
|
await deliver();
|
|
const [next] = await successors(f.runId);
|
|
expect(next).toMatchObject({ status: "queued" });
|
|
expect((await pendingRecord(f.issueId))!.evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: next!.id });
|
|
// The slot frees: the queued continuation is dispatched.
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, holder!.id));
|
|
await heartbeatService(db).resumeQueuedRuns();
|
|
await vi.waitFor(async () => {
|
|
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, next!.id));
|
|
expect(current!.status).not.toBe("queued");
|
|
}, { timeout: 20_000, interval: 100 });
|
|
}, 60_000);
|
|
|
|
// The Retry banner never shows while Paperclip is continuing the task itself.
|
|
it("presents a task whose continuation is recorded or scheduled as continuing, with no Retry", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_restart_recovery_blocked" });
|
|
const presentation = async () => {
|
|
const task = await issue(f.issueId);
|
|
return taskInterruptionPresentation(db, { id: task.id, companyId, status: task.status,
|
|
assigneeAgentId: task.assigneeAgentId, projectId: task.projectId });
|
|
};
|
|
// A scheduled check (exact recovery still has its chance): a timed wait, no Retry.
|
|
await recordFreshContinuationCheck(db, { companyId, actionId: f.actionId! },
|
|
{ kind: "exact_recovery_pending", reason: "native_finalizer_active", retryAt: new Date(Date.now() + 60_000) });
|
|
expect(await presentation()).toMatchObject({ state: "retry_scheduled", canRetry: false });
|
|
// Recorded, not yet started: continuing, no Retry.
|
|
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} - 'freshContinuationCheck'` })
|
|
.where(eq(issueRecoveryActions.id, f.actionId!));
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
|
|
expect(await presentation()).toMatchObject({ state: "working", canRetry: false });
|
|
}, 60_000);
|
|
|
|
it("never starts a second run when a person's Retry already continued the task", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
|
|
// A person's Retry started a run before the continuation was delivered.
|
|
const [retry] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: f.issueId, nativeIssueId: f.issueId,
|
|
status: "queued", contextSnapshot: { issueId: f.issueId, wakeReason: "task_retry" } }).returning();
|
|
await deliver();
|
|
expect(await successors(f.runId)).toHaveLength(0);
|
|
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.issueId, f.issueId))).toHaveLength(2);
|
|
expect((await pendingRecord(f.issueId))!.evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: retry!.id });
|
|
}, 60_000);
|
|
|
|
/** The continuation's own wake, exactly as the delivery worker sends it, without its pre-check. */
|
|
async function deliveryWake(f: { issueId: string; runId: string }) {
|
|
const record = (await pendingRecord(f.issueId))!;
|
|
return heartbeatService(db).wakeup(agentId, { source: "automation", triggerDetail: "system",
|
|
reason: "issue_recovery_action_restored", idempotencyKey: `execution-reconciliation:${record.id}`,
|
|
payload: { issueId: f.issueId, recoveryActionId: record.id }, requestedByActorType: "system", requestedByActorId: "execution-recovery",
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, recoveryActionId: record.id, previousRunId: f.runId,
|
|
retryOfRunId: f.runId, forceFreshSession: true, wakeReason: "issue_recovery_action_restored", source: "execution.reconciled" } });
|
|
}
|
|
|
|
it("lets a run that started and finished in the delivery window consume the continuation under the task lock", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
|
|
// Between the worker's pre-check and admission, saved input ran and finished.
|
|
const [intervening] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: f.issueId, nativeIssueId: f.issueId,
|
|
status: "succeeded", finishedAt: new Date(), contextSnapshot: { issueId: f.issueId, wakeReason: "issue_commented" } }).returning();
|
|
await occupySlot();
|
|
expect(await deliveryWake(f)).toBeNull();
|
|
expect(await successors(f.runId)).toHaveLength(0);
|
|
expect((await pendingRecord(f.issueId))!.evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: intervening!.id });
|
|
}, 60_000);
|
|
|
|
it("starts no run for a task that moved to review in the delivery window", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
|
|
await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, f.issueId));
|
|
await occupySlot();
|
|
expect(await deliveryWake(f)).toBeNull();
|
|
expect(await successors(f.runId)).toHaveLength(0);
|
|
expect((await pendingRecord(f.issueId))!.evidence.continuationDelivery).toBe("invalidated");
|
|
}, 60_000);
|
|
|
|
it("settles coalesced holds with their primary, so a task back in todo is not shown as working and the sweep sees it", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted" });
|
|
// A second no-replay hold of the same run, settled together with the first.
|
|
const [sibling] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog",
|
|
status: "resolved", outcome: "blocked", ownerType: "board", cause: "native_restart_recovery_blocked",
|
|
fingerprint: `native-restart:${f.runId}`, nextAction: "held", resolvedAt: new Date(),
|
|
evidence: { runId: f.runId, automaticRecovery: { policy: "preserve_without_replay_v1", runId: f.runId, replay: "blocked" } } }).returning();
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
|
|
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, sibling!.id)))[0]!.evidence.continuationDelivery)
|
|
.toBe("coalesced");
|
|
// The task moves to review before delivery: the primary is cancelled, and so is its sibling.
|
|
await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, f.issueId));
|
|
await occupySlot();
|
|
expect(await deliveryWake(f)).toBeNull();
|
|
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, sibling!.id)))[0]!.evidence.continuationDelivery)
|
|
.toBe("invalidated");
|
|
// Back to todo with nothing pending: a failed attempt with Retry, not "working".
|
|
await db.update(issues).set({ status: "todo" }).where(eq(issues.id, f.issueId));
|
|
const task = await issue(f.issueId);
|
|
expect(await taskInterruptionPresentation(db, { id: task.id, companyId, status: task.status,
|
|
assigneeAgentId: task.assigneeAgentId, projectId: task.projectId })).toMatchObject({ state: "failed", canRetry: true });
|
|
// The assigned-work sweep sees it as left with nothing pending: inside its
|
|
// settle window after the recent change it schedules the recheck, never "ok".
|
|
const check = await heartbeatService(db).checkAssignedWorkInvariant({ companyId, issueId: f.issueId });
|
|
expect(["violation", "settling"]).toContain(check?.verdict.kind);
|
|
}, 60_000);
|
|
|
|
// The agent is told to check, redo what is missing, and ask only through the
|
|
// normal approval step; never that someone must restore the previous attempt.
|
|
it("guides the agent to check and redo, never to wait for an operator to recover the previous attempt", async () => {
|
|
for (const unverified of [[], ["github.merge_pull_request"]]) {
|
|
const note = freshContinuationNote({ runId: randomUUID(), unverified, workspace: "restored_last_save" });
|
|
if (unverified.length) expect(note).toContain("first check whether it took effect");
|
|
expect(note).toContain("do it again and continue the task");
|
|
// It does not contradict "do not repeat completed actions": it scopes itself to unknown outcomes.
|
|
expect(note).toContain("Completed actions with recorded results are still not repeated");
|
|
expect(note).toContain("request it through the normal approval step");
|
|
expect(note).not.toMatch(/operator|recover its result|execution state|authorize rerunning/i);
|
|
}
|
|
// The same holds for the continuation context the agent actually receives.
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted" });
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
|
|
const continuation = await buildExecutionContinuation({ db, companyId, issueId: f.issueId, agentId,
|
|
context: { previousRunId: f.runId }, summary: null, exposeLowTrustRaw: false });
|
|
const text = JSON.stringify(continuation);
|
|
expect(text).toContain("was cut off before it finished");
|
|
expect(text).not.toMatch(/operator must|recover its result|recover .*execution state|authorize rerunning/i);
|
|
}, 60_000);
|
|
|
|
// A task whose run completes normally gets no fallback activity at all, even
|
|
// with an older hold on the task (a run queued across a swap that then ran).
|
|
it("leaves a task that completes normally alone, even with an older run's hold", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation", hold: "settled_blocked" });
|
|
const [later] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: f.issueId, nativeIssueId: f.issueId,
|
|
status: "succeeded", startedAt: new Date(), finishedAt: new Date(), contextSnapshot: { issueId: f.issueId } }).returning();
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" })).toEqual({ kind: "successor_exists" });
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: later!.id, trigger: "held_recovery" }))
|
|
.toEqual({ kind: "not_applicable", reason: "run_not_unfinished" });
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId));
|
|
await settleUnrecoverableExecutions(db);
|
|
await occupySlot();
|
|
expect(await sweepWake(f.issueId)).toBeNull();
|
|
await deliver();
|
|
expect(await pendingRecord(f.issueId)).toBeUndefined();
|
|
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.issueId, f.issueId))).toHaveLength(2);
|
|
}, 60_000);
|
|
|
|
it("starts no run for a task that finished while its continuation was pending", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId));
|
|
await deliver();
|
|
expect(await successors(f.runId)).toHaveLength(0);
|
|
expect((await pendingRecord(f.issueId))!.evidence.continuationDelivery).toBe("invalidated");
|
|
}, 60_000);
|
|
|
|
// Crash between writes: the decision committed, the process died before any
|
|
// run was created. Nothing is lost: the next delivery pass starts it.
|
|
it("delivers a committed continuation after a crash before its run was created", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
|
|
expect(await successors(f.runId)).toHaveLength(0);
|
|
// A new process: a fresh service instance and worker pass.
|
|
await occupySlot();
|
|
await deliver();
|
|
expect(await successors(f.runId)).toHaveLength(1);
|
|
}, 60_000);
|
|
|
|
// The cut-off run's agent may still be alive in its old, unreachable sandbox.
|
|
// Once the continuation is recorded it can neither write to the task through
|
|
// the API nor publish its workspace, and the next run restores the last save
|
|
// into a new root on a new allocation.
|
|
it("fences the cut-off run: its API writes are refused and its workspace epoch is abandoned", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation",
|
|
error: "runner_remote_process_identity_unavailable" });
|
|
const workspaceId = randomUUID(), snapshotId = null, leaseId = randomUUID(), environmentId = randomUUID();
|
|
const location = { environmentId, leaseId, providerLeaseId: "sandbox-1", remoteCwd: "/home/daytona/.paperclip-workspaces/a" };
|
|
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: f.issueId, mode: "isolated",
|
|
strategyType: "remote", name: "Task workspace", providerType: "remote", providerRef: location.remoteCwd,
|
|
metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 3, snapshotId,
|
|
source: { kind: "empty" }, location, writerRunId: f.runId, dirty: true } } });
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1",
|
|
workspaceId, leaseId, providerLeaseId: location.providerLeaseId, remoteCwd: location.remoteCwd, epoch: 3,
|
|
state: "prepared", snapshotId } } }).where(eq(heartbeatRuns.id, f.runId));
|
|
// Today a failed run's token still writes.
|
|
const [before] = await loadAgentRunAuthentication(db, { runId: f.runId, companyId, agentId });
|
|
expect(agentRunWritesRevoked(before)).toBe(false);
|
|
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
|
|
|
|
// The auth middleware's own row read refuses every write of the old token.
|
|
const [after] = await loadAgentRunAuthentication(db, { runId: f.runId, companyId, agentId });
|
|
expect(agentRunWritesRevoked(after)).toBe(true);
|
|
await expect(db.transaction(tx => assertAgentRunWriteAllowed(tx as unknown as typeof db, companyId, { agentId, runId: f.runId })))
|
|
.rejects.toMatchObject({ status: 403, details: { code: "agent_run_superseded" } });
|
|
// The workspace writer is fenced out and the next preparation starts from the last save in a new root.
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
|
|
expect(workspace!.metadata!.remoteWorkspace).toMatchObject({ epoch: 4, writerRunId: null, dirty: false, location: null });
|
|
expect(workspace!.providerRef).toBeNull();
|
|
expect(workspace!.metadata!.remoteWorkspaceAbandonedEpochs).toEqual([expect.objectContaining({ epoch: 3, runId: f.runId, location })]);
|
|
// The continuation's own token is unaffected.
|
|
await occupySlot();
|
|
await deliver();
|
|
const [next] = await successors(f.runId);
|
|
await expect(db.transaction(tx => assertAgentRunWriteAllowed(tx as unknown as typeof db, companyId, { agentId, runId: next!.id })))
|
|
.resolves.toBeTruthy();
|
|
}, 60_000);
|
|
|
|
// The staging shape: the new server cannot re-reach the old Runner, so the old
|
|
// run's sandbox lease is still active (or its cleanup is pending) with no stop
|
|
// receipt. The continuation must still start, not wait behind that cleanup.
|
|
it.each(["active", "pending_cleanup"] as const)("starts the continuation while the unreachable old lease is %s", async leaseStatus => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation",
|
|
error: "runner_remote_process_identity_unavailable" });
|
|
const [environment] = await db.insert(environments).values({ name: `sandbox-${f.runId}`, driver: "sandbox" }).returning();
|
|
await db.insert(environmentLeases).values({ companyId, issueId: f.issueId, heartbeatRunId: f.runId,
|
|
environmentId: environment!.id, provider: "daytona", providerLeaseId: `sandbox-${f.runId}`, status: leaseStatus,
|
|
leasePolicy: "retain_on_failure", metadata: { remoteCwd: "/home/daytona/work", driver: "sandbox" } });
|
|
await settleUnrecoverableExecutions(db);
|
|
await deliver();
|
|
const [next] = await successors(f.runId);
|
|
expect(next).toBeTruthy();
|
|
// Admission claims it: it leaves the queue instead of being declined for the old run's cleanup.
|
|
await vi.waitFor(async () => {
|
|
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, next!.id));
|
|
expect(current!.status).not.toBe("queued");
|
|
}, { timeout: 20_000, interval: 100 });
|
|
// The abandoned allocation stays fenced: the old run still cannot write.
|
|
const [old] = await loadAgentRunAuthentication(db, { runId: f.runId, companyId, agentId });
|
|
expect(agentRunWritesRevoked(old)).toBe(true);
|
|
}, 60_000);
|
|
|
|
// A legacy run whose local process may still be alive: the continuation must
|
|
// never write the same directory. It gets a new isolated workspace and the
|
|
// decision records the stop as unproven.
|
|
it("gives the continuation a new isolated workspace while the old local writer may be alive", async () => {
|
|
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost", error: "Legacy controller lease lost" });
|
|
// Its controller ran on another host (the replaced server), so no probe from here can prove it gone.
|
|
await db.update(heartbeatRuns).set({ processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000) })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace" });
|
|
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
|
|
expect(hold!.evidence).toMatchObject({ freshContinuation: { isolation: "fresh_isolated_workspace", stopProof: "unproven_isolated" },
|
|
executionReconciliation: { providerStopProof: "isolated" } });
|
|
await occupySlot();
|
|
await deliver();
|
|
const [next] = await successors(f.runId);
|
|
expect(next!.contextSnapshot).toMatchObject({ freshContinuationIsolation: f.actionId });
|
|
// Workspace selection gives it a new isolated workspace, never the old directory.
|
|
expect(await readFreshWorkspaceReadiness(db, next!)).toEqual({ mode: "fresh_isolated_workspace", retryIntentId: f.actionId,
|
|
sourceRunId: f.runId, retainedSourceIds: [] });
|
|
}, 60_000);
|
|
|
|
// The recorded process identity (host, PID space, start time) of this process.
|
|
const liveIdentity = () => ({ processPid: process.pid, processHostHash: currentProcessSpaceHash(),
|
|
processStartedAt: new Date(Date.now() - process.uptime() * 1_000) });
|
|
|
|
/** Moves the recorded start of the pending wait into the past. */
|
|
async function waitedFor(actionId: string, ms: number) {
|
|
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
|
|
'{freshContinuationCheck,firstPendingAt}', to_jsonb(${Date.now() - ms}::bigint))` }).where(eq(issueRecoveryActions.id, actionId));
|
|
}
|
|
|
|
it("lets a previous process still running on this host settle first, for about ten minutes at most", async () => {
|
|
const f = await unfinishedRun({ runtime: "legacy", errorCode: "process_lost" });
|
|
await db.update(heartbeatRuns).set(liveIdentity()).where(eq(heartbeatRuns.id, f.runId));
|
|
const first = await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" });
|
|
expect(first).toMatchObject({ kind: "exact_recovery_pending", reason: "previous_process_running" });
|
|
// Rechecks back off from one minute.
|
|
expect((first as { retryAt: Date }).retryAt.getTime()).toBeLessThanOrEqual(Date.now() + 61_000);
|
|
await recordFreshContinuationCheck(db, { companyId, actionId: f.actionId! }, first);
|
|
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
|
|
expect(hold!.evidence.freshContinuationCheck).toMatchObject({ checks: 1, firstPendingAt: expect.any(Number) });
|
|
// Past the bound it stops waiting and continues in a new isolated workspace.
|
|
await waitedFor(f.actionId!, FRESH_CONTINUATION_PENDING_BOUND_MS + 1_000);
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace" });
|
|
}, 60_000);
|
|
|
|
it("keeps waiting up to fifteen minutes while the exact-recovery owner shows progress", async () => {
|
|
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost" });
|
|
await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 365 * 86_400_000) })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
const waiting = await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" });
|
|
await recordFreshContinuationCheck(db, { companyId, actionId: f.actionId! }, waiting);
|
|
await waitedFor(f.actionId!, FRESH_CONTINUATION_PENDING_BOUND_MS + 1_000);
|
|
// The owner renewed its lease since the last check: progress.
|
|
await db.update(heartbeatRuns).set({ lastOutputAt: new Date(Date.now() + 1_000) }).where(eq(heartbeatRuns.id, f.runId));
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "exact_recovery_pending", reason: "controller_lease_live" });
|
|
await waitedFor(f.actionId!, FRESH_CONTINUATION_PROGRESS_BOUND_MS + 1_000);
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace" });
|
|
}, 60_000);
|
|
|
|
it("does not mistake a recycled PID for the old writer", async () => {
|
|
const f = await unfinishedRun({ runtime: "legacy", errorCode: "process_lost" });
|
|
// The PID is alive now, but it belongs to a process that started after the old one was recorded.
|
|
await db.update(heartbeatRuns).set({ ...liveIdentity(), processStartedAt: new Date("2020-01-01T00:00:00Z") })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "continued" });
|
|
}, 60_000);
|
|
|
|
/** A saved-work (workspace) recovery generation on the run's hold, as workspace recovery records it. */
|
|
async function savedWorkRecovery(f: { runId: string; actionId: string | null }, state: "awaiting_evidence" | "running") {
|
|
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
|
|
await db.update(issueRecoveryActions).set({ evidence: { ...hold!.evidence,
|
|
systemRecovery: { schema: "paperclip.system-recovery.v1", policyVersion: 1, phase: "reconcile_effects", state,
|
|
sourceRunId: f.runId, attempts: 1, retriesUsed: 0, revision: 2, failure: null, intentId: null, deadlineAt: null,
|
|
nextAttemptAt: null, firstFailureAt: null, freshAttemptRunId: null, providerCooldownAt: null,
|
|
attemptId: randomUUID(), generationId: randomUUID(), preservationOperationIds: [] } } })
|
|
.where(eq(issueRecoveryActions.id, f.actionId!));
|
|
}
|
|
|
|
it("bounds a saved-work recovery that is waiting with no progress, on a scheduled deadline", async () => {
|
|
const f = await unfinishedRun({ runtime: "legacy", errorCode: "server_shutdown_interrupted", status: "interrupted" });
|
|
await savedWorkRecovery(f, "awaiting_evidence");
|
|
// The held-run scan selects it: exact recovery keeps its first chance.
|
|
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 0 });
|
|
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
|
|
expect(hold!.evidence.freshContinuationCheck).toMatchObject({ kind: "exact_recovery_pending", reason: "saved_work_recovery_running" });
|
|
// With no other event, the worker is scheduled for the next check.
|
|
expect(await nextFreshContinuationCheckAt(db)).toBe((hold!.evidence.freshContinuationCheck as { nextCheckAt: number }).nextCheckAt);
|
|
// No progress by the bound: the task continues fresh in a new root, and the
|
|
// preempted recovery is no longer scheduled (its hold carries the decision),
|
|
// so a late result cannot reach the new workspace.
|
|
await waitedFor(f.actionId!, FRESH_CONTINUATION_PENDING_BOUND_MS + 1_000);
|
|
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
|
|
'{freshContinuationCheck,nextCheckAt}', to_jsonb(${Date.now() - 1_000}::bigint))` }).where(eq(issueRecoveryActions.id, f.actionId!));
|
|
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 1 });
|
|
const [settled] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
|
|
expect(settled!.evidence).toMatchObject({ executionReconciliation: { runId: f.runId },
|
|
freshContinuation: { isolation: "fresh_isolated_workspace" } });
|
|
}, 60_000);
|
|
|
|
it("lets a running saved-work recovery that keeps advancing use its own budget", async () => {
|
|
const f = await unfinishedRun({ runtime: "legacy", errorCode: "server_shutdown_interrupted", status: "interrupted" });
|
|
await savedWorkRecovery(f, "running");
|
|
const first = await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" });
|
|
await recordFreshContinuationCheck(db, { companyId, actionId: f.actionId! }, first);
|
|
await waitedFor(f.actionId!, FRESH_CONTINUATION_PROGRESS_BOUND_MS + 1_000);
|
|
// The recovery advanced (a new attempt) since the last check: it keeps going past 15 minutes.
|
|
await db.update(issueRecoveryActions).set({ updatedAt: new Date(Date.now() + 1_000),
|
|
evidence: sql`jsonb_set(${issueRecoveryActions.evidence}, '{systemRecovery,attempts}', '2'::jsonb)` })
|
|
.where(eq(issueRecoveryActions.id, f.actionId!));
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
|
|
.toMatchObject({ kind: "exact_recovery_pending", reason: "saved_work_recovery_running" });
|
|
// Its 30-minute budget is the cap.
|
|
await waitedFor(f.actionId!, FRESH_CONTINUATION_SAVED_WORK_BOUND_MS + 1_000);
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
|
|
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace" });
|
|
}, 60_000);
|
|
|
|
it("bounds every exact-recovery wait, here a controller lease that never lapses", async () => {
|
|
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost" });
|
|
await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 365 * 86_400_000) })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
const waiting = await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" });
|
|
expect(waiting).toMatchObject({ kind: "exact_recovery_pending", reason: "controller_lease_live" });
|
|
await recordFreshContinuationCheck(db, { companyId, actionId: f.actionId! }, waiting);
|
|
await waitedFor(f.actionId!, FRESH_CONTINUATION_PENDING_BOUND_MS + 1_000);
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace" });
|
|
}, 60_000);
|
|
|
|
it("keeps the task workspace when the old local writer is proven gone", async () => {
|
|
const f = await unfinishedRun({ runtime: "legacy", errorCode: "process_lost" });
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "continued", isolation: "writer_ended" });
|
|
}, 60_000);
|
|
|
|
it("settles only the reconciliation holds it owns and leaves restore holds and history alone", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted" });
|
|
const [history] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog",
|
|
status: "cancelled", outcome: "cancelled", ownerType: "board", cause: "native_restart_recovery_blocked",
|
|
fingerprint: `history:${f.runId}`, nextAction: "Old note", resolutionNote: "kept", resolvedAt: new Date(),
|
|
evidence: { runId: f.runId } }).returning();
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
|
|
const [untouched] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, history!.id));
|
|
expect(untouched).toMatchObject({ status: "cancelled", nextAction: "Old note", resolutionNote: "kept", ownerType: "board" });
|
|
expect(untouched!.evidence).toEqual({ runId: f.runId });
|
|
|
|
// A restore hold that workspace recovery has not finished keeps the task with it.
|
|
const g = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted" });
|
|
const [restore] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: g.issueId, kind: "active_run_watchdog",
|
|
status: "resolved", outcome: "blocked", ownerType: "system", cause: "native_continuation_requires_reconciliation",
|
|
fingerprint: `native-workspace:${g.runId}`, nextAction: "Recovering saved work", resolvedAt: new Date(),
|
|
evidence: { runId: g.runId, workspaceRestoreFailure: "restore_unsafe_archive",
|
|
automaticRecovery: { policy: "preserve_without_replay_v1", runId: g.runId, replay: "blocked" } } }).returning();
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: g.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "exact_recovery_pending", reason: "workspace_recovery_owns_restore" });
|
|
const [kept] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, restore!.id));
|
|
expect(kept!.evidence).toMatchObject({ workspaceRestoreFailure: "restore_unsafe_archive", automaticRecovery: { replay: "blocked" } });
|
|
expect(kept!.evidence.executionReconciliation).toBeUndefined();
|
|
}, 60_000);
|
|
|
|
it("schedules no recheck for a held task nothing would act on", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted", hold: "settled_blocked" });
|
|
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
|
|
freshContinuationCheck: { kind: "person_owned", reason: "approval_pending", checkedAt: Date.now() - 7_200_000,
|
|
nextCheckAt: Date.now() - 3_600_000 } })}::jsonb` }).where(eq(issueRecoveryActions.id, f.actionId!));
|
|
expect(await nextFreshContinuationCheckAt(db)).not.toBeNull();
|
|
// The task was cancelled after that check: the schedule goes quiet.
|
|
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId));
|
|
expect(await nextFreshContinuationCheckAt(db)).toBeNull();
|
|
expect(await heldExecutionsAwaitingFreshContinuation(db, new Date(), 10)).toEqual([]);
|
|
expect(await executionControlSchedule(db).disposition()).toBeNull();
|
|
}, 60_000);
|
|
|
|
it("schedules no disposition scan for a held record the backlog pass leaves alone, and still does for an eligible one", async () => {
|
|
const overdueCheck = (actionId: string) => db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
|
|
freshContinuationCheck: { kind: "backoff", checkedAt: Date.now() - 7_200_000, nextCheckAt: Date.now() - 3_600_000 } })}::jsonb` })
|
|
.where(eq(issueRecoveryActions.id, actionId));
|
|
const old = await heldBacklog("native_restart_recovery_blocked", 3 * 24 * 60 * 60_000);
|
|
await overdueCheck(old.actionId!);
|
|
expect(await nextFreshContinuationCheckAt(db)).toBeNull();
|
|
expect(await executionControlSchedule(db).disposition()).toBeNull();
|
|
const recent = await heldBacklog("native_restart_recovery_blocked", 60 * 60_000);
|
|
await overdueCheck(recent.actionId!);
|
|
expect(await nextFreshContinuationCheckAt(db)).not.toBeNull();
|
|
}, 60_000);
|
|
|
|
it("tells the agent the previous attempt was cut off and which actions are unverified", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted" });
|
|
await db.insert(toolInvocations).values({ companyId, agentId, issueId: f.issueId, runId: f.runId, toolName: "github.create_pull_request",
|
|
riskLevel: "write", status: "executing", startedAt: new Date() } as typeof toolInvocations.$inferInsert);
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
|
|
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
|
|
const decision = hold!.evidence.executionReconciliation as Record<string, string>;
|
|
expect(decision.outcomeEvidence).toContain(`run ${f.runId}) was cut off`);
|
|
expect(decision.outcomeEvidence).toContain("github.create_pull_request");
|
|
expect(decision.outcomeEvidence).not.toMatch(adminText);
|
|
}, 60_000);
|
|
|
|
it.each([true, false])("relies on saved input only while it still holds an undelivered message (already taken by the cut-off run: %s)", async taken => {
|
|
// R2 cell t003: the cut-off run had taken the follow-up into its turn; the fallback chose saved_input,
|
|
// promotion settled the message as delivered, and no run followed.
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_restart_recovery_blocked" });
|
|
const [comment] = await db.insert(issueComments).values({ companyId, issueId: f.issueId, authorType: "user",
|
|
authorUserId: userId, body: "Additional requirement: include FOLLOWUP-1" } as typeof issueComments.$inferInsert).returning();
|
|
if (taken) await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, wakeCommentIds: [comment!.id] } })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", triggerDetail: "system",
|
|
reason: "issue_execution_deferred", status: "deferred_issue_execution", requestedByActorType: "user", requestedByActorId: userId,
|
|
payload: { issueId: f.issueId, _paperclipWakeContext: { issueId: f.issueId, wakeReason: "issue_commented", wakeCommentIds: [comment!.id] } },
|
|
} as typeof agentWakeupRequests.$inferInsert);
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "continued", delivery: taken ? "recorded" : "saved_input" });
|
|
}, 60_000);
|
|
|
|
it.each([
|
|
{ title: "a pending approval", setup: "approval" as const, errorCode: "native_continuation_requires_reconciliation" },
|
|
{ title: "a credential the person has not shared", setup: "none" as const, errorCode: "configuration_incomplete" },
|
|
{ title: "a budget stop", setup: "none" as const, errorCode: "budget_blocked" },
|
|
])("leaves a task waiting on $title for the person", async ({ setup, errorCode }) => {
|
|
expect(PERSON_OWNED_FAILURE_CODES).toContain("configuration_incomplete");
|
|
const f = await unfinishedRun({ runtime: "native", errorCode });
|
|
if (setup === "approval") {
|
|
const [approval] = await db.insert(approvals).values({ companyId, type: "approve_ceo_strategy", status: "pending",
|
|
payload: {} } as typeof approvals.$inferInsert).returning();
|
|
await db.insert(issueApprovals).values({ companyId, issueId: f.issueId, approvalId: approval!.id } as typeof issueApprovals.$inferInsert);
|
|
}
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "person_owned" });
|
|
await settleUnrecoverableExecutions(db);
|
|
expect(await issue(f.issueId)).toMatchObject({ status: "blocked" });
|
|
await deliver();
|
|
expect(await successors(f.runId)).toHaveLength(0);
|
|
}, 60_000);
|
|
|
|
it("bounds continuations with the server-restart lane's hourly cap", () => {
|
|
expect(FRESH_CONTINUATIONS_PER_HOUR).toBe(SERVER_RESTART_RETRY_MAX_ATTEMPTS);
|
|
expect(FRESH_CONTINUATION_WINDOW_MS).toBe(SERVER_RESTART_RETRY_WINDOW_MS);
|
|
});
|
|
|
|
// Governance gates are never retired or bypassed by the fallback.
|
|
it.each(["question_pending", "user_owned_hold", "operator_continuation_recorded", "owner_gone_attestation"] as const)(
|
|
"leaves a task behind a real governance gate (%s) for the person", async gate => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation" });
|
|
if (gate === "question_pending") await db.insert(issueThreadInteractions).values({ companyId, issueId: f.issueId,
|
|
kind: "ask_user_questions", status: "pending", payload: { questions: [] } } as unknown as typeof issueThreadInteractions.$inferInsert);
|
|
if (gate === "user_owned_hold") await db.update(issueRecoveryActions).set({ ownerType: "user", ownerUserId: userId })
|
|
.where(eq(issueRecoveryActions.id, f.actionId!));
|
|
if (gate === "operator_continuation_recorded" || gate === "owner_gone_attestation") {
|
|
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
|
|
await db.update(issueRecoveryActions).set({ evidence: { ...hold!.evidence,
|
|
...(gate === "owner_gone_attestation" ? { ownerGoneAttestation: { actorId: userId } } : { explicitUserContinuation: { actorId: userId } }) } })
|
|
.where(eq(issueRecoveryActions.id, f.actionId!));
|
|
}
|
|
const before = (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!)))[0]!;
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "person_owned" });
|
|
const after = (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!)))[0]!;
|
|
expect(after.evidence).toEqual(before.evidence);
|
|
expect(after.status).toBe(before.status);
|
|
await deliver();
|
|
expect(await successors(f.runId)).toHaveLength(0);
|
|
}, 60_000);
|
|
|
|
// Interruption with an unknown outcome is not a governance gate: the task
|
|
// continues, but as a fresh run, never a replay, and with an audit trail.
|
|
it("continues an interrupted governed action as a fresh run, never a replay, and keeps the audit trail", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation",
|
|
error: "runner_remote_process_identity_unavailable", hold: "settled_blocked" });
|
|
const [inFlight] = await db.insert(toolInvocations).values({ companyId, agentId, issueId: f.issueId, runId: f.runId,
|
|
toolName: "github.merge_pull_request", riskLevel: "write", status: "executing", startedAt: new Date() } as typeof toolInvocations.$inferInsert)
|
|
.returning();
|
|
const original = (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!)))[0]!.evidence.automaticRecovery;
|
|
await settleUnrecoverableExecutions(db);
|
|
const next = await expectOneFreshSuccessor(f);
|
|
// A fresh session: the interrupted turn is not re-sent.
|
|
expect(next.contextSnapshot).toMatchObject({ forceFreshSession: true });
|
|
expect(next.sessionIdBefore).toBeNull();
|
|
// The cut-off tool call is not re-executed: it stays as recorded and no new invocation exists.
|
|
const invocations = await db.select().from(toolInvocations).where(eq(toolInvocations.issueId, f.issueId));
|
|
expect(invocations).toEqual([expect.objectContaining({ id: inFlight!.id, runId: f.runId, status: "executing" })]);
|
|
// The earlier no-replay decision stays as history beside the attributed fallback decision.
|
|
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
|
|
expect(hold!.evidence.automaticRecoveryHistory).toEqual(original);
|
|
expect(hold!.evidence.freshContinuationSettlement).toMatchObject({ actorType: "system", actorId: "fresh-continuation",
|
|
decision: "continue_in_fresh_run", replay: "none", reason: "exact_resumption_unavailable",
|
|
unverifiedActions: ["github.merge_pull_request"], recordedAt: expect.any(String) });
|
|
expect((hold!.evidence.executionReconciliation as Record<string, string>).outcomeEvidence).toContain("github.merge_pull_request");
|
|
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, f.issueId),
|
|
eq(activityLog.action, "issue.continued_from_saved_work")))).toHaveLength(1);
|
|
}, 60_000);
|
|
|
|
// The isolation decision is a property of the task's next run, whatever path starts it.
|
|
it("moves the task off the old writer's workspace and runs the isolated continuation before saved input", async () => {
|
|
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost", error: "Legacy controller lease lost" });
|
|
const [workspace] = await db.insert(executionWorkspaces).values({ companyId, sourceIssueId: f.issueId, mode: "isolated",
|
|
strategyType: "git_worktree", name: "Task workspace", cwd: "/tmp/old-writer" }).returning();
|
|
await db.update(issues).set({ executionWorkspaceId: workspace!.id, workspaceBindingRevision: 2 }).where(eq(issues.id, f.issueId));
|
|
await db.update(heartbeatRuns).set({ processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000),
|
|
contextSnapshot: { issueId: f.issueId, executionWorkspaceId: workspace!.id } }).where(eq(heartbeatRuns.id, f.runId));
|
|
// A person's saved message is waiting. It carries no isolation marker, so it cannot prepare the
|
|
// new workspace (a projectless task would be refused at admission once the binding is cleared):
|
|
// the isolated continuation runs first and the saved message follows it.
|
|
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: userId, payload: { issueId: f.issueId } });
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace", delivery: "recorded" });
|
|
// No delivery path can reach the old directory: the task is no longer bound to it.
|
|
expect(await issue(f.issueId)).toMatchObject({ executionWorkspaceId: null, workspaceBindingRevision: 3 });
|
|
const [kept] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
|
|
expect(kept!.cwd).toBe("/tmp/old-writer");
|
|
}, 60_000);
|
|
|
|
it("records its own isolated continuation when saved input could not avoid a shared directory", async () => {
|
|
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost" });
|
|
await db.update(heartbeatRuns).set({ processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000) })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: userId, payload: { issueId: f.issueId } });
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
|
|
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace", delivery: "recorded" });
|
|
}, 60_000);
|
|
|
|
it("backs off instead of blocking once a task used its hourly continuations", async () => {
|
|
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
|
|
for (let index = 0; index < FRESH_CONTINUATIONS_PER_HOUR; index++) {
|
|
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog", status: "resolved",
|
|
outcome: "restored", ownerType: "system", cause: "fresh_continuation", fingerprint: `fresh-continuation:${randomUUID()}`,
|
|
nextAction: "continued", evidence: { runId: randomUUID(), continuationDelivery: "delivered",
|
|
freshContinuation: { schema: FRESH_CONTINUATION_SCHEMA, sourceRunId: randomUUID(),
|
|
recordedAt: new Date(Date.now() - (index + 1) * 60_000).toISOString() } } });
|
|
}
|
|
await settleUnrecoverableExecutions(db);
|
|
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
|
|
// Not blocked for a person: the hold stays with the system and is checked again when the window frees a slot.
|
|
expect(hold).toMatchObject({ status: "active" });
|
|
expect(hold!.evidence.freshContinuationCheck).toMatchObject({ kind: "backoff", nextCheckAt: expect.any(Number) });
|
|
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 0 });
|
|
// With no other event, the disposition worker is scheduled for exactly the
|
|
// moment the window has room again.
|
|
const retryAt = (hold!.evidence.freshContinuationCheck as { nextCheckAt: number }).nextCheckAt;
|
|
expect(retryAt).toBeGreaterThan(Date.now());
|
|
expect(await nextFreshContinuationCheckAt(db)).toBe(retryAt);
|
|
expect(await executionControlSchedule(db).disposition()).toBe(retryAt);
|
|
// The hour passes (the window's records age out and the check falls due);
|
|
// the scheduled pass alone continues the task.
|
|
await db.execute(sql`update issue_recovery_actions set evidence = jsonb_set(evidence, '{freshContinuation,recordedAt}',
|
|
to_jsonb((now() - interval '2 hours')::text)) where evidence ? 'freshContinuation'`);
|
|
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
|
|
'{freshContinuationCheck,nextCheckAt}', to_jsonb(${Date.now() - 1_000}::bigint))` }).where(eq(issueRecoveryActions.id, f.actionId!));
|
|
expect(await executionControlSchedule(db).disposition()).toBeLessThanOrEqual(Date.now() + 60_000);
|
|
await settleUnrecoverableExecutions(db);
|
|
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
|
|
.toMatchObject({ kind: "already_continued" });
|
|
expect(await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests)).toHaveLength(0);
|
|
}, 60_000);
|
|
});
|