Files
PaperClipAI/server/src/services/fresh-continuation-fallback.test.ts
T
Devin FoleyandPaperclip c41145e72a fix: wait on pending questions and quiet provider holds (#22363)
## Thinking Path

> - Paperclip manages agents and the tasks they work on.
> - Recovery should continue unfinished work when it is eligible.
> - An active unanswered question is a persisted reason to wait for a
person.
> - Treating a later Retry as an answer makes the recovery sweep
repeatedly request continuations that admission refuses.
> - Provider-limit holds also produce one cancellation toast per task,
even though provider work never started.
> - This PR aligns recovery with persisted interaction status and
presents these waits accurately.

## Linked Issues or Issue Description

Refs: #20466. This addresses the question blocker presentation and
automatic recovery loop; it preserves the existing
interaction-resolution policy.

Related work: #19420 schedules provider quota retries, and #21570
handles notices and reply ownership in agent chats. This change concerns
assigned-task reconciliation, task/inbox status, and per-run
cancellation toasts.

**What happened?**

The assigned-work sweep ignored pending interactions whenever a newer
run carried `explicitUserContinuation`. Continuation admission still
refused the pending question, so repeated sweeps recorded unsuccessful
continuation attempts. The task could meanwhile show an older provider
error as its current blocker. Separately, each pre-dispatch
provider-limit hold generated a cancellation toast, and the one-hour
fallback was described as a known reset and promised start time.

**Expected behavior**

Wait quietly while the question remains active under the shared
interaction eligibility rules, then react to the existing resolution
notification. Dismissed or historical ordinary questions remain
answerable without blocking work; governed requests keep their gates.
Show the current question on task and inbox surfaces. Keep provider
failure history, suppress cancellation toasts for expected provider
holds, and describe scheduled availability checks without promising the
provider will accept work.

**Steps to reproduce**

1. Leave an `ask_user_questions` interaction pending with
`supersedeOnUserComment: false`.
2. Create a later failed/interrupted explicit Retry and run
assigned-work reconciliation repeatedly.
3. Observe that the question remains pending while the sweep attempts
fresh continuations.
4. Separately, let an agent with recent provider rate-limit evidence
receive automatic task wakes without a reported reset time.

## What Changed

- Remove inferred question supersession from both assigned-work fact
loading and candidate selection. Use the existing active-interaction
predicate consistently in those reads, fresh-continuation admission, and
the task projection. Cover dismissed/historical ordinary questions,
retained governed gates, repeated checks, idle workers, and the existing
resolution notification.
- Add the `question` interruption action, load active pending question
IDs alongside approvals in one company-scoped query, and show that
blocker on tasks and inbox rows. Preserve a supported manual Retry after
a failed attempt, suppress duplicate actions for saved input, and keep
other current user gates authoritative. This preserves the existing
issue-detail SQL statement budget. Historical run failures remain
intact. Refresh the shared source SDK input pin.
- Suppress cancellation toasts specifically for cancelled
`provider_limit_wait` runs. Label provider retry deadlines as
availability rechecks, retain relative-time formatting, and record
successful scheduling as an informational lifecycle event.
- Document the wait contract in the continuation architecture guide.

## Verification

- Demonstrated failing regressions before the implementation:
continuation loop, task projection, task notice, cancellation toast,
provider deadline wording, and deadline formatting.
- Initial focused Vitest qualification: 642 tests passed across 14
files, using:

  ```sh
  pnpm exec vitest run \

server/src/__tests__/{assigned-work-invariant,assigned-work-recheck-idle,heartbeat-provider-limit-wait,task-interruption-query}.test.ts
\

server/src/services/{task-interruption-presentation,execution-projection,fresh-continuation-fallback,issue-thread-interactions}.test.ts
\
    server/src/services/recovery/assigned-work-invariant.test.ts \

ui/src/{components/ExecutionBlockerNotice.test.tsx,context/LiveUpdatesProvider.test.ts,context/LiveUpdatesProvider.hook.test.tsx,lib/provider-limit-message.test.ts,pages/Inbox.test.tsx}
  ```
- `pnpm --filter @paperclipai/shared typecheck` — passed.
- Review follow-up: 157 tests passed in `assigned-work-invariant`,
`task-interruption-query`, and `fresh-continuation-fallback`; 575 passed
in `task-interruption-query`, `assigned-work-recheck-idle`,
`task-interruption-presentation`, `issue-question-context`,
`issue-thread-interactions`, and `explicit-native-continuation` (the
query file overlaps). The strengthened final
dismissed/historical/governed projection regression also passed
separately.
- Final query-consolidation verification: 157 tests passed across five
files, including the unchanged 64-statement issue-detail census and
mixed approval/question batches:

  ```sh
  pnpm exec vitest run \

server/src/__tests__/{issue-detail-statements,task-interruption-query,assigned-work-invariant,assigned-work-recheck-idle}.test.ts
\
    server/src/services/task-interruption-presentation.test.ts
  ```
- `pnpm --filter @paperclipai/server typecheck` — passed, including test
layers.
- Manual-recovery follow-up: 204 tests passed across seven
projection/recovery/notice suites, and all 104 Inbox tests passed. The
lost-runner integration still starts exactly one run from a manual Retry
and leaves the question unanswered. The saved-decision census verifies
one decision batch query for a 41-task list, with at most three
statements referencing interactions; the overall 64-statement
issue-detail limit remains unchanged.
- `pnpm --filter @paperclipai/ui typecheck` — passed.
- `pnpm --filter @paperclipai/ui build` — passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse
HEAD) --check` — passed.
- `pnpm verify` — exit 2: broader CI is required for changed global
inputs. Focused tests and package checks supply local evidence; broad CI
remains required.
- `git diff --check` and a local secret/instance-data scan — passed.
- Final CI on `1c16fe7c0d8`: [trusted CI
workflow](https://github.com/paperclipai/paperclip/actions/runs/44650155842)
passed, including all server shards, native-server verification, build,
canary, e2e, release registry, and serialized suites. All PR checks are
complete, with no failures; the branch is conflict-free.
- Greptile reviewed the latest revision, `1c16fe7c0d8`, at 5/5 with no
actionable findings or unresolved review threads.

The focused tests cover genuine provider failures, live work and
terminal tasks, unchanged task locks/retry delivery, preservation of
historical failures, and question resolution waking the affected task
while other active pending questions remain quiet. Review regressions
also cover dismissed/historical ordinary questions retaining Retry and
governed requests retaining their gates. No live task was retried or
answered for validation.

## Risks

- An active pending question now consistently blocks automatic
assigned-work recovery. Shared interaction eligibility determines when
ordinary conversation input becomes deferred or historical; governed
requests retain their authority. Automatic recovery no longer infers
that a newer Retry resolved an active question.
- Manual Retry keeps its existing admission policy and recovery checks.
The question notice preserves that supported action after a failed
attempt without changing approval or credential authority; pending saved
input prevents duplicate actions.
- The shared interruption action gains `question`; the task and inbox
consumers are updated together. The pending-question query returns only
task IDs and uses the existing company/task/status index.
- Provider cooldown policy, quotas, credentials, routing, cancellation
history, and durable retry timing are unchanged. The actual provider can
still return a rate limit at the recheck.
- No migration is required. Existing large owner modules receive small
localized changes; no code file crosses 1,000 lines.

## Model Used

OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, xhigh
reasoning effort, with code execution and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-19 19:54:15 -08:00

1566 lines
118 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
activityLog, agentWakeupRequests, agents, approvals, authUsers, companies, createDb, environmentLeases, environments, executionWorkspaces, heartbeatRuns, issueApprovals, issueRecoveryActions, issueThreadInteractions,
heartbeatRunEvents, issueComments, issues, nativeRunFinalizations, toolInvocations, companyMemberships,
} from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./execution-recovery-resolution.js";
import { PENDING_CONTINUATION_BOUND_MS } from "./recovery-continuation-pacing.js";
import { getExecutionBlocker } from "./execution-blocker.js";
import {
FRESH_CONTINUATIONS_PER_HOUR, FRESH_CONTINUATION_SCHEMA, PERSON_OWNED_FAILURE_CODES, continueHeldExecutions, ensureFreshContinuation,
resetBacklogContinuationPacingForTests,
continueFreshInTransaction, expediteFreshContinuationChecks, freshContinuationNote, heldExecutionsAwaitingFreshContinuation, runsAwaitingSystemContinuation, nextFreshContinuationCheckAt, recordFreshContinuationCheck, FRESH_CONTINUATION_PENDING_BOUND_MS, FRESH_CONTINUATION_PROGRESS_BOUND_MS, FRESH_CONTINUATION_SAVED_WORK_BOUND_MS,
} from "./fresh-continuation-fallback.js";
import { heartbeatService } from "./heartbeat.js";
import { buildExecutionContinuation } from "./execution-continuation.js";
import { currentProcessSpaceHash, observedProcessStartedAt } from "./controller-process-identity.js";
import { executionControlSchedule } from "./execution-control-work.js";
import { readFreshWorkspaceReadiness } from "./fresh-workspace-continuation.js";
import { taskInterruptionPresentation } from "./task-interruption-presentation.js";
import { agentRunWritesRevoked, assertAgentRunWriteAllowed } from "../agent-run-cancellation.js";
import { loadAgentRunAuthentication } from "../middleware/agent-run-authentication.js";
import { SERVER_RESTART_RETRY_MAX_ATTEMPTS, SERVER_RESTART_RETRY_WINDOW_MS } from "./execution-recovery-attempt.js";
import { FRESH_CONTINUATION_WINDOW_MS } from "./fresh-continuation-fallback.js";
import { createPostgresRunDispatchAdapter } from "../modules/run-dispatch/adapters/postgres.js";
import { createSystemRecovery } from "./system-recovery-state.js";
import { requestTaskRetry } from "./task-retry.js";
import { terminalizeLegacyExecution } from "./legacy-execution-recovery.js";
import { sharedContinuationAttempts } from "./recovery/continuation-ledger.js";
import { taskAutomaticAttempts } from "./recovery/task-automatic-attempts.js";
const support = await getEmbeddedPostgresTestSupport();
const suite = support.supported ? describe : describe.skip;
// Copy shown to people never asks them to inspect or repair a sandbox, workspace or run.
const adminText = /sandbox|inspect|repair|administrator|reconcile/i;
type Seed = {
runtime: "native" | "legacy"; status?: "failed" | "interrupted" | "timed_out" | "cancelled";
errorCode: string; error?: string;
/** How today's code leaves the run: an active hold (before disposition) or a settled-but-blocked hold (older releases). */
hold?: "active" | "settled_blocked" | "none";
coordinatorState?: string;
};
suite("fresh-continuation fallback", () => {
let db!: ReturnType<typeof createDb>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
let companyId = "", agentId = "", userId = "";
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("paperclip-fresh-continuation-");
db = createDb(temporary.connectionString);
companyId = randomUUID(); agentId = randomUUID(); userId = randomUUID();
await db.insert(authUsers).values({ id: userId, name: "Board", email: `${userId}@example.test`,
emailVerified: true, createdAt: new Date(), updatedAt: new Date() });
await db.insert(companies).values({ id: companyId, name: "Paperclip", issuePrefix: "FCF",
requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: userId });
await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "idle",
adapterType: "paperclip_runner", adapterConfig: {}, runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } }, permissions: {} });
}, 60_000);
afterEach(async () => {
resetBacklogContinuationPacingForTests(db);
await heartbeatService(db).drainActiveRunExecutions();
await db.delete(companyMemberships).where(and(eq(companyMemberships.companyId, companyId),
eq(companyMemberships.principalId, userId), eq(companyMemberships.principalType, "user")));
// The drain covers every background run, wake and resume dispatch, so
// nothing still holds these tables. A deadlock here is a leak to fix.
await db.execute(sql`truncate table issue_recovery_actions, native_run_finalizations, tool_invocations,
issue_approvals, approvals, issue_thread_interactions, activity_log, agent_wakeup_requests, environment_leases, execution_workspaces, heartbeat_runs, issues cascade`);
});
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
/** A run that ended without finishing, in the shape today's release leaves it. */
async function unfinishedRun(seed: Seed) {
const issueId = randomUUID(), runId = randomUUID();
const finishedAt = new Date(Date.now() - 60_000);
await db.insert(issues).values({ id: issueId, companyId, title: "Ship the release notes", status: "blocked",
assigneeAgentId: agentId, statusVersion: 4, blockedTransitionAt: finishedAt });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "assignment", issueId,
nativeIssueId: seed.runtime === "native" ? issueId : null, runtimeMode: seed.runtime,
status: seed.status ?? "failed", errorCode: seed.errorCode, error: seed.error ?? seed.errorCode,
startedAt: new Date(finishedAt.getTime() - 10 * 60_000), finishedAt, createdAt: new Date(finishedAt.getTime() - 11 * 60_000),
responsibleUserId: userId, contextSnapshot: { issueId, wakeReason: "issue_assigned" },
...(seed.runtime === "native" ? { nativePhase: "terminal_failure" }
: { runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } } }) });
if (seed.runtime === "native") {
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "terminal_failure",
recoveryState: seed.coordinatorState ?? "blocked", failureCode: "native_continuation_requires_reconciliation", leaseOwner: null });
}
let actionId: string | null = null;
if ((seed.hold ?? "active") !== "none") {
const native = seed.runtime === "native";
const settled = seed.hold === "settled_blocked";
const [action] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
status: settled ? "resolved" : "active", outcome: settled ? "blocked" : null, resolvedAt: settled ? finishedAt : null,
ownerType: "board", returnOwnerAgentId: agentId,
cause: native ? "native_continuation_requires_reconciliation" : "legacy_execution_requires_reconciliation",
fingerprint: native ? `native-continuation:${runId}` : `legacy-execution:${runId}`,
nextAction: "Inspect the original failure and reconcile the previous execution before continuing. Automatic recovery cannot start another incident.",
maxAttempts: 3, createdAt: finishedAt,
evidence: { runId, originalFailureCode: seed.errorCode, nativeFailureBlock: { runId, statusVersion: 4 },
...(settled ? { automaticRecovery: { policy: "preserve_without_replay_v1", runId, replay: "blocked",
actionOutcome: "unknown", recordedAt: finishedAt.toISOString() } } : {}) } }).returning();
actionId = action!.id;
}
return { issueId, runId, actionId };
}
const issue = async (id: string) => (await db.select().from(issues).where(eq(issues.id, id)))[0]!;
const successors = async (runId: string) => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId));
/** Keeps the agent's only slot busy, so a delivered continuation stays observable as queued. */
async function occupySlot() {
const [other] = await db.insert(issues).values({ companyId, title: "Other work", status: "in_progress", assigneeAgentId: agentId }).returning();
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: other!.id, status: "running", contextSnapshot: { issueId: other!.id } });
}
async function deliver() {
const heartbeat = heartbeatService(db);
await deliverReconciledExecutions(db, heartbeat.wakeup);
await deliverReconciledExecutions(db, heartbeat.wakeup);
}
async function expectOneFreshSuccessor(f: { issueId: string; runId: string }) {
expect(await getExecutionBlocker(db, companyId, f.issueId)).toBeNull();
expect(await issue(f.issueId)).toMatchObject({ status: "todo" });
await occupySlot();
await deliver();
const next = await successors(f.runId);
expect(next).toHaveLength(1);
expect(next[0]!.contextSnapshot).toMatchObject({ issueId: f.issueId, previousRunId: f.runId, forceFreshSession: true,
wakeReason: "issue_recovery_action_restored" });
const [record] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, f.issueId),
sql`${issueRecoveryActions.evidence}->'freshContinuation' is not null`));
expect(record!.evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: next[0]!.id,
freshContinuation: { schema: FRESH_CONTINUATION_SCHEMA, sourceRunId: f.runId },
executionReconciliation: { runId: f.runId, providerStopped: true, actionOutcome: "mixed" } });
expect(record!.nextAction).not.toMatch(adminText);
expect(record!.ownerType).toBe("system");
return next[0]!;
}
// The five most frequent endings on staging (2026-11-12): each left the task
// Blocked for a person with no successor run.
it.each([
{ errorCode: "native_continuation_requires_reconciliation", error: "runner_remote_process_identity_unavailable", runtime: "native" as const },
{ errorCode: "controller_lease_lost", error: "Legacy controller lease lost", runtime: "legacy" as const },
{ errorCode: "process_lost", error: "Process lost -- server may have restarted; retrying once", runtime: "legacy" as const },
{ errorCode: "server_shutdown_interrupted", error: "Interrupted by graceful server shutdown (SIGTERM)", runtime: "legacy" as const,
status: "interrupted" as const },
{ errorCode: "native_session_interrupted", error: "runner_ingress_auth_failed: Runner ingress did not complete PRP authentication",
runtime: "native" as const },
])("continues a task whose run ended with $errorCode in exactly one fresh run from saved work", async seed => {
const f = await unfinishedRun(seed);
await settleUnrecoverableExecutions(db);
const hold = (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!)))[0]!;
expect(hold).toMatchObject({ status: "resolved", outcome: "restored" });
expect(hold.evidence.automaticRecovery).toBeUndefined();
await expectOneFreshSuccessor(f);
}, 60_000);
// The only native swap failure left on a later staging swap: the Runner
// resumed a harness that was not suspended or was gone.
// Exact resume is fixed elsewhere; whatever still fails must fall back.
it.each([
{ errorCode: "native_session_interrupted",
error: "runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_not_suspended" },
{ errorCode: "native_session_interrupted",
error: "runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_unavailable" },
{ errorCode: "runner_harness_state_mismatch", error: "runner_harness_state_mismatch: resumed_unavailable" },
])("continues a task whose resumed Runner harness was in the wrong state ($error)", async seed => {
const f = await unfinishedRun({ runtime: "native", ...seed });
await settleUnrecoverableExecutions(db);
await expectOneFreshSuccessor(f);
}, 60_000);
// Staging: the person had already pressed Retry, and the automatic successor
// could not carry that authorization forward. That is the system's failure,
// not the person's: the task continues under its responsible user.
it("starts the continuation when the failed run's exhaustion hold lands after the continuation was recorded", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_runner_process_exited",
error: "native_session_recovery_failed: Error: runner_local_connect_failed: runnerd exited unexpectedly" });
await settleUnrecoverableExecutions(db);
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog", status: "active",
ownerType: "board", returnOwnerAgentId: agentId, cause: "native_session_retry_exhausted", fingerprint: `native-retry:${f.runId}`,
nextAction: "Automatic recovery stopped.", maxAttempts: 3,
evidence: { runId: f.runId, coordinatorAttempt: 3, recoveryDisposition: "native_session_retry_exhausted" } });
await deliver();
await continueHeldExecutions(db); await settleUnrecoverableExecutions(db); await deliver();
expect(await successors(f.runId)).toHaveLength(1);
}, 60_000);
it("keeps a person's newer block when the failed run's exhaustion hold lands after the continuation was recorded", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_runner_process_exited",
error: "native_session_recovery_failed: Error: runner_local_connect_failed: runnerd exited unexpectedly" });
await settleUnrecoverableExecutions(db);
const decisionId = randomUUID();
await db.update(issues).set({ status: "blocked", statusVersion: 9, lastStatusDecisionId: decisionId, blockedTransitionAt: new Date() })
.where(eq(issues.id, f.issueId));
const [late] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog", status: "active",
ownerType: "board", returnOwnerAgentId: agentId, cause: "native_session_retry_exhausted", fingerprint: `native-retry:${f.runId}`,
nextAction: "Automatic recovery stopped.", maxAttempts: 3,
evidence: { runId: f.runId, coordinatorAttempt: 3, recoveryDisposition: "native_session_retry_exhausted", nativeFailureBlock: { runId: f.runId, statusVersion: 4 } } }).returning();
await settleUnrecoverableExecutions(db);
expect(await issue(f.issueId)).toMatchObject({ status: "blocked", lastStatusDecisionId: decisionId });
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, late!.id)))[0]).toMatchObject({ status: "resolved", ownerType: "system" });
}, 60_000);
it("continues a task whose automatic retry lost the person's continuation authorization", async () => {
expect(PERSON_OWNED_FAILURE_CODES).not.toContain("continuation_user_authorization_missing");
const f = await unfinishedRun({ runtime: "legacy", errorCode: "continuation_user_authorization_missing",
error: "The automatic retry could not revalidate the original user continuation" });
await settleUnrecoverableExecutions(db);
const next = await expectOneFreshSuccessor(f);
expect(next.responsibleUserId).toBe(userId);
}, 60_000);
it("falls back for a failure code nobody has classified yet", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "runner_new_failure_added_next_week" });
await settleUnrecoverableExecutions(db);
await expectOneFreshSuccessor(f);
}, 60_000);
it("creates one successor when two servers call it at once", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation",
error: "runner_remote_process_identity_unavailable" });
const results = await Promise.all([
ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }),
ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }),
settleUnrecoverableExecutions(db).then(() => null),
]);
expect(results.filter(result => result?.kind === "continued")).toHaveLength(1);
expect(results.filter(result => result && result.kind !== "continued").map(result => result!.kind))
.toEqual([expect.stringMatching(/already_continued|successor_exists/)]);
await expectOneFreshSuccessor(f);
// A later call, from either server or a sweep, adds nothing.
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
.toMatchObject({ kind: expect.stringMatching(/already_continued|successor_exists/) });
await deliver();
expect(await successors(f.runId)).toHaveLength(1);
}, 60_000);
// Upgrade safety: rows saved by the previous release (a settled-but-blocked
// no-replay hold, its coordinator still awaiting a runner reattach) and a
// task already Blocked continue on the first pass after deploy, without Retry.
it("continues tasks that older releases left blocked by these codes", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation",
error: "runner_remote_process_identity_unavailable", hold: "settled_blocked", coordinatorState: "awaiting_runner_reattach" });
expect(await getExecutionBlocker(db, companyId, f.issueId)).toMatchObject({ recoveryActionId: f.actionId });
await settleUnrecoverableExecutions(db);
await expectOneFreshSuccessor(f);
}, 60_000);
/** A held record from an older release (settled blocked), its source run ended `ageMs` ago with `errorCode`. */
async function heldBacklog(errorCode: string, ageMs: number) {
const f = await unfinishedRun({ runtime: "native", errorCode, hold: "settled_blocked" });
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - ageMs) }).where(eq(heartbeatRuns.id, f.runId));
return f;
}
const continuedFrom = async (runId: string) => db.select().from(issueRecoveryActions)
.where(sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,sourceRunId}' = ${runId}`);
const evidenceOf = async (actionId: string) =>
(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, actionId)))[0]!;
it("continues only recent interruptions from the held backlog, newest first, and leaves the rest untouched (staging herd)", async () => {
// paperclip.staging, 22:53Z: the first release with the fallback continued 176 held tasks in 30 minutes;
// 74 had a source run that ended more than 7 days ago, many with configuration or provider failures.
const HOUR = 60 * 60_000;
const recentOld = await heldBacklog("native_restart_recovery_blocked", 6 * HOUR);
const recentNew = await heldBacklog("native_continuation_requires_reconciliation", 1 * HOUR);
const weekOld = await heldBacklog("native_restart_recovery_blocked", 3 * 24 * HOUR);
const monthOld = await heldBacklog("native_session_interrupted", 28 * 24 * HOUR);
const providerFailure = await heldBacklog("runner_remote_provider_artifact_incompatible", 2 * HOUR);
const adapterFailure = await heldBacklog("adapter_failed", 2 * HOUR);
const untouched = [weekOld, monthOld, providerFailure, adapterFailure];
const before = await Promise.all(untouched.map(f => evidenceOf(f.actionId!)));
expect(await continueHeldExecutions(db, new Date(), 1)).toMatchObject({ continued: 1 });
expect(await continuedFrom(recentNew.runId)).toHaveLength(1);
expect(await continuedFrom(recentOld.runId)).toHaveLength(0);
for (let pass = 0; pass < 3; pass += 1) await continueHeldExecutions(db, new Date());
expect(await continuedFrom(recentOld.runId)).toHaveLength(1);
for (const [index, f] of untouched.entries()) {
expect(await continuedFrom(f.runId)).toHaveLength(0);
const after = await evidenceOf(f.actionId!);
expect({ status: after.status, evidence: after.evidence, updatedAt: after.updatedAt })
.toEqual({ status: before[index]!.status, evidence: before[index]!.evidence, updatedAt: before[index]!.updatedAt });
}
}, 120_000);
it("paces the held backlog to ten continuations a minute per server", async () => {
const records = [];
for (let index = 0; index < 12; index += 1) records.push(await heldBacklog("native_restart_recovery_blocked", (index + 1) * 60_000));
const now = new Date();
for (let pass = 0; pass < 4; pass += 1) await continueHeldExecutions(db, now);
const started = (await Promise.all(records.map(f => continuedFrom(f.runId)))).filter(rows => rows.length > 0).length;
expect(started).toBe(10);
// The next minute has room again.
await continueHeldExecutions(db, new Date(now.getTime() + 61_000));
expect((await Promise.all(records.map(f => continuedFrom(f.runId)))).filter(rows => rows.length > 0).length).toBe(12);
}, 180_000);
const sweepWake = (issueId: string) => heartbeatService(db).wakeup(agentId, { source: "automation", triggerDetail: "system",
reason: "issue_continuation_needed", payload: { issueId }, requestedByActorType: "system",
requestedByActorId: "idle-task-sweep", contextSnapshot: { issueId, taskId: issueId, wakeReason: "issue_continuation_needed" } });
const pendingRecord = async (issueId: string) => (await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.sourceIssueId, issueId), sql`${issueRecoveryActions.evidence}->'freshContinuation' is not null`)))[0];
// An ordinary system wake (the idle-task sweep's) reaches the fallback. The
// continuation is recorded pending and counts as delivered only once a run
// for it exists.
it("records a pending continuation for an ordinary continuation wake and delivers it once", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation",
error: "runner_remote_process_identity_unavailable", hold: "settled_blocked" });
await occupySlot();
expect(await sweepWake(f.issueId)).toBeNull();
expect(await getExecutionBlocker(db, companyId, f.issueId)).toBeNull();
expect((await pendingRecord(f.issueId))!.evidence).toMatchObject({ continuationDelivery: "pending",
freshContinuation: { sourceRunId: f.runId, trigger: "continuation_wake", delivery: "recorded" } });
// A second sweep wake adds nothing.
expect(await sweepWake(f.issueId)).toBeNull();
await deliver();
expect(await successors(f.runId)).toHaveLength(1);
expect((await pendingRecord(f.issueId))!.evidence.continuationDelivery).toBe("delivered");
}, 60_000);
it("runs a person's Retry pressed before delivery as the recorded isolated continuation, never in the shared directory", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost", error: "Legacy controller lease lost" });
await db.update(heartbeatRuns).set({ processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000) })
.where(eq(heartbeatRuns.id, f.runId));
await db.update(issues).set({ projectId: null, projectWorkspaceId: null, executionWorkspaceId: null }).where(eq(issues.id, f.issueId));
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace", delivery: "recorded" });
await occupySlot();
const taskRuns = async () => db.select().from(heartbeatRuns).where(and(
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${f.issueId}`, sql`${heartbeatRuns.id} <> ${f.runId}`));
const retry = await requestTaskRetry({ db, companyId, issueId: f.issueId, actorId: userId, observedRunId: f.runId, idempotencyKey: randomUUID() })
.catch((error: unknown) => ({ error: String(error) }));
await deliver();
expect(retry).toMatchObject({ accepted: true });
const retryWake = async () => (await db.select().from(agentWakeupRequests)
.where(eq(agentWakeupRequests.id, (retry as { requestId: string }).requestId)))[0];
const runs = await taskRuns();
expect(runs).toHaveLength(1);
expect(await retryWake()).toMatchObject({ status: "coalesced", runId: runs[0]!.id });
expect(runs[0]!.contextSnapshot).toMatchObject({ freshContinuationIsolation: f.actionId });
expect(await readFreshWorkspaceReadiness(db, runs[0]!)).toMatchObject({ mode: "fresh_isolated_workspace", retryIntentId: f.actionId });
const endedAt = new Date();
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: endedAt })
.where(and(eq(heartbeatRuns.status, "running"), sql`${heartbeatRuns.contextSnapshot}->>'issueId' <> ${f.issueId}`));
await heartbeatService(db).cancelRun(runs[0]!.id);
await heartbeatService(db).resumeQueuedRuns();
await deliver();
const later = (await taskRuns()).filter(run => run.id !== runs[0]!.id);
expect(later.filter(run => run.wakeupRequestId === (retry as { requestId: string }).requestId
|| run.contextSnapshot?.wakeReason === "task_retry")).toEqual([]);
expect(await retryWake()).toMatchObject({ status: "coalesced", runId: runs[0]!.id });
}, 60_000);
it("consumes a person's Retry exactly once when the server stops right after admitting the isolated continuation", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost", error: "Legacy controller lease lost" });
await db.update(heartbeatRuns).set({ processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000) })
.where(eq(heartbeatRuns.id, f.runId));
await db.update(issues).set({ projectId: null, projectWorkspaceId: null, executionWorkspaceId: null }).where(eq(issues.id, f.issueId));
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace", delivery: "recorded" });
await occupySlot();
const retry = await requestTaskRetry({ db, companyId, issueId: f.issueId, actorId: userId, observedRunId: f.runId, idempotencyKey: randomUUID() });
expect(retry).toMatchObject({ accepted: true });
const retryWake = async () => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!)))[0];
const taskRuns = async () => db.select().from(heartbeatRuns).where(and(
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${f.issueId}`, sql`${heartbeatRuns.id} <> ${f.runId}`));
const admitted = await heartbeatService(db).wakeup(agentId, { source: "automation", triggerDetail: "system",
reason: "issue_recovery_action_restored", idempotencyKey: `execution-reconciliation:${f.actionId}`,
payload: { issueId: f.issueId, recoveryActionId: f.actionId }, requestedByActorType: "system", requestedByActorId: "execution-recovery",
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, recoveryActionId: f.actionId, previousRunId: f.runId, retryOfRunId: f.runId,
forceFreshSession: true, wakeReason: "issue_recovery_action_restored", source: "execution.reconciled",
freshContinuationIsolation: f.actionId } });
expect(admitted).toBeTruthy();
expect(await retryWake()).toMatchObject({ status: "coalesced", runId: admitted!.id });
await db.update(agentWakeupRequests).set({ status: "deferred_issue_execution", runId: null, finishedAt: null })
.where(eq(agentWakeupRequests.id, retry.requestId!));
await deliver();
expect((await taskRuns()).map(run => run.id)).toEqual([admitted!.id]);
expect(await retryWake()).toMatchObject({ status: "coalesced", runId: admitted!.id });
expect((await pendingRecord(f.issueId))!.evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: admitted!.id });
}, 60_000);
it("never folds a Retry for the admitted continuation's own failure into that ended continuation", async () => {
await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user",
status: "active", membershipRole: "member" });
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost", error: "Legacy controller lease lost" });
await db.update(heartbeatRuns).set({ processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000) })
.where(eq(heartbeatRuns.id, f.runId));
await db.update(issues).set({ projectId: null, projectWorkspaceId: null, executionWorkspaceId: null }).where(eq(issues.id, f.issueId));
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace", delivery: "recorded" });
const admitted = await heartbeatService(db).wakeup(agentId, { source: "automation", triggerDetail: "system",
reason: "issue_recovery_action_restored", idempotencyKey: `execution-reconciliation:${f.actionId}`,
payload: { issueId: f.issueId, recoveryActionId: f.actionId }, requestedByActorType: "system", requestedByActorId: "execution-recovery",
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, recoveryActionId: f.actionId, previousRunId: f.runId, retryOfRunId: f.runId,
forceFreshSession: true, wakeReason: "issue_recovery_action_restored", source: "execution.reconciled",
freshContinuationIsolation: f.actionId } });
await heartbeatService(db).cancelRun(admitted!.id);
const retry = await requestTaskRetry({ db, companyId, issueId: f.issueId, actorId: userId, observedRunId: admitted!.id, idempotencyKey: randomUUID() });
expect(retry).toMatchObject({ accepted: true });
await deliver();
await heartbeatService(db).resumeQueuedRuns();
await vi.waitFor(async () => {
const [retryWake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!));
expect(retryWake!.runId, JSON.stringify(retryWake)).toBeTruthy();
expect(retryWake!.runId).not.toBe(admitted!.id);
}, { timeout: 20_000, interval: 200 });
}, 60_000);
it("re-drives an isolated continuation pending past the bound itself; a repair wake never replaces it", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost", error: "Legacy controller lease lost" });
await db.update(heartbeatRuns).set({ processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000) })
.where(eq(heartbeatRuns.id, f.runId));
await db.update(issues).set({ projectId: null, projectWorkspaceId: null, executionWorkspaceId: null }).where(eq(issues.id, f.issueId));
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace", delivery: "recorded" });
await occupySlot();
const taskRuns = async () => db.select().from(heartbeatRuns).where(and(
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${f.issueId}`, sql`${heartbeatRuns.id} <> ${f.runId}`));
expect(await sweepWake(f.issueId)).toBeNull();
expect(await taskRuns()).toHaveLength(0);
const recordedAt = new Date(Date.now() - PENDING_CONTINUATION_BOUND_MS - 60_000).toISOString();
await db.update(issueRecoveryActions).set({ resolvedAt: new Date(recordedAt), evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
'{freshContinuation,recordedAt}', to_jsonb(${recordedAt}::text))` }).where(eq(issueRecoveryActions.id, f.actionId!));
expect(await sweepWake(f.issueId)).toBeNull();
expect(await taskRuns()).toHaveLength(0);
await deliver();
const runs = await taskRuns();
expect(runs).toHaveLength(1);
expect(runs[0]!.contextSnapshot).toMatchObject({ freshContinuationIsolation: f.actionId, forceFreshSession: true });
expect(await readFreshWorkspaceReadiness(db, runs[0]!)).toMatchObject({ mode: "fresh_isolated_workspace", retryIntentId: f.actionId });
const [record] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
expect(record!.evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: runs[0]!.id });
await sweepWake(f.issueId);
await deliver();
expect(await taskRuns()).toHaveLength(1);
}, 60_000);
// A gate after the hook must never consume the recovery: it stays pending
// and is delivered once the gate clears.
it("keeps the continuation pending while the agent's daily run limit declines it, then delivers it", async () => {
// The daily limit counts runs started in the current UTC day, and it reads
// the clock on every check. Freeze Date (timers stay real) so every check in
// this test sees the same day, even when UTC midnight passes mid-test.
const now = new Date();
vi.useFakeTimers({ toFake: ["Date"], now });
try {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted", hold: "settled_blocked" });
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1, maxDailyRuns: 1 } } }).where(eq(agents.id, agentId));
// Pin the runs to that day so the result does not depend on the hour:
// the source run started yesterday (as it does just after 00:00 UTC), and
// another run started today used the single run.
const todayStart = Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate());
await db.update(heartbeatRuns).set({ startedAt: new Date(todayStart - 60_000), createdAt: new Date(todayStart - 120_000) })
.where(eq(heartbeatRuns.id, f.runId));
const [other] = await db.insert(issues).values({ companyId, title: "Used today's run", status: "done", assigneeAgentId: agentId }).returning();
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: other!.id, status: "succeeded",
startedAt: now, finishedAt: now, contextSnapshot: { issueId: other!.id } });
expect(await sweepWake(f.issueId)).toBeNull();
await deliver();
expect(await successors(f.runId)).toHaveLength(0);
expect((await pendingRecord(f.issueId))!.evidence.continuationDelivery).toBe("pending");
// The limit clears: the next pass delivers it.
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, agentId));
await occupySlot();
await deliver();
expect(await successors(f.runId)).toHaveLength(1);
} finally {
vi.useRealTimers();
await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, agentId));
}
}, 60_000);
it("leaves a paused agent's task waiting with that reason, not idle and not consumed", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted", hold: "settled_blocked" });
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, agentId));
try {
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "continuation_wake" }))
.toEqual({ kind: "person_owned", reason: "agent_not_available" });
expect(await pendingRecord(f.issueId)).toBeUndefined();
expect(await getExecutionBlocker(db, companyId, f.issueId)).toMatchObject({ recoveryActionId: f.actionId });
} finally {
await db.update(agents).set({ status: "idle" }).where(eq(agents.id, agentId));
}
}, 60_000);
// The chaos repro: the agent's only slot is held. The delivered continuation
// waits queued (never lost) and starts as soon as the slot frees.
it("leaves the delivered continuation queued while the agent's slot is full and starts it when the slot frees", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
const [other] = await db.insert(issues).values({ companyId, title: "Holds the slot", status: "in_progress", assigneeAgentId: agentId }).returning();
const [holder] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: other!.id, status: "running",
contextSnapshot: { issueId: other!.id } }).returning();
await deliver();
const [next] = await successors(f.runId);
expect(next).toMatchObject({ status: "queued" });
expect((await pendingRecord(f.issueId))!.evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: next!.id });
// The slot frees: the queued continuation is dispatched.
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, holder!.id));
await heartbeatService(db).resumeQueuedRuns();
await vi.waitFor(async () => {
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, next!.id));
expect(current!.status).not.toBe("queued");
}, { timeout: 20_000, interval: 100 });
}, 60_000);
// The Retry banner never shows while Paperclip is continuing the task itself.
it("presents a task whose continuation is recorded or scheduled as continuing, with no Retry", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_restart_recovery_blocked" });
const presentation = async () => {
const task = await issue(f.issueId);
return taskInterruptionPresentation(db, { id: task.id, companyId, status: task.status,
assigneeAgentId: task.assigneeAgentId, projectId: task.projectId });
};
// A scheduled check (exact recovery still has its chance): a timed wait, no Retry.
await recordFreshContinuationCheck(db, { companyId, actionId: f.actionId! },
{ kind: "exact_recovery_pending", reason: "native_finalizer_active", retryAt: new Date(Date.now() + 60_000) });
expect(await presentation()).toMatchObject({ state: "retry_scheduled", canRetry: false });
// Recorded, not yet started: continuing, no Retry.
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} - 'freshContinuationCheck'` })
.where(eq(issueRecoveryActions.id, f.actionId!));
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
expect(await presentation()).toMatchObject({ state: "working", canRetry: false });
}, 60_000);
it("never starts a second run when a person's Retry already continued the task", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
// A person's Retry started a run before the continuation was delivered.
const [retry] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: f.issueId, nativeIssueId: f.issueId,
status: "queued", contextSnapshot: { issueId: f.issueId, wakeReason: "task_retry" } }).returning();
await deliver();
expect(await successors(f.runId)).toHaveLength(0);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.issueId, f.issueId))).toHaveLength(2);
expect((await pendingRecord(f.issueId))!.evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: retry!.id });
}, 60_000);
/** The continuation's own wake, exactly as the delivery worker sends it, without its pre-check. */
async function deliveryWake(f: { issueId: string; runId: string }) {
const record = (await pendingRecord(f.issueId))!;
return heartbeatService(db).wakeup(agentId, { source: "automation", triggerDetail: "system",
reason: "issue_recovery_action_restored", idempotencyKey: `execution-reconciliation:${record.id}`,
payload: { issueId: f.issueId, recoveryActionId: record.id }, requestedByActorType: "system", requestedByActorId: "execution-recovery",
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, recoveryActionId: record.id, previousRunId: f.runId,
retryOfRunId: f.runId, forceFreshSession: true, wakeReason: "issue_recovery_action_restored", source: "execution.reconciled" } });
}
it("lets a run that started and finished in the delivery window consume the continuation under the task lock", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
// Between the worker's pre-check and admission, saved input ran and finished.
const [intervening] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: f.issueId, nativeIssueId: f.issueId,
status: "succeeded", finishedAt: new Date(), contextSnapshot: { issueId: f.issueId, wakeReason: "issue_commented" } }).returning();
await occupySlot();
expect(await deliveryWake(f)).toBeNull();
expect(await successors(f.runId)).toHaveLength(0);
expect((await pendingRecord(f.issueId))!.evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: intervening!.id });
}, 60_000);
it("starts no run for a task that moved to review in the delivery window", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, f.issueId));
await occupySlot();
expect(await deliveryWake(f)).toBeNull();
expect(await successors(f.runId)).toHaveLength(0);
expect((await pendingRecord(f.issueId))!.evidence.continuationDelivery).toBe("invalidated");
}, 60_000);
it("settles coalesced holds with their primary, so a task back in todo is not shown as working and the sweep sees it", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted" });
// A second no-replay hold of the same run, settled together with the first.
const [sibling] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog",
status: "resolved", outcome: "blocked", ownerType: "board", cause: "native_restart_recovery_blocked",
fingerprint: `native-restart:${f.runId}`, nextAction: "held", resolvedAt: new Date(),
evidence: { runId: f.runId, automaticRecovery: { policy: "preserve_without_replay_v1", runId: f.runId, replay: "blocked" } } }).returning();
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, sibling!.id)))[0]!.evidence.continuationDelivery)
.toBe("coalesced");
// The task moves to review before delivery: the primary is cancelled, and so is its sibling.
await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, f.issueId));
await occupySlot();
expect(await deliveryWake(f)).toBeNull();
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, sibling!.id)))[0]!.evidence.continuationDelivery)
.toBe("invalidated");
// Back to todo with nothing pending: a failed attempt with Retry, not "working".
await db.update(issues).set({ status: "todo" }).where(eq(issues.id, f.issueId));
const task = await issue(f.issueId);
expect(await taskInterruptionPresentation(db, { id: task.id, companyId, status: task.status,
assigneeAgentId: task.assigneeAgentId, projectId: task.projectId })).toMatchObject({ state: "failed", canRetry: true });
// The assigned-work sweep sees it as left with nothing pending: inside its
// settle window after the recent change it schedules the recheck, never "ok".
const check = await heartbeatService(db).checkAssignedWorkInvariant({ companyId, issueId: f.issueId });
expect(["violation", "settling"]).toContain(check?.verdict.kind);
}, 60_000);
// The agent is told to check, redo what is missing, and ask only through the
// normal approval step; never that someone must restore the previous attempt.
it("guides the agent to check and redo, never to wait for an operator to recover the previous attempt", async () => {
for (const unverified of [[], ["github.merge_pull_request"]]) {
const note = freshContinuationNote({ runId: randomUUID(), unverified, workspace: "restored_last_save" });
if (unverified.length) expect(note).toContain("first check whether it took effect");
expect(note).toContain("do it again and continue the task");
// It does not contradict "do not repeat completed actions": it scopes itself to unknown outcomes.
expect(note).toContain("Completed actions with recorded results are still not repeated");
expect(note).toContain("request it through the normal approval step");
expect(note).not.toMatch(/operator|recover its result|execution state|authorize rerunning/i);
}
// The same holds for the continuation context the agent actually receives.
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted" });
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
const continuation = await buildExecutionContinuation({ db, companyId, issueId: f.issueId, agentId,
context: { previousRunId: f.runId }, summary: null, exposeLowTrustRaw: false });
const text = JSON.stringify(continuation);
expect(text).toContain("was cut off before it finished");
expect(text).not.toMatch(/operator must|recover its result|recover .*execution state|authorize rerunning/i);
}, 60_000);
// A task whose run completes normally gets no fallback activity at all, even
// with an older hold on the task (a run queued across a swap that then ran).
it("leaves a task that completes normally alone, even with an older run's hold", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation", hold: "settled_blocked" });
const [later] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: f.issueId, nativeIssueId: f.issueId,
status: "succeeded", startedAt: new Date(), finishedAt: new Date(), contextSnapshot: { issueId: f.issueId } }).returning();
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" })).toEqual({ kind: "successor_exists" });
expect(await ensureFreshContinuation(db, { companyId, runId: later!.id, trigger: "held_recovery" }))
.toEqual({ kind: "not_applicable", reason: "run_not_unfinished" });
await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId));
await settleUnrecoverableExecutions(db);
await occupySlot();
expect(await sweepWake(f.issueId)).toBeNull();
await deliver();
expect(await pendingRecord(f.issueId)).toBeUndefined();
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.issueId, f.issueId))).toHaveLength(2);
}, 60_000);
it("starts no run for a task that finished while its continuation was pending", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId));
await deliver();
expect(await successors(f.runId)).toHaveLength(0);
expect((await pendingRecord(f.issueId))!.evidence.continuationDelivery).toBe("invalidated");
}, 60_000);
// Crash between writes: the decision committed, the process died before any
// run was created. Nothing is lost: the next delivery pass starts it.
it("delivers a committed continuation after a crash before its run was created", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
expect(await successors(f.runId)).toHaveLength(0);
// A new process: a fresh service instance and worker pass.
await occupySlot();
await deliver();
expect(await successors(f.runId)).toHaveLength(1);
}, 60_000);
// The cut-off run's agent may still be alive in its old, unreachable sandbox.
// Once the continuation is recorded it can neither write to the task through
// the API nor publish its workspace, and the next run restores the last save
// into a new root on a new allocation.
it("fences the cut-off run: its API writes are refused and its workspace epoch is abandoned", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation",
error: "runner_remote_process_identity_unavailable" });
const workspaceId = randomUUID(), snapshotId = null, leaseId = randomUUID(), environmentId = randomUUID();
const location = { environmentId, leaseId, providerLeaseId: "sandbox-1", remoteCwd: "/home/daytona/.paperclip-workspaces/a" };
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: f.issueId, mode: "isolated",
strategyType: "remote", name: "Task workspace", providerType: "remote", providerRef: location.remoteCwd,
metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 3, snapshotId,
source: { kind: "empty" }, location, writerRunId: f.runId, dirty: true } } });
await db.update(heartbeatRuns).set({ runnerProfileJson: { remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1",
workspaceId, leaseId, providerLeaseId: location.providerLeaseId, remoteCwd: location.remoteCwd, epoch: 3,
state: "prepared", snapshotId } } }).where(eq(heartbeatRuns.id, f.runId));
// Today a failed run's token still writes.
const [before] = await loadAgentRunAuthentication(db, { runId: f.runId, companyId, agentId });
expect(agentRunWritesRevoked(before)).toBe(false);
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
// The auth middleware's own row read refuses every write of the old token.
const [after] = await loadAgentRunAuthentication(db, { runId: f.runId, companyId, agentId });
expect(agentRunWritesRevoked(after)).toBe(true);
await expect(db.transaction(tx => assertAgentRunWriteAllowed(tx as unknown as typeof db, companyId, { agentId, runId: f.runId })))
.rejects.toMatchObject({ status: 403, details: { code: "agent_run_superseded" } });
// The workspace writer is fenced out and the next preparation starts from the last save in a new root.
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
expect(workspace!.metadata!.remoteWorkspace).toMatchObject({ epoch: 4, writerRunId: null, dirty: false, location: null });
expect(workspace!.providerRef).toBeNull();
expect(workspace!.metadata!.remoteWorkspaceAbandonedEpochs).toEqual([expect.objectContaining({ epoch: 3, runId: f.runId, location })]);
// The continuation's own token is unaffected.
await occupySlot();
await deliver();
const [next] = await successors(f.runId);
await expect(db.transaction(tx => assertAgentRunWriteAllowed(tx as unknown as typeof db, companyId, { agentId, runId: next!.id })))
.resolves.toBeTruthy();
}, 60_000);
// The staging shape: the new server cannot re-reach the old Runner, so the old
// run's sandbox lease is still active (or its cleanup is pending) with no stop
// receipt. The continuation must still start, not wait behind that cleanup.
it.each(["active", "pending_cleanup"] as const)("starts the continuation while the unreachable old lease is %s", async leaseStatus => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation",
error: "runner_remote_process_identity_unavailable" });
const [environment] = await db.insert(environments).values({ name: `sandbox-${f.runId}`, driver: "sandbox" }).returning();
await db.insert(environmentLeases).values({ companyId, issueId: f.issueId, heartbeatRunId: f.runId,
environmentId: environment!.id, provider: "daytona", providerLeaseId: `sandbox-${f.runId}`, status: leaseStatus,
leasePolicy: "retain_on_failure", metadata: { remoteCwd: "/home/daytona/work", driver: "sandbox" } });
await settleUnrecoverableExecutions(db);
await deliver();
const [next] = await successors(f.runId);
expect(next).toBeTruthy();
// Admission claims it: it leaves the queue instead of being declined for the old run's cleanup.
await vi.waitFor(async () => {
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, next!.id));
expect(current!.status).not.toBe("queued");
}, { timeout: 20_000, interval: 100 });
// The abandoned allocation stays fenced: the old run still cannot write.
const [old] = await loadAgentRunAuthentication(db, { runId: f.runId, companyId, agentId });
expect(agentRunWritesRevoked(old)).toBe(true);
}, 60_000);
// A legacy run whose local process may still be alive: the continuation must
// never write the same directory. It gets a new isolated workspace and the
// decision records the stop as unproven.
it("gives the continuation a new isolated workspace while the old local writer may be alive", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost", error: "Legacy controller lease lost" });
// Its controller ran on another host (the replaced server), so no probe from here can prove it gone.
await db.update(heartbeatRuns).set({ processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000) })
.where(eq(heartbeatRuns.id, f.runId));
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace" });
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
expect(hold!.evidence).toMatchObject({ freshContinuation: { isolation: "fresh_isolated_workspace", stopProof: "unproven_isolated" },
executionReconciliation: { providerStopProof: "isolated" } });
await occupySlot();
await deliver();
const [next] = await successors(f.runId);
expect(next!.contextSnapshot).toMatchObject({ freshContinuationIsolation: f.actionId });
// Workspace selection gives it a new isolated workspace, never the old directory.
expect(await readFreshWorkspaceReadiness(db, next!)).toEqual({ mode: "fresh_isolated_workspace", retryIntentId: f.actionId,
sourceRunId: f.runId, retainedSourceIds: [] });
}, 60_000);
// The recorded process identity (host, PID space, start time) of this process.
const liveIdentity = async () => {
const startedAt = await observedProcessStartedAt(process.pid);
if (startedAt === null) throw new Error("This host cannot report this process's start time");
return { processPid: process.pid, processHostHash: currentProcessSpaceHash(), processStartedAt: new Date(startedAt) };
};
/** Moves the recorded start of the pending wait into the past. */
async function waitedFor(actionId: string, ms: number) {
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
'{freshContinuationCheck,firstPendingAt}', to_jsonb(${Date.now() - ms}::bigint))` }).where(eq(issueRecoveryActions.id, actionId));
}
it("lets a previous process still running on this host settle first, for about ten minutes at most", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "process_lost" });
await db.update(heartbeatRuns).set(await liveIdentity()).where(eq(heartbeatRuns.id, f.runId));
const first = await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" });
expect(first).toMatchObject({ kind: "exact_recovery_pending", reason: "previous_process_running" });
// Rechecks back off from one minute.
expect((first as { retryAt: Date }).retryAt.getTime()).toBeLessThanOrEqual(Date.now() + 61_000);
await recordFreshContinuationCheck(db, { companyId, actionId: f.actionId! }, first);
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
expect(hold!.evidence.freshContinuationCheck).toMatchObject({ checks: 1, firstPendingAt: expect.any(Number) });
// Past the bound it stops waiting and continues in a new isolated workspace.
await waitedFor(f.actionId!, FRESH_CONTINUATION_PENDING_BOUND_MS + 1_000);
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace" });
}, 60_000);
it("keeps waiting up to fifteen minutes while the exact-recovery owner shows progress", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost" });
await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 365 * 86_400_000) })
.where(eq(heartbeatRuns.id, f.runId));
const waiting = await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" });
await recordFreshContinuationCheck(db, { companyId, actionId: f.actionId! }, waiting);
await waitedFor(f.actionId!, FRESH_CONTINUATION_PENDING_BOUND_MS + 1_000);
// The owner renewed its lease since the last check: progress.
await db.update(heartbeatRuns).set({ lastOutputAt: new Date(Date.now() + 1_000) }).where(eq(heartbeatRuns.id, f.runId));
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "exact_recovery_pending", reason: "controller_lease_live" });
await waitedFor(f.actionId!, FRESH_CONTINUATION_PROGRESS_BOUND_MS + 1_000);
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace" });
}, 60_000);
it("does not mistake a recycled PID for the old writer", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "process_lost" });
// The PID is alive now, but it belongs to a process that started after the old one was recorded.
await db.update(heartbeatRuns).set({ ...(await liveIdentity()), processStartedAt: new Date("2020-01-01T00:00:00Z") })
.where(eq(heartbeatRuns.id, f.runId));
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued" });
}, 60_000);
/** A saved-work (workspace) recovery generation on the run's hold, as workspace recovery records it. */
async function savedWorkRecovery(f: { runId: string; actionId: string | null }, state: "awaiting_evidence" | "running") {
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
await db.update(issueRecoveryActions).set({ evidence: { ...hold!.evidence,
systemRecovery: { schema: "paperclip.system-recovery.v1", policyVersion: 1, phase: "reconcile_effects", state,
sourceRunId: f.runId, attempts: 1, retriesUsed: 0, revision: 2, failure: null, intentId: null, deadlineAt: null,
nextAttemptAt: null, firstFailureAt: null, freshAttemptRunId: null, providerCooldownAt: null,
attemptId: randomUUID(), generationId: randomUUID(), preservationOperationIds: [] } } })
.where(eq(issueRecoveryActions.id, f.actionId!));
}
it("bounds a saved-work recovery that is waiting with no progress, on a scheduled deadline", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "server_shutdown_interrupted", status: "interrupted" });
await savedWorkRecovery(f, "awaiting_evidence");
// The held-run scan selects it: exact recovery keeps its first chance.
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 0 });
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
expect(hold!.evidence.freshContinuationCheck).toMatchObject({ kind: "exact_recovery_pending", reason: "saved_work_recovery_running" });
// With no other event, the worker is scheduled for the next check.
expect(await nextFreshContinuationCheckAt(db)).toBe((hold!.evidence.freshContinuationCheck as { nextCheckAt: number }).nextCheckAt);
// No progress by the bound: the task continues fresh in a new root, and the
// preempted recovery is no longer scheduled (its hold carries the decision),
// so a late result cannot reach the new workspace.
await waitedFor(f.actionId!, FRESH_CONTINUATION_PENDING_BOUND_MS + 1_000);
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
'{freshContinuationCheck,nextCheckAt}', to_jsonb(${Date.now() - 1_000}::bigint))` }).where(eq(issueRecoveryActions.id, f.actionId!));
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 1 });
const [settled] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
expect(settled!.evidence).toMatchObject({ executionReconciliation: { runId: f.runId },
freshContinuation: { isolation: "fresh_isolated_workspace" } });
}, 60_000);
async function setupFailedAfterWork(state: "scheduled" | "complete" | "exhausted") {
const f = await unfinishedRun({ runtime: "native", errorCode: "setup_failed", hold: "settled_blocked" });
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
const recovery = { ...createSystemRecovery(f.runId), state, phase: state === "complete" ? "reconcile_effects" : "verify_execution",
...(state === "exhausted" ? { failure: { code: "export_interrupted", retryClass: "retry_transient", evidenceFingerprint: "x" } } : {}) };
await db.update(issueRecoveryActions).set({ ownerType: state === "exhausted" ? "board" : "system", evidence: { ...hold!.evidence,
workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [] },
systemRecovery: recovery } }).where(eq(issueRecoveryActions.id, f.actionId!));
return f;
}
it("continues a setup failure of a run that already did work once its saved work is recovered", async () => {
const f = await setupFailedAfterWork("complete");
expect(await heldExecutionsAwaitingFreshContinuation(db, new Date(), 10)).toEqual([expect.objectContaining({ id: f.actionId })]);
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 1 });
expect((await evidenceOf(f.actionId!)).evidence).toMatchObject({ freshContinuation: { sourceRunId: f.runId, failureCode: "setup_failed" } });
}, 60_000);
it("keeps a setup failure whose saved-work recovery is still under way with that recovery, then continues it on completion", async () => {
const f = await setupFailedAfterWork("scheduled");
expect(await continueHeldExecutions(db)).toMatchObject({ checked: 1, continued: 0 });
const waiting = (await evidenceOf(f.actionId!)).evidence.freshContinuationCheck as { kind: string; nextCheckAt: number };
expect(waiting).toMatchObject({ kind: "exact_recovery_pending" });
expect(waiting.nextCheckAt).toBeGreaterThan(Date.now());
expect(await runsAwaitingSystemContinuation(db, companyId, [f.issueId], [f.runId])).toEqual(new Set([`${f.issueId}:${f.runId}`]));
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence}, '{systemRecovery,state}', '"complete"')` })
.where(eq(issueRecoveryActions.id, f.actionId!));
await db.transaction(tx => expediteFreshContinuationChecks(tx as unknown as typeof db, { companyId, issueId: f.issueId, runId: f.runId }));
const due = (await evidenceOf(f.actionId!)).evidence.freshContinuationCheck as { nextCheckAt: number; firstPendingAt: number };
expect(due.nextCheckAt).toBeLessThanOrEqual(Date.now());
expect(due.firstPendingAt).toBe((waiting as unknown as { firstPendingAt: number }).firstPendingAt);
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 1 });
}, 60_000);
it("makes a recovery that completes while the backlog pass decides due at once, not at the backed-off check", async () => {
const f = await setupFailedAfterWork("scheduled");
let completion: Promise<unknown> | undefined;
await db.transaction(async tx => {
const decided = await continueFreshInTransaction(tx as unknown as typeof db, { companyId, runId: f.runId, trigger: "held_recovery" });
expect(decided).toMatchObject({ kind: "exact_recovery_pending" });
completion = db.transaction(async other => {
await other.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence}, '{systemRecovery,state}', '"complete"')` })
.where(eq(issueRecoveryActions.id, f.actionId!));
await expediteFreshContinuationChecks(other as unknown as typeof db, { companyId, issueId: f.issueId, runId: f.runId });
});
await new Promise(resolve => setTimeout(resolve, 200));
await recordFreshContinuationCheck(tx as unknown as typeof db, { companyId, actionId: f.actionId! }, decided);
});
await completion;
const check = (await evidenceOf(f.actionId!)).evidence.freshContinuationCheck as { nextCheckAt: number };
expect(check.nextCheckAt).toBeLessThanOrEqual(Date.now());
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 1 });
}, 60_000);
it("shows a task at the hourly continuation cap as scheduled, not as continuing", async () => {
const f = await setupFailedAfterWork("complete");
const nextCheckAt = Date.now() + 30 * 60_000;
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
freshContinuationCheck: { kind: "backoff", checks: 1, checkedAt: Date.now(), nextCheckAt } })}::jsonb` })
.where(eq(issueRecoveryActions.id, f.actionId!));
expect(await runsAwaitingSystemContinuation(db, companyId, [f.issueId], [f.runId])).toEqual(new Set());
expect(await taskInterruptionPresentation(db, await issue(f.issueId))).toMatchObject({ state: "retry_scheduled", canRetry: false,
nextRetryAt: new Date(nextCheckAt).toISOString() });
}, 60_000);
it("leaves a setup failure for a person once its saved-work recovery is exhausted", async () => {
const f = await setupFailedAfterWork("exhausted");
expect(await heldExecutionsAwaitingFreshContinuation(db, new Date(), 10)).toEqual([]);
expect(await runsAwaitingSystemContinuation(db, companyId, [f.issueId], [f.runId])).toEqual(new Set());
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 0 });
}, 60_000);
it("lets a running saved-work recovery that keeps advancing use its own budget", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "server_shutdown_interrupted", status: "interrupted" });
await savedWorkRecovery(f, "running");
const first = await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" });
await recordFreshContinuationCheck(db, { companyId, actionId: f.actionId! }, first);
await waitedFor(f.actionId!, FRESH_CONTINUATION_PROGRESS_BOUND_MS + 1_000);
// The recovery advanced (a new attempt) since the last check: it keeps going past 15 minutes.
await db.update(issueRecoveryActions).set({ updatedAt: new Date(Date.now() + 1_000),
evidence: sql`jsonb_set(${issueRecoveryActions.evidence}, '{systemRecovery,attempts}', '2'::jsonb)` })
.where(eq(issueRecoveryActions.id, f.actionId!));
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
.toMatchObject({ kind: "exact_recovery_pending", reason: "saved_work_recovery_running" });
// Its 30-minute budget is the cap.
await waitedFor(f.actionId!, FRESH_CONTINUATION_SAVED_WORK_BOUND_MS + 1_000);
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace" });
}, 60_000);
it("bounds every exact-recovery wait, here a controller lease that never lapses", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost" });
await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 365 * 86_400_000) })
.where(eq(heartbeatRuns.id, f.runId));
const waiting = await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" });
expect(waiting).toMatchObject({ kind: "exact_recovery_pending", reason: "controller_lease_live" });
await recordFreshContinuationCheck(db, { companyId, actionId: f.actionId! }, waiting);
await waitedFor(f.actionId!, FRESH_CONTINUATION_PENDING_BOUND_MS + 1_000);
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace" });
}, 60_000);
it("keeps the task workspace when the old local writer is proven gone", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "process_lost" });
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", isolation: "writer_ended" });
}, 60_000);
it("settles only the reconciliation holds it owns and leaves restore holds and history alone", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted" });
const [history] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog",
status: "cancelled", outcome: "cancelled", ownerType: "board", cause: "native_restart_recovery_blocked",
fingerprint: `history:${f.runId}`, nextAction: "Old note", resolutionNote: "kept", resolvedAt: new Date(),
evidence: { runId: f.runId } }).returning();
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
const [untouched] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, history!.id));
expect(untouched).toMatchObject({ status: "cancelled", nextAction: "Old note", resolutionNote: "kept", ownerType: "board" });
expect(untouched!.evidence).toEqual({ runId: f.runId });
// A restore hold that workspace recovery has not finished keeps the task with it.
const g = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted" });
const [restore] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: g.issueId, kind: "active_run_watchdog",
status: "resolved", outcome: "blocked", ownerType: "system", cause: "native_continuation_requires_reconciliation",
fingerprint: `native-workspace:${g.runId}`, nextAction: "Recovering saved work", resolvedAt: new Date(),
evidence: { runId: g.runId, workspaceRestoreFailure: "restore_unsafe_archive",
automaticRecovery: { policy: "preserve_without_replay_v1", runId: g.runId, replay: "blocked" } } }).returning();
expect(await ensureFreshContinuation(db, { companyId, runId: g.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "exact_recovery_pending", reason: "workspace_recovery_owns_restore" });
const [kept] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, restore!.id));
expect(kept!.evidence).toMatchObject({ workspaceRestoreFailure: "restore_unsafe_archive", automaticRecovery: { replay: "blocked" } });
expect(kept!.evidence.executionReconciliation).toBeUndefined();
}, 60_000);
it("schedules no recheck for a held task nothing would act on", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted", hold: "settled_blocked" });
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
freshContinuationCheck: { kind: "person_owned", reason: "approval_pending", checkedAt: Date.now() - 7_200_000,
nextCheckAt: Date.now() - 3_600_000 } })}::jsonb` }).where(eq(issueRecoveryActions.id, f.actionId!));
expect(await nextFreshContinuationCheckAt(db)).not.toBeNull();
// The task was cancelled after that check: the schedule goes quiet.
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId));
expect(await nextFreshContinuationCheckAt(db)).toBeNull();
expect(await heldExecutionsAwaitingFreshContinuation(db, new Date(), 10)).toEqual([]);
expect(await executionControlSchedule(db).disposition()).toBeNull();
}, 60_000);
it("schedules no disposition scan for a held record the backlog pass leaves alone, and still does for an eligible one", async () => {
const overdueCheck = (actionId: string) => db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
freshContinuationCheck: { kind: "backoff", checkedAt: Date.now() - 7_200_000, nextCheckAt: Date.now() - 3_600_000 } })}::jsonb` })
.where(eq(issueRecoveryActions.id, actionId));
const old = await heldBacklog("native_restart_recovery_blocked", 3 * 24 * 60 * 60_000);
await overdueCheck(old.actionId!);
expect(await nextFreshContinuationCheckAt(db)).toBeNull();
expect(await executionControlSchedule(db).disposition()).toBeNull();
const recent = await heldBacklog("native_restart_recovery_blocked", 60 * 60_000);
await overdueCheck(recent.actionId!);
expect(await nextFreshContinuationCheckAt(db)).not.toBeNull();
}, 60_000);
it("tells the agent the previous attempt was cut off and which actions are unverified", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_session_interrupted" });
await db.insert(toolInvocations).values({ companyId, agentId, issueId: f.issueId, runId: f.runId, toolName: "github.create_pull_request",
riskLevel: "write", status: "executing", startedAt: new Date() } as typeof toolInvocations.$inferInsert);
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
const decision = hold!.evidence.executionReconciliation as Record<string, string>;
expect(decision.outcomeEvidence).toContain(`run ${f.runId}) was cut off`);
expect(decision.outcomeEvidence).toContain("github.create_pull_request");
expect(decision.outcomeEvidence).not.toMatch(adminText);
}, 60_000);
it("names the subtasks, comments and documents the cut-off run already created, so the fresh run does not repeat them", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_restart_recovery_blocked" });
for (const title of ["SM sub 1 b3d70e77s1", "SM sub 2 b3d70e77s2"])
await db.insert(issues).values({ companyId, title, status: "todo", assigneeAgentId: agentId, parentId: f.issueId,
originRunId: f.runId } as typeof issues.$inferInsert);
await db.insert(issueComments).values({ companyId, issueId: f.issueId, authorAgentId: agentId, createdByRunId: f.runId,
body: "DONE-0d981334" } as typeof issueComments.$inferInsert);
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
const decision = hold!.evidence.executionReconciliation as Record<string, string>;
expect(decision.outcomeEvidence).toContain('"SM sub 1 b3d70e77s1"');
expect(decision.outcomeEvidence).toContain('"SM sub 2 b3d70e77s2"');
expect(decision.outcomeEvidence).toContain('"DONE-0d981334"');
expect(decision.outcomeEvidence).toContain("Do not create or post them again");
expect(hold!.evidence.freshContinuation).toMatchObject({ createdObjects: { subtasks: expect.arrayContaining([expect.stringContaining("SM sub 1")]) } });
}, 60_000);
it("names the follow-up the cut-off run had read and its own final comment, so the agent answers once (t003)", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_restart_recovery_blocked" });
const [followUp] = await db.insert(issueComments).values({ companyId, issueId: f.issueId, authorType: "user",
authorUserId: "board-user", body: "Additional requirement: include the line FOLLOWUP-0d981334" } as typeof issueComments.$inferInsert).returning();
await db.insert(heartbeatRunEvents).values({ companyId, runId: f.runId, agentId, seq: 1, eventType: "task_history.read",
stream: "system", level: "info", payload: { comments: { [followUp!.id]: followUp!.updatedAt.toISOString() } },
} as typeof heartbeatRunEvents.$inferInsert);
await db.insert(issueComments).values({ companyId, issueId: f.issueId, authorAgentId: agentId, createdByRunId: f.runId,
body: "DONE-0d981334" } as typeof issueComments.$inferInsert);
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
const note = (hold!.evidence.executionReconciliation as Record<string, string>).outcomeEvidence;
expect(note).toContain("had already read these messages");
expect(note).toContain("FOLLOWUP-0d981334");
expect(note).toContain('posted comments "DONE-0d981334"');
expect(note.indexOf("had already read")).toBeLessThan(note.indexOf("DONE-0d981334"));
}, 60_000);
it("does not list a follow-up edited after the cut-off run read it as already read; it is marked as changed", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_restart_recovery_blocked" });
const [followUp] = await db.insert(issueComments).values({ companyId, issueId: f.issueId, authorType: "user",
authorUserId: "board-user", body: "Additional requirement: include the line FOLLOWUP-OLD" } as typeof issueComments.$inferInsert).returning();
await db.insert(heartbeatRunEvents).values({ companyId, runId: f.runId, agentId, seq: 1, eventType: "task_history.read",
stream: "system", level: "info", payload: { comments: { [followUp!.id]: followUp!.updatedAt.toISOString() } },
} as typeof heartbeatRunEvents.$inferInsert);
await db.update(issueComments).set({ body: "Additional requirement: include the line FOLLOWUP-NEW",
updatedAt: new Date(followUp!.updatedAt.getTime() + 60_000) }).where(eq(issueComments.id, followUp!.id));
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" })).toMatchObject({ kind: "continued" });
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
const note = (hold!.evidence.executionReconciliation as Record<string, string>).outcomeEvidence;
expect(note).not.toContain("had already read");
expect(note).toContain("edited these messages after that run read them");
expect(note).toContain("FOLLOWUP-NEW");
}, 60_000);
it.each([true, false])("relies on saved input only while it still holds an undelivered message (already taken by the cut-off run: %s)", async taken => {
// R2 cell t003: the cut-off run had taken the follow-up into its turn; the fallback chose saved_input,
// promotion settled the message as delivered, and no run followed.
const f = await unfinishedRun({ runtime: "native", errorCode: "native_restart_recovery_blocked" });
const [comment] = await db.insert(issueComments).values({ companyId, issueId: f.issueId, authorType: "user",
authorUserId: userId, body: "Additional requirement: include FOLLOWUP-1" } as typeof issueComments.$inferInsert).returning();
if (taken) await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, wakeCommentIds: [comment!.id] } })
.where(eq(heartbeatRuns.id, f.runId));
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", triggerDetail: "system",
reason: "issue_execution_deferred", status: "deferred_issue_execution", requestedByActorType: "user", requestedByActorId: userId,
payload: { issueId: f.issueId, _paperclipWakeContext: { issueId: f.issueId, wakeReason: "issue_commented", wakeCommentIds: [comment!.id] } },
} as typeof agentWakeupRequests.$inferInsert);
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", delivery: taken ? "recorded" : "saved_input" });
}, 60_000);
it.each(["native", "legacy"] as const)("starts a run for the person's message once the task continues from saved work (%s)", async runtime => {
const f = await unfinishedRun({ runtime, errorCode: runtime === "native" ? "native_provider_terminal_failed" : "process_lost" });
const [comment] = await db.insert(issueComments).values({ companyId, issueId: f.issueId, authorType: "user",
authorUserId: userId, body: "try again" } as typeof issueComments.$inferInsert).returning();
const heartbeat = heartbeatService(db);
await heartbeat.wakeup(agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented",
payload: { issueId: f.issueId, commentId: comment!.id, mutation: "comment" },
requestedByActorType: "user", requestedByActorId: userId,
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: comment!.id, wakeCommentId: comment!.id,
source: "issue.comment", wakeReason: "issue_commented" } });
const wakes = () => db.select().from(agentWakeupRequests).where(sql`${agentWakeupRequests.payload}->>'issueId' = ${f.issueId}`);
expect((await wakes()).map(w => w.status)).toEqual(["deferred_issue_execution"]);
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", delivery: "saved_input" });
await occupySlot();
await deliver();
await heartbeat.resumeQueuedRuns();
const runs = await db.select().from(heartbeatRuns).where(and(
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${f.issueId}`, sql`${heartbeatRuns.id} <> ${f.runId}`));
const saved = await wakes();
expect(runs, JSON.stringify({ issue: await issue(f.issueId),
wakes: saved.map(w => ({ status: w.status, error: w.error, runId: w.runId, payload: w.payload })) })).toHaveLength(1);
expect(runs[0]!.contextSnapshot?.wakeCommentIds).toContain(comment!.id);
expect(saved).toHaveLength(1);
expect(saved[0]!.runId).toBe(runs[0]!.id);
}, 60_000);
it.each([
{ title: "a pending approval", setup: "approval" as const, errorCode: "native_continuation_requires_reconciliation" },
{ title: "a credential the person has not shared", setup: "none" as const, errorCode: "configuration_incomplete" },
{ title: "a budget stop", setup: "none" as const, errorCode: "budget_blocked" },
])("leaves a task waiting on $title for the person", async ({ setup, errorCode }) => {
expect(PERSON_OWNED_FAILURE_CODES).toContain("configuration_incomplete");
const f = await unfinishedRun({ runtime: "native", errorCode });
if (setup === "approval") {
const [approval] = await db.insert(approvals).values({ companyId, type: "approve_ceo_strategy", status: "pending",
payload: {} } as typeof approvals.$inferInsert).returning();
await db.insert(issueApprovals).values({ companyId, issueId: f.issueId, approvalId: approval!.id } as typeof issueApprovals.$inferInsert);
}
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "person_owned" });
await settleUnrecoverableExecutions(db);
expect(await issue(f.issueId)).toMatchObject({ status: "blocked" });
await deliver();
expect(await successors(f.runId)).toHaveLength(0);
}, 60_000);
it("bounds continuations with the server-restart lane's hourly cap", () => {
expect(FRESH_CONTINUATIONS_PER_HOUR).toBe(SERVER_RESTART_RETRY_MAX_ATTEMPTS);
expect(FRESH_CONTINUATION_WINDOW_MS).toBe(SERVER_RESTART_RETRY_WINDOW_MS);
});
// Governance gates are never retired or bypassed by the fallback.
it.each(["question_pending", "user_owned_hold", "operator_continuation_recorded", "owner_gone_attestation"] as const)(
"leaves a task behind a real governance gate (%s) for the person", async gate => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation" });
if (gate === "question_pending") await db.insert(issueThreadInteractions).values({ companyId, issueId: f.issueId,
kind: "ask_user_questions", status: "pending", payload: { questions: [] } } as unknown as typeof issueThreadInteractions.$inferInsert);
if (gate === "user_owned_hold") await db.update(issueRecoveryActions).set({ ownerType: "user", ownerUserId: userId })
.where(eq(issueRecoveryActions.id, f.actionId!));
if (gate === "operator_continuation_recorded" || gate === "owner_gone_attestation") {
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
await db.update(issueRecoveryActions).set({ evidence: { ...hold!.evidence,
...(gate === "owner_gone_attestation" ? { ownerGoneAttestation: { actorId: userId } } : { explicitUserContinuation: { actorId: userId } }) } })
.where(eq(issueRecoveryActions.id, f.actionId!));
}
const before = (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!)))[0]!;
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "person_owned" });
const after = (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!)))[0]!;
expect(after.evidence).toEqual(before.evidence);
expect(after.status).toBe(before.status);
await deliver();
expect(await successors(f.runId)).toHaveLength(0);
}, 60_000);
it.each([false, true])("respects deferred question authority in fresh continuation (governed=%s)", async governed => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation" });
await db.insert(issueThreadInteractions).values({ companyId, issueId: f.issueId, kind: "ask_user_questions",
status: "pending", dismissedAt: new Date(), payload: { version: 1, questions: [],
...(governed ? { toolAction: {} } : {}) } } as never);
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject(governed ? { kind: "person_owned", reason: "question_pending" } : { kind: "continued" });
}, 60_000);
// Interruption with an unknown outcome is not a governance gate: the task
// continues, but as a fresh run, never a replay, and with an audit trail.
it("continues an interrupted governed action as a fresh run, never a replay, and keeps the audit trail", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "native_continuation_requires_reconciliation",
error: "runner_remote_process_identity_unavailable", hold: "settled_blocked" });
const [inFlight] = await db.insert(toolInvocations).values({ companyId, agentId, issueId: f.issueId, runId: f.runId,
toolName: "github.merge_pull_request", riskLevel: "write", status: "executing", startedAt: new Date() } as typeof toolInvocations.$inferInsert)
.returning();
const original = (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!)))[0]!.evidence.automaticRecovery;
await settleUnrecoverableExecutions(db);
const next = await expectOneFreshSuccessor(f);
// A fresh session: the interrupted turn is not re-sent.
expect(next.contextSnapshot).toMatchObject({ forceFreshSession: true });
expect(next.sessionIdBefore).toBeNull();
// The cut-off tool call is not re-executed: it stays as recorded and no new invocation exists.
const invocations = await db.select().from(toolInvocations).where(eq(toolInvocations.issueId, f.issueId));
expect(invocations).toEqual([expect.objectContaining({ id: inFlight!.id, runId: f.runId, status: "executing" })]);
// The earlier no-replay decision stays as history beside the attributed fallback decision.
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
expect(hold!.evidence.automaticRecoveryHistory).toEqual(original);
expect(hold!.evidence.freshContinuationSettlement).toMatchObject({ actorType: "system", actorId: "fresh-continuation",
decision: "continue_in_fresh_run", replay: "none", reason: "exact_resumption_unavailable",
unverifiedActions: ["github.merge_pull_request"], recordedAt: expect.any(String) });
expect((hold!.evidence.executionReconciliation as Record<string, string>).outcomeEvidence).toContain("github.merge_pull_request");
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, f.issueId),
eq(activityLog.action, "issue.continued_from_saved_work")))).toHaveLength(1);
}, 60_000);
// The isolation decision is a property of the task's next run, whatever path starts it.
it("moves the task off the old writer's workspace and runs the isolated continuation before saved input", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost", error: "Legacy controller lease lost" });
const [workspace] = await db.insert(executionWorkspaces).values({ companyId, sourceIssueId: f.issueId, mode: "isolated",
strategyType: "git_worktree", name: "Task workspace", cwd: "/tmp/old-writer" }).returning();
await db.update(issues).set({ executionWorkspaceId: workspace!.id, workspaceBindingRevision: 2 }).where(eq(issues.id, f.issueId));
await db.update(heartbeatRuns).set({ processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000),
contextSnapshot: { issueId: f.issueId, executionWorkspaceId: workspace!.id } }).where(eq(heartbeatRuns.id, f.runId));
// A person's saved message is waiting. It carries no isolation marker, so it cannot prepare the
// new workspace (a projectless task would be refused at admission once the binding is cleared):
// the isolated continuation runs first and the saved message follows it.
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", status: "deferred_issue_execution",
requestedByActorType: "user", requestedByActorId: userId, payload: { issueId: f.issueId } });
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace", delivery: "recorded" });
// No delivery path can reach the old directory: the task is no longer bound to it.
expect(await issue(f.issueId)).toMatchObject({ executionWorkspaceId: null, workspaceBindingRevision: 3 });
const [kept] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id));
expect(kept!.cwd).toBe("/tmp/old-writer");
}, 60_000);
it("records its own isolated continuation when saved input could not avoid a shared directory", async () => {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "controller_lease_lost" });
await db.update(heartbeatRuns).set({ processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000) })
.where(eq(heartbeatRuns.id, f.runId));
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", status: "deferred_issue_execution",
requestedByActorType: "user", requestedByActorId: userId, payload: { issueId: f.issueId } });
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", isolation: "fresh_isolated_workspace", delivery: "recorded" });
}, 60_000);
it("backs off instead of blocking once a task used its hourly continuations", async () => {
const f = await unfinishedRun({ runtime: "native", errorCode: "process_lost" });
for (let index = 0; index < FRESH_CONTINUATIONS_PER_HOUR; index++) {
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog", status: "resolved",
outcome: "restored", ownerType: "system", cause: "fresh_continuation", fingerprint: `fresh-continuation:${randomUUID()}`,
nextAction: "continued", evidence: { runId: randomUUID(), continuationDelivery: "delivered",
freshContinuation: { schema: FRESH_CONTINUATION_SCHEMA, sourceRunId: randomUUID(),
recordedAt: new Date(Date.now() - (index + 1) * 60_000).toISOString() } } });
}
await settleUnrecoverableExecutions(db);
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
// Not blocked for a person: the hold stays with the system and is checked again when the window frees a slot.
expect(hold).toMatchObject({ status: "active" });
expect(hold!.evidence.freshContinuationCheck).toMatchObject({ kind: "backoff", nextCheckAt: expect.any(Number) });
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 0 });
// With no other event, the disposition worker is scheduled for exactly the
// moment the window has room again.
const retryAt = (hold!.evidence.freshContinuationCheck as { nextCheckAt: number }).nextCheckAt;
expect(retryAt).toBeGreaterThan(Date.now());
expect(await nextFreshContinuationCheckAt(db)).toBe(retryAt);
expect(await executionControlSchedule(db).disposition()).toBe(retryAt);
// The hour passes (the window's records age out and the check falls due);
// the scheduled pass alone continues the task.
await db.execute(sql`update issue_recovery_actions set evidence = jsonb_set(evidence, '{freshContinuation,recordedAt}',
to_jsonb((now() - interval '2 hours')::text)) where evidence ? 'freshContinuation'`);
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
'{freshContinuationCheck,nextCheckAt}', to_jsonb(${Date.now() - 1_000}::bigint))` }).where(eq(issueRecoveryActions.id, f.actionId!));
expect(await executionControlSchedule(db).disposition()).toBeLessThanOrEqual(Date.now() + 60_000);
await settleUnrecoverableExecutions(db);
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
.toMatchObject({ kind: "already_continued" });
expect(await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests)).toHaveLength(0);
}, 60_000);
async function heldByOlderRun(options: { endedAgoMs?: number; recovery?: "exhausted" | "running"; humanGate?: boolean } = {}) {
const f = await unfinishedRun({ runtime: "legacy", errorCode: "process_lost" });
const endedAt = new Date(Date.now() - (options.endedAgoMs ?? 60_000));
await db.update(heartbeatRuns).set({ finishedAt: endedAt, startedAt: new Date(endedAt.getTime() - 600_000),
createdAt: new Date(endedAt.getTime() - 660_000), processPid: 4242, processHostHash: "replaced-server",
processStartedAt: new Date(endedAt.getTime() - 600_000) }).where(eq(heartbeatRuns.id, f.runId));
const [workspace] = await db.insert(executionWorkspaces).values({ companyId, sourceIssueId: f.issueId, mode: "isolated",
strategyType: "git_worktree", name: "Task workspace", providerType: "local_fs", cwd: "/workspaces/old-task-dir" }).returning();
await db.update(issues).set({ executionWorkspaceId: workspace!.id }).where(eq(issues.id, f.issueId));
const recovery = { ...createSystemRecovery(f.runId), state: options.recovery ?? "exhausted", attempts: 3,
failure: { code: "export_interrupted", retryClass: "retry_transient", evidenceFingerprint: "export" } };
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId!));
await db.update(issueRecoveryActions).set({ ownerUserId: options.humanGate ? userId : null,
createdAt: endedAt, updatedAt: endedAt,
evidence: { ...hold!.evidence, workspaceRestoreFailure: "restore_unsafe_archive", systemRecovery: recovery } })
.where(eq(issueRecoveryActions.id, f.actionId!));
return { ...f, workspaceId: workspace!.id, endedAt };
}
async function refusedAtDispatch(f: { issueId: string }, options: { createdAt?: Date; wakeReason?: string } = {}) {
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", issueId: f.issueId,
runtimeMode: "legacy", status: "queued", createdAt: options.createdAt ?? new Date(),
contextSnapshot: { issueId: f.issueId, wakeReason: options.wakeReason ?? "issue_continuation_needed" } }).returning();
const outcome = await createPostgresRunDispatchAdapter(db).cancelStaleQueuedRun({ companyId, runId: run!.id,
expectedStatus: "queued", now: options.createdAt ?? new Date() });
return { run: run!, outcome };
}
const holdRow = async (id: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!;
const makeDue = (id: string) => db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
'{freshContinuationCheck,nextCheckAt}', to_jsonb(${Date.now() - 1_000}::bigint))` }).where(eq(issueRecoveryActions.id, id));
it("continues a task held by an older run's saved-work hold in a new isolated workspace, after it refused a run at dispatch", async () => {
const f = await heldByOlderRun();
const refused = await refusedAtDispatch(f);
expect(refused.outcome).toMatchObject({ outcome: "cancelled", errorCode: "execution_reconciliation_required" });
expect((await holdRow(f.actionId!)).evidence.freshContinuationCheck)
.toMatchObject({ kind: "exact_recovery_pending", firstPendingAt: expect.any(Number) });
expect(await heldExecutionsAwaitingFreshContinuation(db, new Date(Date.now() + 10 * 60_000), 10))
.toEqual([expect.objectContaining({ id: f.actionId, runId: f.runId })]);
await waitedFor(f.actionId!, FRESH_CONTINUATION_PENDING_BOUND_MS + 1_000);
await makeDue(f.actionId!);
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 1 });
const settled = await holdRow(f.actionId!);
expect(settled).toMatchObject({ status: "resolved", ownerType: "system" });
expect(settled.evidence).toMatchObject({ workspaceRestoreFailure: "restore_unsafe_archive",
systemRecovery: { state: "exhausted" },
freshContinuation: { sourceRunId: f.runId, isolation: "fresh_isolated_workspace" },
freshContinuationSettlement: { actorType: "system", actorId: "fresh-continuation", heldFiles: "kept_not_applied",
refusedAtDispatchRunId: refused.run.id, requestedBy: "system" } });
expect((await issue(f.issueId)).executionWorkspaceId).toBeNull();
const [old] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId));
expect(old).toMatchObject({ status: "active", cwd: "/workspaces/old-task-dir" });
const [oldRun] = await loadAgentRunAuthentication(db, { runId: f.runId, companyId, agentId });
expect(agentRunWritesRevoked(oldRun)).toBe(true);
expect(await getExecutionBlocker(db, companyId, f.issueId)).toBeNull();
await occupySlot();
await deliver();
const [next] = await successors(f.runId);
expect(next!.contextSnapshot).toMatchObject({ freshContinuationIsolation: f.actionId });
expect(await readFreshWorkspaceReadiness(db, next!)).toMatchObject({ mode: "fresh_isolated_workspace", sourceRunId: f.runId });
expect(await createPostgresRunDispatchAdapter(db).cancelStaleQueuedRun({ companyId, runId: next!.id,
expectedStatus: "queued", now: new Date() })).not.toMatchObject({ errorCode: "execution_reconciliation_required" });
}, 60_000);
it("keeps the wait of a saved-work recovery that is still advancing, even behind a run it refused", async () => {
const f = await heldByOlderRun({ recovery: "running" });
await refusedAtDispatch(f);
await waitedFor(f.actionId!, FRESH_CONTINUATION_PROGRESS_BOUND_MS + 1_000);
await db.update(issueRecoveryActions).set({ updatedAt: new Date(Date.now() + 1_000),
evidence: sql`jsonb_set(${issueRecoveryActions.evidence}, '{systemRecovery,attempts}', '4'::jsonb)` })
.where(eq(issueRecoveryActions.id, f.actionId!));
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
.toMatchObject({ kind: "exact_recovery_pending", reason: "saved_work_recovery_running" });
expect((await issue(f.issueId)).executionWorkspaceId).toBe(f.workspaceId);
await refusedAtDispatch(f, { wakeReason: "task_retry" });
await db.update(issueRecoveryActions).set({ updatedAt: new Date(Date.now() + 2_000),
evidence: sql`jsonb_set(${issueRecoveryActions.evidence}, '{systemRecovery,attempts}', '5'::jsonb)` })
.where(eq(issueRecoveryActions.id, f.actionId!));
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
.toMatchObject({ kind: "exact_recovery_pending" });
}, 60_000);
it("keeps a hold a person owns blocking, behind any number of refused runs", async () => {
const f = await heldByOlderRun({ humanGate: true });
await refusedAtDispatch(f, { wakeReason: "task_retry" });
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
.toMatchObject({ kind: "person_owned" });
expect(await getExecutionBlocker(db, companyId, f.issueId)).toMatchObject({ recoveryActionId: f.actionId });
expect((await holdRow(f.actionId!)).status).toBe("active");
}, 60_000);
it("leaves an old hold for a person's Retry, and that Retry runs isolated (refused run)", async () => {
const f = await heldByOlderRun({ endedAgoMs: 3 * 86_400_000 });
await refusedAtDispatch(f, { createdAt: new Date(Date.now() - 30 * 3_600_000) });
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 0 });
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "continuation_wake" }))
.toEqual({ kind: "person_owned", reason: "retry_needed" });
expect((await holdRow(f.actionId!)).status).toBe("active");
const retry = await refusedAtDispatch(f, { wakeReason: "task_retry" });
expect(retry.outcome).toMatchObject({ outcome: "cancelled" });
expect((await holdRow(f.actionId!)).evidence).toMatchObject({
freshContinuation: { isolation: "fresh_isolated_workspace" },
freshContinuationSettlement: { requestedBy: "person_retry", heldFiles: "kept_not_applied" } });
await occupySlot();
await deliver();
const [next] = await successors(f.runId);
expect(await readFreshWorkspaceReadiness(db, next!)).toMatchObject({ mode: "fresh_isolated_workspace" });
}, 60_000);
it("leaves an old hold for a person's Retry, and that Retry runs isolated (waiting Retry)", async () => {
const f = await heldByOlderRun({ endedAgoMs: 3 * 86_400_000 });
const old = await refusedAtDispatch(f, { createdAt: new Date(Date.now() - 30 * 3_600_000) });
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 0 });
const receipt = await requestTaskRetry({ db, companyId, issueId: f.issueId, actorId: userId,
idempotencyKey: randomUUID(), observedRunId: old.run.id });
expect(receipt).toMatchObject({ accepted: true });
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 1 });
expect((await holdRow(f.actionId!)).evidence).toMatchObject({ freshContinuation: { isolation: "fresh_isolated_workspace" },
freshContinuationSettlement: { requestedBy: "person_retry" } });
await occupySlot();
await deliver();
expect(await successors(f.runId)).toHaveLength(1);
const [retryWake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
expect(retryWake!.status).toBe("coalesced");
}, 60_000);
async function preparedRemoteRunStoppedBeforeDispatch(options: { endedAgoMs?: number } = {}) {
const issueId = randomUUID(), runId = randomUUID();
const endedAt = new Date(Date.now() - (options.endedAgoMs ?? 60_000));
await db.insert(issues).values({ id: issueId, companyId, title: "Ship the release notes", status: "in_progress", assigneeAgentId: agentId });
const [workspace] = await db.insert(executionWorkspaces).values({ companyId, sourceIssueId: issueId, mode: "isolated",
strategyType: "git_worktree", name: "Task workspace", cwd: "/work/remote-task" }).returning();
await db.update(issues).set({ executionWorkspaceId: workspace!.id }).where(eq(issues.id, issueId));
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "assignment", issueId,
runtimeMode: "legacy", status: "running", startedAt: new Date(endedAt.getTime() - 120_000),
createdAt: new Date(endedAt.getTime() - 180_000), responsibleUserId: userId,
contextSnapshot: { issueId, executionWorkspaceId: workspace!.id, wakeReason: "issue_assigned" } }).returning();
const [environment] = await db.insert(environments).values({ name: `sandbox-${runId}`, driver: "sandbox" }).returning();
const [lease] = await db.insert(environmentLeases).values({ companyId, issueId, executionWorkspaceId: workspace!.id,
heartbeatRunId: runId, environmentId: environment!.id, status: "active", leasePolicy: "ephemeral", provider: "daytona",
providerLeaseId: `sandbox-${runId}`, metadata: { remoteCwd: "/work/remote-task", driver: "sandbox",
workspaceRealization: { authoritativeRoot: "/work/remote-task" } } }).returning();
await terminalizeLegacyExecution({ db, run: run!, status: "interrupted", patch: { errorCode: "server_shutdown_interrupted",
error: "Interrupted by graceful server shutdown (SIGTERM)", finishedAt: endedAt,
resultJson: { workspaceRestoreFailure: "restore_failed" } } });
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
return { issueId, runId, actionId: hold!.id, leaseId: lease!.id, workspaceId: workspace!.id };
}
it("continues a recent restart-stopped run whose prepared remote workspace could not be restored, in a new isolated workspace", async () => {
const f = await preparedRemoteRunStoppedBeforeDispatch();
const held = await holdRow(f.actionId);
await settleUnrecoverableExecutions(db);
await waitedFor(f.actionId, FRESH_CONTINUATION_PENDING_BOUND_MS + 1_000).catch(() => undefined);
if ((await holdRow(f.actionId)).evidence.freshContinuationCheck) await makeDue(f.actionId);
await settleUnrecoverableExecutions(db);
const settled = await holdRow(f.actionId);
expect(settled.evidence).toMatchObject({ freshContinuation: { sourceRunId: f.runId, isolation: "fresh_isolated_workspace" },
workspaceRestoreRecovery: { leaseIds: [f.leaseId] }, freshContinuationSettlement: { heldFiles: "kept_not_applied" } });
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId));
expect(lease!.leasePolicy).toBe("retain_on_failure");
expect((await issue(f.issueId)).executionWorkspaceId).toBeNull();
expect(await getExecutionBlocker(db, companyId, f.issueId)).toBeNull();
await occupySlot();
await deliver();
const [next] = await successors(f.runId);
expect(await readFreshWorkspaceReadiness(db, next!)).toMatchObject({ mode: "fresh_isolated_workspace", sourceRunId: f.runId });
expect(held.nextAction).not.toMatch(/stop receipt|recover the missing files|workspaceRepairEvidence|allocation|repair|inspect|_/i);
expect(held.nextAction).toContain("Retry");
}, 60_000);
it("leaves an older restart-stopped remote restore failure for a person's Retry, which then runs isolated", async () => {
const f = await preparedRemoteRunStoppedBeforeDispatch({ endedAgoMs: 30 * 3_600_000 });
await settleUnrecoverableExecutions(db);
await settleUnrecoverableExecutions(db);
const kept = await holdRow(f.actionId);
expect(kept.evidence.freshContinuation).toBeUndefined();
expect(await getExecutionBlocker(db, companyId, f.issueId)).toMatchObject({ recoveryActionId: f.actionId });
await refusedAtDispatch(f, { wakeReason: "task_retry" });
expect((await holdRow(f.actionId)).evidence).toMatchObject({ freshContinuation: { isolation: "fresh_isolated_workspace" },
freshContinuationSettlement: { requestedBy: "person_retry" } });
await occupySlot();
await deliver();
const [next] = await successors(f.runId);
expect(await readFreshWorkspaceReadiness(db, next!)).toMatchObject({ mode: "fresh_isolated_workspace" });
}, 60_000);
async function followUpFailedBeforeProviderWork(options: { restoreFailed?: boolean } = {}) {
const issueId = randomUUID(), runId = randomUUID();
const endedAt = new Date(Date.now() - 60_000);
await db.insert(issues).values({ id: issueId, companyId, title: "Summarize the subtasks", status: "in_progress", assigneeAgentId: agentId });
const [workspace] = await db.insert(executionWorkspaces).values({ companyId, sourceIssueId: issueId, mode: "isolated",
strategyType: "git_worktree", name: "Task workspace", cwd: "/work/remote-task" }).returning();
await db.update(issues).set({ executionWorkspaceId: workspace!.id }).where(eq(issues.id, issueId));
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "automation", issueId,
runtimeMode: "legacy", status: "running", startedAt: new Date(endedAt.getTime() - 10_000),
createdAt: new Date(endedAt.getTime() - 60_000), responsibleUserId: userId,
contextSnapshot: { issueId, executionWorkspaceId: workspace!.id, wakeReason: "issue_children_completed" } }).returning();
const [environment] = await db.insert(environments).values({ name: `sandbox-${runId}`, driver: "sandbox" }).returning();
await db.insert(environmentLeases).values({ companyId, issueId, executionWorkspaceId: workspace!.id,
heartbeatRunId: runId, environmentId: environment!.id, status: "active", leasePolicy: "reuse_by_environment", provider: "daytona",
providerLeaseId: `sandbox-${runId}`, metadata: { remoteCwd: "/work/remote-task", driver: "sandbox",
workspaceRealization: { authoritativeRoot: "/work/remote-task" } } });
await terminalizeLegacyExecution({ db, run: run!, status: "failed", patch: { errorCode: "adapter_failed",
error: "native_instruction_materialization_handoff_unavailable", finishedAt: endedAt,
resultJson: options.restoreFailed === false ? { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }
: { workspaceRestoreFailure: "restore_failed" } } });
const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
return { issueId, runId, actionId: hold?.id ?? null };
}
async function savedWorkRecoveryState(actionId: string, runId: string, state: "running" | "complete") {
await db.update(issueRecoveryActions).set({ updatedAt: new Date(),
evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({ systemRecovery: { ...createSystemRecovery(runId), state } })}::jsonb` })
.where(eq(issueRecoveryActions.id, actionId));
}
async function systemRetryAtClaim(f: { issueId: string; runId: string }) {
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", issueId: f.issueId,
runtimeMode: "legacy", status: "queued", retryOfRunId: f.runId,
contextSnapshot: { issueId: f.issueId, wakeReason: "issue_continuation_needed", retryReason: "issue_continuation_needed",
retryOfRunId: f.runId } }).returning();
const outcome = await createPostgresRunDispatchAdapter(db).cancelStaleQueuedRun({ companyId, runId: run!.id,
expectedStatus: "queued", now: new Date() });
return { run: run!, outcome };
}
async function deliveredContinuationIsClaimable(f: { issueId: string; runId: string; actionId: string | null }, refusedRunId: string) {
await heartbeatService(db).reconcileStrandedAssignedIssues({ companyId, issueId: f.issueId });
expect((await issue(f.issueId)).status).not.toBe("blocked");
expect(await getExecutionBlocker(db, companyId, f.issueId)).toBeNull();
await occupySlot();
await deliver();
const next = (await successors(f.runId)).filter(run => run.id !== refusedRunId);
expect(next).toHaveLength(1);
expect(next[0]!.status).toBe("queued");
expect(await createPostgresRunDispatchAdapter(db).cancelStaleQueuedRun({ companyId, runId: next[0]!.id,
expectedStatus: "queued", now: new Date() })).toEqual({ outcome: "not_stale" });
return next[0]!;
}
it("continues a follow-up that failed before provider work with unrestored files, when the saved work was recovered before the system retry was refused at claim", async () => {
const f = await followUpFailedBeforeProviderWork();
expect(f.actionId).not.toBeNull();
await savedWorkRecoveryState(f.actionId!, f.runId, "complete");
const refused = await systemRetryAtClaim(f);
expect(refused.outcome).toMatchObject({ outcome: "cancelled", errorCode: "execution_reconciliation_required" });
expect((await holdRow(f.actionId!)).evidence).toMatchObject({
freshContinuation: { sourceRunId: f.runId }, continuationDelivery: "pending",
freshContinuationSettlement: { refusedAtDispatchRunId: refused.run.id, requestedBy: "system" } });
await deliveredContinuationIsClaimable(f, refused.run.id);
}, 60_000);
it("continues a follow-up that failed before provider work with unrestored files, when the saved work was recovered after the system retry was refused at claim", async () => {
const f = await followUpFailedBeforeProviderWork();
await savedWorkRecoveryState(f.actionId!, f.runId, "running");
const refused = await systemRetryAtClaim(f);
expect(refused.outcome).toMatchObject({ outcome: "cancelled", errorCode: "execution_reconciliation_required" });
expect((await holdRow(f.actionId!)).evidence.freshContinuationCheck)
.toMatchObject({ kind: "exact_recovery_pending", reason: "saved_work_recovery_running" });
await savedWorkRecoveryState(f.actionId!, f.runId, "complete");
await makeDue(f.actionId!);
expect(await continueHeldExecutions(db)).toMatchObject({ continued: 1 });
expect((await holdRow(f.actionId!)).evidence).toMatchObject({ freshContinuation: { sourceRunId: f.runId },
freshContinuationSettlement: { refusedAtDispatchRunId: refused.run.id, requestedBy: "system" } });
await deliveredContinuationIsClaimable(f, refused.run.id);
}, 60_000);
it("continues a follow-up that failed before provider work with its files in place, after its system retry was refused at claim", async () => {
const f = await followUpFailedBeforeProviderWork({ restoreFailed: false });
const refused = await systemRetryAtClaim(f);
expect(refused.outcome).toMatchObject({ outcome: "cancelled", errorCode: "execution_reconciliation_required" });
expect((await holdRow(f.actionId!)).evidence).toMatchObject({ freshContinuation: { sourceRunId: f.runId },
freshContinuationSettlement: { refusedAtDispatchRunId: refused.run.id, requestedBy: "system" } });
await deliveredContinuationIsClaimable(f, refused.run.id);
}, 60_000);
it("counts only automatic retries that started, or still may, against the task's hourly budget", async () => {
const f = await followUpFailedBeforeProviderWork();
for (let index = 0; index < FRESH_CONTINUATIONS_PER_HOUR; index++) {
await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", issueId: f.issueId, runtimeMode: "legacy",
status: "cancelled", errorCode: "execution_reconciliation_required", finishedAt: new Date(), retryOfRunId: f.runId,
scheduledRetryReason: "transient_failure", contextSnapshot: { issueId: f.issueId, wakeReason: "transient_failure_retry" } });
}
expect(await sharedContinuationAttempts(db, companyId, f.issueId, new Date())).toHaveLength(0);
expect((await taskAutomaticAttempts(db, companyId, f.issueId, new Date())).started).toHaveLength(0);
await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", issueId: f.issueId, runtimeMode: "legacy",
status: "failed", errorCode: "adapter_failed", startedAt: new Date(), finishedAt: new Date(), retryOfRunId: f.runId,
scheduledRetryReason: "transient_failure", contextSnapshot: { issueId: f.issueId, wakeReason: "transient_failure_retry" } });
await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", issueId: f.issueId, runtimeMode: "legacy",
status: "scheduled_retry", retryOfRunId: f.runId,
scheduledRetryReason: "transient_failure", contextSnapshot: { issueId: f.issueId, wakeReason: "transient_failure_retry" } });
expect(await sharedContinuationAttempts(db, companyId, f.issueId, new Date())).toHaveLength(2);
expect((await taskAutomaticAttempts(db, companyId, f.issueId, new Date())).started).toHaveLength(2);
}, 60_000);
it("still leaves a person's failed Retry of a follow-up for that person after its successor is refused at claim", async () => {
const f = await followUpFailedBeforeProviderWork();
await savedWorkRecoveryState(f.actionId!, f.runId, "complete");
await db.update(heartbeatRuns).set({ contextSnapshot: sql`${heartbeatRuns.contextSnapshot} || '{"wakeReason":"task_retry"}'::jsonb` })
.where(eq(heartbeatRuns.id, f.runId));
const refused = await systemRetryAtClaim(f);
expect(refused.outcome).toMatchObject({ outcome: "cancelled", errorCode: "execution_reconciliation_required" });
expect((await holdRow(f.actionId!)).evidence.freshContinuation).toBeUndefined();
expect(await ensureFreshContinuation(db, { companyId, runId: f.runId, trigger: "held_recovery" }))
.toEqual({ kind: "person_owned", reason: "retry_needed" });
}, 60_000);
});