Files
PaperClipAI/server/src/__tests__/task-interruption-automatic-path.test.ts
T
DottaandClaude Opus 5.5 5026093d0f feat(presentation): offer Move it forward now while a continuation is still waiting to start (#21385)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, Paperclip records a fresh continuation
and delivers it later as a new run. Since #21353 the task reads as
continuing (`working`, no Retry) while it waits
> - #21373 added "Move it forward now" (`retryAction: "move_forward"`).
It is offered only after the delivered continuation run waited 15
minutes in the agent's queue
> - Before delivery (`continuationDelivery: "pending"`) the task showed
no button. On hosted staging such continuations waited 30 to 110 minutes
behind a deep queue, and the person had no control
> - #21378 and #21379 made the click safe in that state: a
`move_forward` request with a recorded, undelivered continuation is
accepted and delivers that continuation as the one run, with its
recorded isolation
> - This pull request also offers "Move it forward now" while the
continuation waits for delivery, once it has waited 15 minutes since it
was recorded
> - The benefit is one action, with the same label and copy, for both
waiting states. The click never starts a second run

## Linked Issues or Issue Description

Refs #21353, #21373, #21378, #21379

**What happened?**

A task has a fresh continuation that is recorded but not delivered
(`continuationDelivery: "pending"`). It waits behind a deep agent queue
for 15 minutes or more. The task shows "continuing" with no action.

**Expected behavior**

After 15 minutes the task still reads as continuing (`working`,
responsibility `system`, never `failed`). It also offers "Move it
forward now". The click delivers the recorded continuation as the one
run.

**Steps to reproduce**

1. Record a fresh continuation for a task while its agent is busy. Do
not run the delivery pass.
2. Wait 15 minutes.
3. Read the task presentation. It shows `canRetry: false` and no
`retryAction`.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: the
fresh-continuation read also loads the record status, schema,
`recordedAt` and reconciled run. A `continuing` state can now carry
`pendingSince` when the primary record is `resolved`,
`continuationDelivery` is `pending`, the schema is the
fresh-continuation schema, and the reconciled run is the observed run.
These are the same facts that `undeliveredContinuation` checks at click
time.
- The projection offers `canRetry: true, retryAction: "move_forward"`
(state `working`) when `pendingSince` is 15 minutes or older, no person
Retry for that run already waits, and the run's agent is the current
assignee. Under 15 minutes, nothing changes.
- No button when the system gave up delivery (`invalidated`; the normal
Retry rules apply), when the delivery is `not_requested`, when a person
already asked, or for a held restore (the existing execution-blocker
post-filter removes the offer).
- New integration test
`server/src/services/fresh-continuation-move-forward-offer.test.ts`. It
uses the real click path for the three recorded kinds
(`fresh_isolated_workspace`, `writer_ended`, native `remote_epoch`):
presentation, `requestTaskRetry` with the shown `observedRunId` and
`retryAction`, `resumeTaskRetry`, then the delivery pass.
- Updated guards in `task-interruption-automatic-path.test.ts` and
`recovery-wake-herd.test.ts` (#21353, #21373). Their assertions that a
pending continuation shows no Retry at 30, 45, 110 and 240 minutes now
expect the move-forward offer. The `working` / `system` / never-failed
assertions stay. New cases cover 14 minutes, `not_requested`, a waiting
person Retry and reassignment.
- `doc/execution-semantics.md`: describes the new offer and its races.
- No shared type, validator, UI or API client change. The action and its
UI copy ("Waiting for the agent to finish other work. Paperclip
continues this task by itself." / "Move it forward now") already exist
and do not depend on the state. No runner-e2e SDK public-input pin
covers the changed files.

## Verification

- Red: with the presentation change reverted, 16 tests fail (`expected {
intentId: null, …(7) } to match object { responsibility: 'system', …
retryAction: 'move_forward' }`).
- Green: `npx vitest run
src/services/fresh-continuation-move-forward-offer.test.ts`: 14 of 14
pass. For each kind:
- The click gives exactly one run. It is the recorded continuation
(`recoveryActionId`, `previousRunId`, `retryOfRunId`,
`forceFreshSession`). The fresh isolated workspace is kept
(`freshContinuationIsolation`, `readFreshWorkspaceReadiness`). The
record reads `delivered` with that run. Two later delivery passes and a
repeated resume create no second run, and the offer then ends.
  - The click and two concurrent delivery passes give one run.
- Delivered between the offer and the click: the click is refused
(`accepted: false`, no wake) and the only run is the delivered
continuation.
- The system gave up the delivery between the offer and the click:
`alreadyMovedOn: true`, no run.
- The task moved to another run between the offer and the click:
refused, the only run is that other run.
  - Under 15 minutes: no button.
- Guards: `task-interruption-automatic-path.test.ts` (37/37),
`recovery-wake-herd.test.ts`, `task-interruption-presentation.test.ts`,
`task-interruption-query.test.ts`,
`fresh-continuation-retry-delivery.test.ts`,
`fresh-continuation-fallback.test.ts`, `task-retry.test.ts`: all pass.
Also the saved-work, workspace-recovery, native restart and
closed-session retry suites that read the presentation.
- `npx tsc --noEmit -p .` in `server/`: no errors.

## Risks

Low risk. The change is presentation only. It adds four selected JSON
fields to a read that already exists and one conditional module import.
The click path (`task-retry.ts`, `fresh-continuation-retry-fold.ts`) is
unchanged and already holds the task row lock. A record without
`recordedAt` or with an older schema gets no button, as before. People
now see a button on tasks that showed none after 15 minutes. The button
can only deliver the recorded continuation or do nothing.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:24:12 -06:00

455 lines
33 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, agentWakeupRequests, applyPendingMigrations, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db";
import { taskInterruptionPresentation } from "../services/task-interruption-presentation.js";
import { createSystemRecovery } from "../services/system-recovery-state.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TASK_INTERRUPTION_TEST_DATABASE_URL");
const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
vi.mock("../sentry.js", () => ({ captureTaskInterruptionFailure: vi.fn() }));
describe.skipIf(!support.supported)("Retry is offered only when no automatic path exists", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
if (externalDatabaseUrl) await applyPendingMigrations(externalDatabaseUrl);
database = externalDatabaseUrl ? { connectionString: externalDatabaseUrl, cleanup: async () => {} }
: await startEmbeddedPostgresTestDatabase("interruption-automatic-");
db = createDb(database.connectionString);
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
async function seedTask(status = "in_progress") {
const companyId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Swap company", issuePrefix: `S${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Swap agent", adapterType: "paperclip_runner" });
const [issue] = await db.insert(issues).values({ companyId, title: "Work across a release swap",
status, assigneeAgentId: agentId }).returning();
return { companyId, agentId, issue: issue! };
}
async function seedSwapEndedRun(finishedAt: Date) {
const task = await seedTask();
const runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: task.companyId, agentId: task.agentId,
nativeIssueId: task.issue.id, scopeKind: "issue", status: "failed", runtimeMode: "native",
nativePhase: "terminal_failure", errorCode: "native_restart_recovery_blocked",
error: "controller_process_space_unverified", startedAt: new Date(finishedAt.getTime() - 60_000), finishedAt });
await db.insert(nativeRunFinalizations).values({ companyId: task.companyId, issueId: task.issue.id, runId,
phase: "terminal_failure", attempt: 1, failureCode: "native_restart_recovery_blocked", recoveryState: "blocked" });
const [hold] = await db.insert(issueRecoveryActions).values({ companyId: task.companyId, sourceIssueId: task.issue.id,
kind: "active_run_watchdog", status: "active", ownerType: "system", returnOwnerAgentId: task.agentId,
cause: "native_restart_recovery_blocked", fingerprint: `native-restart:${runId}`,
evidence: { runId, reason: "controller_process_space_unverified", executorStopEvidence: "unverified", providerAttempt: 1 },
nextAction: "The server running this task restarted. Paperclip continues the task in a fresh run from its saved work.",
maxAttempts: 3, createdAt: finishedAt, updatedAt: finishedAt }).returning();
return { ...task, runId, hold: hold! };
}
it("shows a run ended by a release swap as working while its system hold awaits the automatic continuation", async () => {
const { issue, runId } = await seedSwapEndedRun(new Date(Date.now() - 5_000));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "working",
canRetry: false, observedRunId: runId });
});
it("still offers Retry when the system hold was not taken up by the continuation worker in time", async () => {
const { issue } = await seedSwapEndedRun(new Date(Date.now() - 10 * 60_000));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
});
it("still offers Retry once the continuation worker decided it cannot continue the task", async () => {
const { issue, hold } = await seedSwapEndedRun(new Date(Date.now() - 5_000));
await db.update(issueRecoveryActions).set({ evidence: { ...hold.evidence,
freshContinuationCheck: { kind: "not_applicable", reason: "task_changed", checks: 1, checkedAt: Date.now(), nextCheckAt: null } } })
.where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
});
it("still offers Retry when a person owns the hold", async () => {
const { issue, hold, companyId } = await seedSwapEndedRun(new Date(Date.now() - 5_000));
await db.update(issueRecoveryActions).set({ evidence: { ...hold.evidence, ownerGoneAttestation: { companyId } } })
.where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
});
it("still offers Retry when recovery gave up and handed the hold to the board", async () => {
const { issue, hold } = await seedSwapEndedRun(new Date(Date.now() - 5_000));
await db.update(issueRecoveryActions).set({ ownerType: "board" }).where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
});
async function seedRecordedContinuation(minutesAgo: number) {
const recordedAt = new Date(Date.now() - minutesAgo * 60_000);
const task = await seedSwapEndedRun(new Date(recordedAt.getTime() - 30_000));
const freshContinuation = { schema: "paperclip.fresh-continuation.v1", sourceRunId: task.runId, trigger: "held_recovery",
failureCode: "native_restart_recovery_blocked", delivery: "recorded", recordedAt: recordedAt.toISOString(),
isolation: "remote_epoch", stopProof: "unproven_epoch_abandoned", unverifiedActions: [], createdObjects: [] };
const settlement = { schema: "paperclip.fresh-continuation.v1", actorType: "system", actorId: "fresh-continuation",
decision: "continue_in_fresh_run", replay: "none", sourceRunId: task.runId, reason: "exact_resumption_unavailable",
isolation: "remote_epoch", recordedAt: recordedAt.toISOString() };
const reconciliation = { runId: task.runId, providerStopped: true, providerStopProof: "isolated", actionOutcome: "mixed",
actorId: "fresh-continuation", recordedAt: recordedAt.toISOString() };
const [primary] = await db.insert(issueRecoveryActions).values({ companyId: task.companyId, sourceIssueId: task.issue.id,
kind: "active_run_watchdog", status: "resolved", outcome: "restored", ownerType: "system", returnOwnerAgentId: task.agentId,
cause: "native_continuation_requires_reconciliation", fingerprint: `native-continuation:${task.runId}`,
nextAction: "Paperclip continues the task in a fresh run from its saved work.", resolutionNote: "continued_fresh_from_last_saved",
resolvedAt: recordedAt, createdAt: recordedAt, updatedAt: recordedAt,
evidence: { runId: task.runId, executionReconciliation: reconciliation, freshContinuationSettlement: settlement,
freshContinuation, continuationDelivery: "pending" } }).returning();
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: recordedAt, updatedAt: recordedAt,
evidence: { ...task.hold.evidence, executionReconciliation: reconciliation, freshContinuationSettlement: settlement,
continuationDelivery: "coalesced", coalescedIntoRecoveryActionId: primary!.id } })
.where(eq(issueRecoveryActions.id, task.hold.id));
return { ...task, primary: primary! };
}
it("shows a task whose recorded continuation is still waiting to start as continuing, however long delivery takes", async () => {
for (const minutesAgo of [1, 14, 30, 110]) {
const { issue, runId } = await seedRecordedContinuation(minutesAgo);
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "working",
canRetry: minutesAgo >= 15, observedRunId: runId });
}
});
it("keeps a recorded continuation as continuing after a refused delivery attempt, since delivery tries again", async () => {
const { issue, primary } = await seedRecordedContinuation(45);
await db.update(issueRecoveryActions).set({ evidence: { ...primary.evidence,
continuationDeliveryAttemptedAt: new Date(Date.now() - 60_000).toISOString() } }).where(eq(issueRecoveryActions.id, primary.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "working", canRetry: true, retryAction: "move_forward" });
});
it("offers Retry once the system gives up on a recorded continuation", async () => {
const { issue, primary, hold } = await seedRecordedContinuation(45);
await db.update(issueRecoveryActions).set({ evidence: { ...primary.evidence, continuationDelivery: "invalidated" } })
.where(eq(issueRecoveryActions.id, primary.id));
await db.update(issueRecoveryActions).set({ evidence: { ...hold.evidence, continuationDelivery: "invalidated",
coalescedIntoRecoveryActionId: primary.id } }).where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "failed", canRetry: true });
});
it("follows the new run once a recorded continuation is delivered", async () => {
const { companyId, agentId, issue, primary } = await seedRecordedContinuation(45);
const nextRunId = randomUUID();
await db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
status: "queued", runtimeMode: "native", retryOfRunId: primary.evidence.runId as string });
await db.update(issueRecoveryActions).set({ evidence: { ...primary.evidence, continuationDelivery: "delivered",
continuationRunId: nextRunId } }).where(eq(issueRecoveryActions.id, primary.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "working", canRetry: false, observedRunId: nextRunId });
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "adapter_failed", finishedAt: new Date(Date.now() - 10 * 60_000) })
.where(eq(heartbeatRuns.id, nextRunId));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true, observedRunId: nextRunId });
});
async function seedQueuedContinuation(queuedMinutesAgo: number, status: "queued" | "running" = "queued") {
const task = await seedRecordedContinuation(queuedMinutesAgo + 2);
await db.update(heartbeatRuns).set({ createdAt: new Date(Date.now() - (queuedMinutesAgo + 4) * 60_000) })
.where(eq(heartbeatRuns.id, task.runId));
const nextRunId = randomUUID();
await db.insert(heartbeatRuns).values({ id: nextRunId, companyId: task.companyId, agentId: task.agentId, nativeIssueId: task.issue.id,
scopeKind: "issue", status, runtimeMode: "native", retryOfRunId: task.runId,
createdAt: new Date(Date.now() - queuedMinutesAgo * 60_000),
contextSnapshot: { issueId: task.issue.id, wakeReason: "issue_recovery_action_restored", previousRunId: task.runId,
recoveryActionId: task.primary.id } });
await db.update(issueRecoveryActions).set({ evidence: { ...task.primary.evidence, continuationDelivery: "delivered",
continuationRunId: nextRunId } }).where(eq(issueRecoveryActions.id, task.primary.id));
return { ...task, nextRunId };
}
it("offers a person a way to move a continuation forward once it has waited 15 minutes to start, without calling it failed", async () => {
for (const minutesAgo of [15, 45, 110]) {
const { issue, nextRunId } = await seedQueuedContinuation(minutesAgo);
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "working",
canRetry: true, retryAction: "move_forward", observedRunId: nextRunId });
}
});
it("keeps a continuation that has waited under 15 minutes as continuing with no Retry", async () => {
const { issue, nextRunId } = await seedQueuedContinuation(14);
const presentation = await taskInterruptionPresentation(db, issue);
expect(presentation).toMatchObject({ state: "working", canRetry: false, observedRunId: nextRunId });
expect(presentation?.retryAction).toBeUndefined();
});
it("offers to move a recorded continuation forward once it has waited 15 minutes for delivery, without calling it failed", async () => {
for (const minutesAgo of [15, 45, 110]) {
const { issue, runId } = await seedRecordedContinuation(minutesAgo);
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "working",
canRetry: true, retryAction: "move_forward", observedRunId: runId });
}
});
it("offers no Retry while a recorded continuation has waited under 15 minutes for delivery", async () => {
const { issue } = await seedRecordedContinuation(14);
const presentation = await taskInterruptionPresentation(db, issue);
expect(presentation).toMatchObject({ state: "working", canRetry: false });
expect(presentation?.retryAction).toBeUndefined();
});
it("offers no Retry for a continuation recorded without a delivery request", async () => {
const { issue, primary } = await seedRecordedContinuation(45);
await db.update(issueRecoveryActions).set({ evidence: { ...primary.evidence, continuationDelivery: "not_requested" } })
.where(eq(issueRecoveryActions.id, primary.id));
const presentation = await taskInterruptionPresentation(db, issue);
expect(presentation).toMatchObject({ state: "working", canRetry: false });
expect(presentation?.retryAction).toBeUndefined();
});
it("offers no second Retry once a person already asked to move the undelivered continuation forward", async () => {
const { companyId, agentId, issue, runId } = await seedRecordedContinuation(45);
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", triggerDetail: "manual",
reason: "task_retry", status: "deferred_issue_execution", requestedByActorType: "user", requestedByActorId: randomUUID(),
payload: { issueId: issue.id, taskRetry: { sourceRunId: runId } } });
const presentation = await taskInterruptionPresentation(db, issue);
expect(presentation).toMatchObject({ state: "working", canRetry: false });
expect(presentation?.retryAction).toBeUndefined();
});
it("offers no move-forward for an undelivered continuation once the task is reassigned", async () => {
const { companyId, issue } = await seedRecordedContinuation(45);
const otherAgentId = randomUUID();
await db.insert(agents).values({ id: otherAgentId, companyId, name: "Other agent", adapterType: "paperclip_runner" });
const [reassigned] = await db.update(issues).set({ assigneeAgentId: otherAgentId }).where(eq(issues.id, issue.id)).returning();
expect((await taskInterruptionPresentation(db, reassigned!))?.retryAction).toBeUndefined();
});
it("offers no Retry for a continuation that is already running", async () => {
const { issue } = await seedQueuedContinuation(45, "running");
const presentation = await taskInterruptionPresentation(db, issue);
expect(presentation).toMatchObject({ state: "working", canRetry: false });
expect(presentation?.retryAction).toBeUndefined();
});
it("offers no second Retry once a person already moved the waiting continuation forward", async () => {
const { companyId, agentId, issue, nextRunId } = await seedQueuedContinuation(45);
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", triggerDetail: "manual",
reason: "task_retry", status: "coalesced", requestedByActorType: "user", requestedByActorId: randomUUID(),
runId: nextRunId, finishedAt: new Date(), payload: { issueId: issue.id } });
const presentation = await taskInterruptionPresentation(db, issue);
expect(presentation).toMatchObject({ state: "working", canRetry: false });
expect(presentation?.retryAction).toBeUndefined();
});
it("offers no Retry for queued work that is not a recorded continuation, however long it waits", async () => {
const { companyId, agentId, issue } = await seedTask("todo");
await db.insert(heartbeatRuns).values({ companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue", status: "queued",
runtimeMode: "native", createdAt: new Date(Date.now() - 45 * 60_000), contextSnapshot: { issueId: issue.id, wakeReason: "issue_assigned" } });
const presentation = await taskInterruptionPresentation(db, issue);
expect(presentation).toMatchObject({ state: "working", canRetry: false });
expect(presentation?.retryAction).toBeUndefined();
});
it("asks a person to resume a paused agent instead of offering to move its waiting continuation forward", async () => {
const { agentId, issue } = await seedQueuedContinuation(45);
await db.update(agents).set({ status: "paused", pauseReason: "manual" }).where(eq(agents.id, agentId));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "user", state: "waiting_for_user",
userAction: "resume", canRetry: false });
});
it("still offers Retry for a resolved recovery record that recorded no continuation", async () => {
const { issue, hold } = await seedSwapEndedRun(new Date(Date.now() - 45 * 60_000));
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: new Date(Date.now() - 40 * 60_000),
evidence: { ...hold.evidence, freshContinuationSettlement: { schema: "paperclip.fresh-continuation.v1", decision: "continue_in_fresh_run" } } })
.where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "failed", canRetry: true });
});
it("still offers Retry when the board owns the hold and no continuation is recorded", async () => {
const { issue, hold } = await seedSwapEndedRun(new Date(Date.now() - 45 * 60_000));
await db.update(issueRecoveryActions).set({ ownerType: "board" }).where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
});
async function seedProviderTerminalFailure(finishedAt: Date) {
const task = await seedTask("blocked");
const runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: task.companyId, agentId: task.agentId,
nativeIssueId: task.issue.id, scopeKind: "issue", status: "failed", runtimeMode: "native",
nativePhase: "terminal_failure", errorCode: "native_provider_terminal_failed",
error: "ACP agent reported a terminal service failure.", startedAt: new Date(finishedAt.getTime() - 60_000), finishedAt });
await db.insert(nativeRunFinalizations).values({ companyId: task.companyId, issueId: task.issue.id, runId,
phase: "terminal_failure", attempt: 1, failureCode: "native_provider_terminal_failed", updatedAt: finishedAt });
const [hold] = await db.insert(issueRecoveryActions).values({ companyId: task.companyId, sourceIssueId: task.issue.id,
kind: "active_run_watchdog", status: "active", ownerType: "board", returnOwnerAgentId: task.agentId,
cause: "native_provider_terminal_failed", fingerprint: `provider-terminal:${runId}`,
evidence: { runId, nativeFailureBlock: { runId, statusVersion: 1 }, coordinatorAttempt: 1,
sourceFailureCode: "native_provider_terminal_failed", recoveryDisposition: "native_provider_terminal_failed",
recoveryMode: "replacement", providerSessionEstablished: true },
nextAction: "Verify that the failed provider stopped and reconcile its action outcomes.",
maxAttempts: 3, createdAt: finishedAt, updatedAt: finishedAt }).returning();
return { ...task, issue: { ...task.issue, status: "blocked" }, runId, hold: hold! };
}
it("shows a provider failure the system is still recovering as working through every recovery step, never offering Retry in between", async () => {
const { issue, runId, hold, companyId } = await seedProviderTerminalFailure(new Date(Date.now() - 2_000));
const working = { responsibility: "system", state: "working", canRetry: false, observedRunId: runId };
expect(await taskInterruptionPresentation(db, issue)).toMatchObject(working);
const running = { ...createSystemRecovery(runId), state: "running" as const, phase: "preserve_files" as const };
await db.update(issueRecoveryActions).set({ ownerType: "system", updatedAt: new Date(),
evidence: { ...hold.evidence, systemRecovery: running, systemRecoveryActiveAt: Date.now() } })
.where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject(working);
await db.update(issueRecoveryActions).set({ updatedAt: new Date(),
evidence: { ...hold.evidence, systemRecovery: { ...running, state: "complete" }, systemRecoveryActiveAt: Date.now() - 60_000 } })
.where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject(working);
await db.update(nativeRunFinalizations).set({ failureDetail: { replacementDenied: "provider_failure_meaning_unverified" },
updatedAt: new Date() }).where(eq(nativeRunFinalizations.runId, runId));
await db.update(issueRecoveryActions).set({ ownerType: "board", cause: "provider_failure_meaning_unverified",
fingerprint: `native-replacement:${runId}`, evidence: { runId, attempts: 1 }, nextAction: "Review the provider failure.",
updatedAt: new Date() }).where(and(eq(issueRecoveryActions.id, hold.id), eq(issueRecoveryActions.companyId, companyId)));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject(working);
});
it("offers Retry for a provider failure once the recovery bound passes with nothing recorded", async () => {
const { issue } = await seedProviderTerminalFailure(new Date(Date.now() - 3 * 60_000));
expect(await taskInterruptionPresentation(db, { ...issue, status: "todo" })).toMatchObject({ state: "failed", canRetry: true });
});
it("offers Retry for a provider failure at once when the continuation worker decided it cannot continue the task", async () => {
const { issue, hold } = await seedProviderTerminalFailure(new Date(Date.now() - 2_000));
await db.update(issueRecoveryActions).set({ evidence: { ...hold.evidence,
freshContinuationCheck: { kind: "not_applicable", reason: "task_changed", checks: 1, checkedAt: Date.now(), nextCheckAt: null } } })
.where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, { ...issue, status: "todo" })).toMatchObject({ state: "failed", canRetry: true });
});
it("offers Retry for a provider failure at once when a person took over its hold", async () => {
const { issue, hold, companyId } = await seedProviderTerminalFailure(new Date(Date.now() - 2_000));
await db.update(issueRecoveryActions).set({ evidence: { ...hold.evidence, ownerGoneAttestation: { companyId } } })
.where(eq(issueRecoveryActions.id, hold.id));
expect(await taskInterruptionPresentation(db, { ...issue, status: "todo" })).toMatchObject({ state: "failed", canRetry: true });
});
it("shows a healthy task as working while the message queued during its succeeded run is delivered", async () => {
const { companyId, agentId, issue } = await seedTask();
const runId = randomUUID(), finishedAt = new Date(Date.now() - 3_000);
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
status: "succeeded", runtimeMode: "native", startedAt: new Date(Date.now() - 5 * 60_000), finishedAt });
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", reason: "issue_commented",
status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: new Date(Date.now() - 2 * 60_000),
payload: { issueId: issue.id } });
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "working", canRetry: false });
expect(await taskInterruptionPresentation(db, { ...issue, status: "in_review" })).toMatchObject({ state: "working", canRetry: false });
expect(await taskInterruptionPresentation(db, { ...issue, status: "blocked" })).toMatchObject({ state: "failed", canRetry: true });
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - 10 * 60_000) }).where(eq(heartbeatRuns.id, runId));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ canRetry: true });
});
async function seedParentWaitingOnSubtasks(childStatus = "in_progress") {
const { companyId, agentId, issue } = await seedTask("blocked");
for (const title of ["Subtask 1", "Subtask 2"])
await db.insert(issues).values({ companyId, title, status: childStatus, assigneeAgentId: agentId, parentId: issue.id });
const runId = randomUUID(), finishedAt = new Date(Date.now() - 1_000);
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
status: "succeeded", runtimeMode: "native", startedAt: new Date(Date.now() - 25 * 60_000), finishedAt });
return { companyId, agentId, issue, runId };
}
async function seedCommentQueuedDuringRun(companyId: string, agentId: string, issueId: string) {
const commentId = randomUUID();
const [wake] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation",
reason: "issue_execution_deferred", triggerDetail: "system", status: "deferred_issue_execution",
requestedByActorType: "user", requestedByActorId: "local-board", requestedAt: new Date(Date.now() - 13 * 60_000),
payload: { issueId, mutation: "comment", commentId,
_paperclipWakeContext: { source: "issue.comment", issueId, wakeReason: "issue_commented", wakeCommentIds: [commentId] } },
}).returning();
return wake!;
}
it("shows a parent whose run just succeeded waiting on its subtasks as working while its queued message is settled", async () => {
const { companyId, agentId, issue, runId } = await seedParentWaitingOnSubtasks();
await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "working",
canRetry: false, observedRunId: runId });
});
it("shows the parent as resolved once its queued message is cancelled as already delivered", async () => {
const { companyId, agentId, issue } = await seedParentWaitingOnSubtasks();
const wake = await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date(),
error: "Queued messages were already delivered to a completed run" }).where(eq(agentWakeupRequests.id, wake.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "resolved", canRetry: false });
});
it("shows the parent as working once its queued message is promoted to a new run", async () => {
const { companyId, agentId, issue } = await seedParentWaitingOnSubtasks();
const wake = await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
const nextRunId = randomUUID();
await db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
status: "queued", runtimeMode: "native", wakeupRequestId: wake.id });
await db.update(agentWakeupRequests).set({ status: "queued", runId: nextRunId }).where(eq(agentWakeupRequests.id, wake.id));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "working", canRetry: false, observedRunId: nextRunId });
});
it("offers Retry on the parent once its queued message is not settled within the grace", async () => {
const { companyId, agentId, issue, runId } = await seedParentWaitingOnSubtasks();
await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - 3 * 60_000) }).where(eq(heartbeatRuns.id, runId));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
});
it("still offers Retry on a blocked task with a queued message and no open subtasks", async () => {
for (const childStatus of [null, "done", "cancelled"]) {
const { companyId, agentId, issue } = await seedTask("blocked");
if (childStatus) await db.insert(issues).values({ companyId, title: "Finished subtask", status: childStatus,
assigneeAgentId: agentId, parentId: issue.id });
await db.insert(heartbeatRuns).values({ companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
status: "succeeded", runtimeMode: "native", finishedAt: new Date(Date.now() - 1_000) });
await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
}
});
it("still offers Retry on a parent with open subtasks when its queued wake carries no message", async () => {
const { companyId, agentId, issue } = await seedParentWaitingOnSubtasks();
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", reason: "issue_assigned",
status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: new Date(Date.now() - 13 * 60_000),
payload: { issueId: issue.id } });
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
});
it("still offers Retry on a parent with open subtasks and a queued message when its run failed and recovery gave up", async () => {
const { companyId, agentId, issue, runId, hold } = await seedSwapEndedRun(new Date(Date.now() - 5_000));
await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, issue.id));
await db.insert(issues).values({ companyId, title: "Open subtask", status: "in_progress", assigneeAgentId: agentId, parentId: issue.id });
await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
await db.update(issueRecoveryActions).set({ ownerType: "board" }).where(eq(issueRecoveryActions.id, hold.id));
const blocked = { ...issue, status: "blocked" };
expect(await taskInterruptionPresentation(db, blocked)).toMatchObject({ state: "failed", canRetry: true, observedRunId: runId });
});
it("still offers Retry on a parent with open subtasks and a queued message when saved-work recovery is exhausted", async () => {
const { companyId, agentId, issue, runId } = await seedParentWaitingOnSubtasks();
await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issue.id, kind: "active_run_watchdog",
status: "escalated", ownerType: "system", returnOwnerAgentId: agentId, cause: "native_restart_recovery_blocked",
fingerprint: `exhausted:${runId}`, nextAction: "Retry to continue.", maxAttempts: 3,
evidence: { runId, systemRecovery: { ...createSystemRecovery(runId), state: "exhausted", attempts: 3, retriesUsed: 3 } } });
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
});
it("does not show a failed attempt on a parent whose run succeeded with no queued message", async () => {
for (const childStatus of ["in_progress", "blocked", "cancelled"]) {
const { issue } = await seedParentWaitingOnSubtasks(childStatus);
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "resolved", canRetry: false });
}
});
it("does not offer Retry on a parent whose run succeeded and that waits for its subtasks", async () => {
const { companyId, agentId, issue } = await seedTask();
await db.insert(issues).values({ companyId, title: "Subtask", status: "in_progress", assigneeAgentId: agentId, parentId: issue.id });
await db.insert(heartbeatRuns).values({ companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
status: "succeeded", runtimeMode: "native", finishedAt: new Date(Date.now() - 3_000) });
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", reason: "issue_children_completed",
status: "queued", requestedByActorType: "system", payload: { issueId: issue.id } });
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ canRetry: false });
});
});