Files
PaperClipAI/server/src/services/task-interruption-presentation.test.ts
T
DottaandClaude Opus 5.5 e82c4f0a31 fix(runner): prove a warm Runner's suspension on a slow sandbox host so the next run does not fail (#21472)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**

This PR extends the failed-close retry that #21442 (merged) added. It is
based on `master`, which contains #21442.

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote sandbox is reused by the next run of the same task.
Before the next run starts, the previous run's Runner must be retired:
suspended, with durable proof.
> - An idle warm close had 10 s for everything. On a loaded sandbox host
one liveness probe or one read of the durable state used that up. The
suspend was never sent, or it completed but was never proven.
> - The next run then replayed that cached close failure, failed before
any provider work, and the task showed a failed run with Retry
> - For a follow-up on a task blocked on its subtasks, the automatic
continuation was refused and the Retry stayed until another wake
> - This pull request bounds each probe, gives the suspend proof its own
read window, lets a later close prove the suspension again, and lets
that follow-up continue by itself
> - The benefit is that a slow host no longer turns a completed
suspension into a failed task, and the person sees no Retry for it

## Linked Issues or Issue Description

**What happened?**

Local release-swap cells on Daytona (G5c and G6c, 2026-11-13) each
showed one Retry banner: "previous run's session could not be retired
(runner did not durably suspend before checkpoint)". G6c had no fault
injection, so this is a real class.

- In the G6c case, the Runner completed its suspend and exited. The
server's proof read (a sandbox exec under load) ran out of the remaining
0.6 s of the close grace, so the suspension was "not proven".
- In the G5c case (an adopted Runner), the liveness probe
`adoptedRunner.isAlive()` took about 220 s and used the whole close
deadline. The suspend was never queued. Five adopted sessions failed the
same way in the same second.
- The successor run's retirement called `transport.close()`, which
returned the cached rejection of the predecessor's failed close. It did
not send a new suspend and did not read the state again.
- The successor failed before provider work. Its fresh-sandbox
continuation (`interrupted_preparation`) was refused as "blocked after
recovery", because the parent task was blocked on its subtasks before
the follow-up started. The task showed a failed run with Retry for 27 s,
until a blocker-stall wake.

**Expected behavior**

A completed suspend is proven, even on a slow host. A follow-up that
fails before any provider work on a task that was already blocked
continues by itself, and the task shows as working while that
continuation is queued.

**Steps to reproduce**

1. Close an idle warm Runner whose durable state read takes longer than
the close grace, or whose liveness probe stalls.
2. On master the close fails and the next close replays the same
failure.
3. The new tests in `runnerd-codex-transport-close-proof.test.ts` and
`heartbeat-native-predispatch-retry.test.ts` do this.

**Paperclip version or commit**

`master` at c6b497d831 (contains #21442).

## What Changed

- `runnerd-codex-transport.ts` (controller-side TypeScript):
- Each liveness probe in the close is bounded. A probe that does not
answer counts as possibly alive, never as gone, and the suspend goes
over the live connection. A connected Runner's probe gets a 1 s slice,
so drain and suspend keep their budget.
- Once the suspend command's completion is observed, what the
durable-state read shows decides how long the proof may take:

    | Read after the completion | Proof window | Maximum close time |
    | --- | --- | --- |
    | Suspended, right Runner | proven at once | unchanged |
| Another run's or Runner's state | ends at the close deadline, as on
master | close grace (10 s; 45 s for Pi) |
| Right Runner, not yet `suspended` | settle window of 3 s from the
first such read (`RUNNER_SUSPENSION_PROOF_SETTLE_MS`, test option
`closeProofSettleMs`) | close grace + 3 s |
| Unreadable or slow read | full proof window of 30 s from the
completion (`RUNNER_SUSPENSION_PROOF_READ_MS`, `closeProofReadMs`) |
close grace + 30 s |

All rows add the existing bounded exit-probe slices. runner-core saves
the `suspended` lifecycle before it delivers the suspend result
(`persist_lifecycle_before_command_delivery` in `durable/runner.rs`), so
a readable non-suspended state after the completion can only be a stale
remote read; the settle window covers a few read attempts and is not a
write-ordering wait.
- The settlement diagnostic names the branch that ended the window
(`proofWindowEnd`: `proven`, `wrong_identity`,
`not_suspended_after_settle`, `unreadable_after_window`,
`suspend_not_completed`).
- A close that failed only because readable proof was missing is retried
by the next close: it reads the durable state again with a fresh bound
and finishes settled. A close whose proof was refuted stays memoized, as
on master. This extends the failed-close retry from #21442.
  - The close failure carries per-step timings.
- `native-runtime/predecessor-retirement.ts`,
`heartbeat/run-completion.ts`: the predecessor-retirement fallback
records the task's status when the run failed.
- `modules/run-dispatch/domain/policy.ts`,
`modules/run-dispatch/adapters/postgres.ts`: a continuation of a run
that did no provider work, on a task that was already blocked then, is
not treated as blocked after recovery. Its dependency check follows the
wake it continues. The continuation keeps a person's comment wake.
- `task-interruption-presentation.ts`: while that run's failure handling
queues the continuation (up to 30 s), the task shows as working with no
Retry. The window ends when the continuation is scheduled, when the run
records `continuationScheduling: "refused"` (written by
`run-completion.ts` when the fallback does not schedule it), or after 30
s. The fallback record alone no longer ends it.

## Verification

Red on master, green on this branch:

- `runnerd-codex-transport-close-proof.test.ts`
- `proves a completed suspend on its own read window when the durable
state read outlasts the close grace`
- `suspends five adopted idle Runners whose liveness probes stall
together, instead of spending the close on the probe`
- `heartbeat-native-predispatch-retry.test.ts`
- `continues a follow-up on a blocked task whose predecessor could not
be retired`
- `does not continue a task that was blocked only after its run failed`
- `task-interruption-presentation.test.ts`
- `shows the task as working while its fresh-sandbox continuation is
being queued`
- `keeps the task working after the fallback is recorded while its
continuation is still being scheduled` (review fix)
  - `offers Retry when the continuation was refused or never queued`
- `runnerd-codex-transport.test.ts` (review fixes)
- `gives a suspend that completes near the close deadline its whole
proof window from the completion`
- `bounds the proof window from the observed completion, so a proof read
that never answers still ends`
- `proves a suspension whose suspended state appears 1.5 s after the
command completed`
- `fails a suspension whose state stays ready after the settle window,
not after the whole proof window`
- `fails at the close deadline when the durable state belongs to another
run`
- The three existing tests in
`runnerd-codex-transport-suspension.test.ts` that expect a conclusive
negative to fail within the close grace pass unchanged.

Local runs on this branch: `tsc --noEmit` is clean for `server` and
`packages/paperclip-runner`. Passing on this branch:
`runnerd-codex-transport-suspension`, `-close-proof`, `-adoption`,
`-remote-retention`, `runnerd-detach-durability` (all files), the
proof-window cases of `runnerd-codex-transport.test.ts`,
`task-interruption-presentation.test.ts` (72), `run-dispatch` policy and
Postgres adapter tests and `heartbeat-native-predispatch-retry.test.ts`
(166 together).

Conflict resolved: `task-interruption-presentation.ts` gained
`WAITING_CONTINUATION_MOVE_FORWARD_MS` on master (#21385). Both
constants are kept.

Runner content ID: the Runner change is controller-side TypeScript under
`packages/paperclip-runner/src` only. There is no Rust, `protocol/` or
`generated/` change. Existing Runner images work with this controller.

Daytona: this fix has not run on Daytona at all. The two banners above
are the evidence of the class (G5c and G6c ran on 588c6ba267, without
this fix).

## Risks

- Not proven on Daytona.
- A close can now take up to 30 s longer when the proof read is
unreadable or slow, and up to 3 s longer when the read is stale. This is
on the close path only, not before `turn.start`.
- A probe that does not answer now counts as possibly alive, so the
close sends a suspend instead of treating the Runner as gone.
- Ownership: this PR changes
`server/src/modules/run-dispatch/domain/policy.ts`,
`server/src/modules/run-dispatch/adapters/postgres.ts` and
`server/src/services/heartbeat/run-completion.ts`. CODEOWNERS does not
list these files. Their recent changes come from the restart-recovery
and Retry work (#21348, #21256, #20863, #21302). That area should review
the dispatch-policy change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:32:22 -06:00

598 lines
40 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { describe, expect, it } from "vitest";
import { projectTaskInterruption } from "./task-interruption-presentation.js";
const issue = { id: "task", companyId: "11111111-1111-4111-8111-111111111111", status: "blocked", assigneeAgentId: "agent", projectId: "22222222-2222-4222-8222-222222222222" };
const run = { id: "run", agentId: "agent", status: "failed", scheduledRetryAt: null,
errorCode: "project_access_denied", error: null, livenessState: null,
resultJson: { projectAccessDenied: { projectId: "22222222-2222-4222-8222-222222222222" } } };
const agent = { status: "idle", pauseReason: null };
describe("saved governed wait shutdown", () => {
const now = Date.parse("2026-11-09T12:00:00Z");
const pending = { ...run, runtimeMode: "native", errorCode: "native_governed_wait_settlement_pending", resultJson: null };
it("names automatic shutdown settlement without offering a duplicate Retry", () => {
expect(projectTaskInterruption(issue, { run: { ...pending, nativeResumeAt: new Date(now + 30_000) }, agent,
paused: false, approval: false }, now)).toMatchObject({ state: "working", canRetry: false,
retryWait: { reason: "governed_wait_settlement", message: "Your work is saved. Paperclip is finishing shutdown." } });
});
it("explains exhausted shutdown recovery while preserving the saved work", () => {
expect(projectTaskInterruption(issue, { run: pending, agent, paused: false, approval: false }, now))
.toMatchObject({ state: "failed", canRetry: false, retryWait: { reason: "governed_wait_settlement",
message: "Paperclip couldn’t confirm shutdown. Your work is saved; continuation is paused." } });
});
it("keeps genuine approvals authoritative", () => {
expect(projectTaskInterruption(issue, { run: pending, agent, paused: false, approval: true }, now))
.toMatchObject({ state: "waiting_for_user", userAction: "approval" });
});
});
describe("queued dependency wait presentation", () => {
const waitingRun = { ...run, status: "cancelled", runtimeMode: "legacy" as const, startedAt: null,
errorCode: "issue_dependencies_blocked", resultJson: {
stopReason: "issue_dependencies_blocked", timeoutSource: "dependency_gate",
cancellation: { source: "unknown", expected: false },
} };
it("keeps a dependency-gated wake out of the failed-attempt UI, including retained input", () => {
for (const wait of [undefined, { id: "saved-input", requestedAt: new Date(0) }]) {
expect(projectTaskInterruption(issue, { run: waitingRun, agent, wait, paused: false, approval: false }))
.toMatchObject({ state: "resolved", canRetry: false });
}
});
it.each(["legacy", "native"])("keeps an unstarted %s dispatcher dependency wait neutral", runtimeMode => {
const dispatchedWait = { ...waitingRun, runtimeMode,
resultJson: { ...waitingRun.resultJson, timeoutSource: "stale_queued_run_gate" } };
expect(projectTaskInterruption(issue, { run: dispatchedWait, agent, paused: false, approval: false }))
.toMatchObject({ state: "resolved", canRetry: false });
expect(projectTaskInterruption(issue, { run: { ...dispatchedWait, startedAt: new Date() }, agent, paused: false, approval: false }))
.toMatchObject({ state: "failed", canRetry: true });
});
it("keeps genuine failures and current user gates visible", () => {
for (const changed of [{ status: "failed" }, { runtimeMode: "native" }, { startedAt: new Date() },
{ errorCode: "setup_failed" }, { resultJson: {} }, { resultJson: { ...waitingRun.resultJson, timeoutFired: true } }]) {
expect(projectTaskInterruption(issue, { run: { ...waitingRun, ...changed }, agent, paused: false, approval: false }))
.toMatchObject({ state: "failed", canRetry: true });
}
expect(projectTaskInterruption(issue, { run: waitingRun, agent, paused: true, approval: false }))
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
expect(projectTaskInterruption(issue, { run: waitingRun, agent, paused: false, approval: true }))
.toMatchObject({ state: "waiting_for_user", userAction: "approval" });
expect(projectTaskInterruption(issue, { run: waitingRun, agent, paused: false, approval: false,
action: { evidence: {}, status: "active" } as never })).toMatchObject({ state: "failed", canRetry: true });
expect(projectTaskInterruption(issue, { run: waitingRun, agent, paused: false, approval: false,
retryWait: { id: "retry", sourceRunId: waitingRun.id, requestedAt: new Date(), preparationStatus: "failed" } }))
.toMatchObject({ state: "failed", canRetry: true });
});
});
describe("never-started task refused by admission", () => {
const todo = { ...issue, status: "todo" };
const admissionWait = { id: "wake", status: "skipped", reason: "budget.blocked", agentId: "agent" };
it("names an accounting hold as an automatic wait without a Retry", () => {
expect(projectTaskInterruption(todo, { agent, admissionWait, paused: false, approval: false })).toMatchObject({
responsibility: "system", state: "working", canRetry: false,
retryWait: { reason: "budget.blocked", message: expect.stringContaining("starts automatically") } });
});
it("keeps pauses and budget stops as user actions instead of promising an automatic start", () => {
for (const [state, action] of [
[{ project: { pausedAt: new Date(), pauseReason: null } }, "resume"],
[{ company: { status: "paused", pauseReason: "manual" } }, "resume"],
[{ paused: true }, "resume"],
[{ agent: { status: "paused", pauseReason: "manual" } }, "resume"],
[{ company: { status: "paused", pauseReason: "budget" } }, "budget"],
[{ approval: true }, "approval"],
] as const) {
expect(projectTaskInterruption(todo, { agent, admissionWait, paused: false, approval: false, ...state }))
.toMatchObject({ responsibility: "user", state: "waiting_for_user", userAction: action });
}
});
it("shows nothing for other refusals, started tasks or another assignee", () => {
for (const changed of [{ reason: "agent.not_invokable" }, { status: "queued" }, { agentId: "other" }]) {
expect(projectTaskInterruption(todo, { agent, admissionWait: { ...admissionWait, ...changed }, paused: false, approval: false }))
.toBeNull();
}
expect(projectTaskInterruption({ ...todo, status: "cancelled" }, { agent, admissionWait, paused: false, approval: false })).toBeNull();
});
});
describe("unadmitted chat Retry presentation", () => {
const now = Date.parse("2026-01-01T12:00:00Z");
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
const wait = { id: "retry", requestedAt: new Date(now), sourceRunId: run.id,
preparationStatus: "issued", preparationRetryAt: now + 250 };
const input = { run: failed, agent, wait, paused: false, approval: false };
it.each(["issued", "processing"])("uses the persisted %s outbox deadline without implying provider work", status => {
for (const offset of [0, 20_000]) expect(projectTaskInterruption(issue, {
...input, wait: { ...wait, preparationStatus: status },
}, now + offset)).toMatchObject({ state: "retry_scheduled", canRetry: false,
nextRetryAt: new Date(wait.preparationRetryAt).toISOString() });
});
it.each(["failed", "cancelled"])("keeps terminal %s preparation out of the working state", status => {
expect(projectTaskInterruption(issue, { ...input, wait: { ...wait, preparationStatus: status,
preparationRetryAt: null } }, now)).toMatchObject({ state: "failed", nextRetryAt: null });
});
it("keeps current gates and admitted execution authoritative", () => {
expect(projectTaskInterruption(issue, { ...input, paused: true }, now))
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
expect(projectTaskInterruption(issue, { ...input, approval: true }, now))
.toMatchObject({ state: "waiting_for_user", userAction: "approval" });
for (const status of ["queued", "running"]) expect(projectTaskInterruption(issue, {
...input, run: { ...failed, id: "successor", status },
}, now)).toMatchObject({ state: "working", nextRetryAt: null });
expect(projectTaskInterruption(issue, { ...input, wait: { ...wait, sourceRunId: "old-run" } }, now))
.toMatchObject({ state: "working", nextRetryAt: null });
});
});
// The native coordinator resumes the same run itself after an attach failure.
// A manual Retry offered meanwhile would race that automatic resume.
it("shows a pending native same-run resume instead of a competing Retry", () => {
const now = Date.parse("2026-11-06T15:25:50.000Z");
const failed = { ...run, runtimeMode: "native" as const, errorCode: "native_session_interrupted", resultJson: null };
const resumeAt = new Date("2026-11-06T15:26:18.687Z");
expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: resumeAt }, agent, paused: false, approval: false }, now))
.toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() });
// A due resume is being dispatched; it is still not the user's turn.
expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 30_000) }, agent, paused: false, approval: false }, now))
.toMatchObject({ state: "working", canRetry: false });
// A resume that never ran, or no resume at all, leaves exactly one Retry.
expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 10 * 60_000) }, agent, paused: false, approval: false }, now))
.toMatchObject({ state: "failed", canRetry: true });
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false }, now))
.toMatchObject({ state: "failed", canRetry: true });
});
it("explains a persisted Retry wait without exposing the saved payload", () => {
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
const wait = { id: "retry", requestedAt: new Date(), sourceRunId: failed.id,
executionWaitReason: "workspace_repair_required" };
const presentation = projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false });
expect(presentation).toMatchObject({ state: "failed", canRetry: true,
retryWait: { reason: "workspace_repair_required", message: "Retry is waiting for Paperclip to recover the saved files. Your message is saved." } });
expect(projectTaskInterruption(issue, { run: failed, agent, wait: { ...wait, sourceRunId: "another-run" }, paused: false, approval: false }))
.not.toHaveProperty("retryWait");
expect(projectTaskInterruption(issue, { run: failed, agent, wait: { ...wait, executionWaitReason: "secret-payload" }, paused: false, approval: false }))
.toMatchObject({ retryWait: { reason: "execution_recovery", message: "Retry is waiting for execution checks. Your message is saved." } });
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: true, approval: false }))
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
});
it("names a budget gate instead of generic execution checks for a refused Retry", () => {
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
wait: { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: "budget.blocked" } }))
.toMatchObject({ state: "failed", canRetry: true, retryWait: { reason: "budget.blocked",
message: "Retry is waiting for the budget to allow new work. Check Costs for unrecorded or unpriced usage." } });
});
it.each(["cancelled_by_operator", "retry_source_changed"])("does not offer Retry for terminal Retry waits: %s", reason => {
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
wait: { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: reason } }))
.toMatchObject({ state: "failed", canRetry: false, retryWait: { reason } });
});
it("holds a Retry on a cleanup gate only while an owner settles it", () => {
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
const wait = { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: "cleanup_quarantined" };
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false, wait, cleanupSettling: true }))
.toMatchObject({ state: "failed", canRetry: false, retryWait: { reason: "cleanup_quarantined" } });
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false, wait }))
.toMatchObject({ state: "failed", canRetry: true, retryWait: { reason: "cleanup_quarantined" } });
});
it("keeps Retry for a Retry declined for its requester's access, since a new click replaces it", () => {
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
wait: { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: "retry_authority_denied" } }))
.toMatchObject({ state: "failed", canRetry: true, retryWait: { reason: "retry_authority_denied" } });
});
it("preserves the actual workspace preparation error without changing retry authority", () => {
const failedRun = { ...run, errorCode: "workspace_validation_failed",
error: "The selected project has no repository source for a remote workspace.", resultJson: null };
expect(projectTaskInterruption(issue, { run: failedRun, agent, paused: false, approval: false }))
.toMatchObject({ state: "failed", canRetry: true, workspaceFailure: { message: failedRun.error } });
for (const status of ["done", "cancelled"]) expect(projectTaskInterruption({ ...issue, status },
{ run: failedRun, agent, paused: false, approval: false })).not.toHaveProperty("workspaceFailure");
expect(projectTaskInterruption(issue, { run: { ...failedRun, errorCode: "unrelated_failure" }, agent, paused: false, approval: false }))
.not.toHaveProperty("workspaceFailure");
});
describe("setup command failure presentation (CF1)", () => {
const setupError = "Execution workspace provision command \"bash ./scripts/provision-worktree.sh\" failed: bash: ./scripts/provision-worktree.sh: No such file or directory";
const setupRun = { ...run, errorCode: "setup_failed", error: setupError, resultJson: { setupCommandFailure: {
step: "workspace_provision", command: "bash ./scripts/provision-worktree.sh", exitCode: 127,
output: "bash: ./scripts/provision-worktree.sh: No such file or directory" } } };
it("names the broken setup command on the task without changing retry authority", () => {
const withCause = projectTaskInterruption(issue, { run: setupRun, agent, paused: false, approval: false });
const withoutCause = projectTaskInterruption(issue, { run: { ...setupRun, error: "x", resultJson: null }, agent, paused: false, approval: false });
expect(withCause).toMatchObject({ state: "failed", canRetry: true, userAction: null, setupFailure: {
step: "workspace_provision", command: "bash ./scripts/provision-worktree.sh", exitCode: 127,
message: "The project's workspace provision command \"bash ./scripts/provision-worktree.sh\" failed with exit code 127: bash: ./scripts/provision-worktree.sh: No such file or directory" } });
const { setupFailure: _named, ...rest } = withCause!;
expect(rest).toEqual(withoutCause);
expect(withCause).not.toHaveProperty("workspaceFailure");
});
it("recognizes a setup failure recorded before the structured cause existed", () => {
expect(projectTaskInterruption(issue, { run: { ...setupRun, resultJson: null }, agent, paused: false, approval: false }))
.toMatchObject({ state: "failed", setupFailure: { step: "workspace_provision", command: "bash ./scripts/provision-worktree.sh",
exitCode: null, output: "bash: ./scripts/provision-worktree.sh: No such file or directory" } });
});
it("keeps a transient setup retry scheduled and only adds the name", () => {
const retrying = { ...setupRun, status: "scheduled_retry", scheduledRetryAt: new Date(Date.now() + 60_000), scheduledRetryReason: "transient_failure" };
const named = projectTaskInterruption(issue, { run: retrying, agent, paused: false, approval: false });
const unnamed = projectTaskInterruption(issue, { run: { ...retrying, error: "x", resultJson: null }, agent, paused: false, approval: false });
expect(named?.setupFailure?.command).toBe("bash ./scripts/provision-worktree.sh");
const { setupFailure: _named, ...rest } = named!;
expect(rest).toEqual(unnamed);
});
it("bounds the output tail and strips credentials from the command", () => {
const noisy = { ...setupRun, resultJson: { setupCommandFailure: { step: "workspace_provision",
command: "GITHUB_TOKEN=ghp_abcdefghijklmnopqrstuvwxyz0123456789 ./setup.sh --password hunter22", exitCode: 1,
output: `${"x".repeat(5_000)}\nfinal line` } } };
const failure = projectTaskInterruption(issue, { run: noisy, agent, paused: false, approval: false })?.setupFailure;
expect(failure?.command).not.toContain("ghp_abcdefghijklmnopqrstuvwxyz0123456789");
expect(failure?.command).not.toContain("hunter22");
expect(failure?.output?.length).toBeLessThanOrEqual(300);
expect(failure?.output).toMatch(/final line$/);
});
it("recognizes older quoted and multiline commands", () => {
for (const command of ['bash "./scripts/provision-worktree.sh"', "set -e\n./setup.sh"]) {
expect(projectTaskInterruption(issue, { run: { ...setupRun, resultJson: null,
error: `Execution workspace provision command "${command}" failed with exit code 2` }, agent, paused: false, approval: false })?.setupFailure)
.toMatchObject({ command, exitCode: 2 });
}
});
it("does not name setup for other failures or settled tasks", () => {
expect(projectTaskInterruption(issue, { run: { ...setupRun, errorCode: "adapter_failed" }, agent, paused: false, approval: false }))
.not.toHaveProperty("setupFailure");
expect(projectTaskInterruption({ ...issue, status: "done" }, { run: setupRun, agent, paused: false, approval: false }))
.not.toHaveProperty("setupFailure");
});
});
function operatorStoppedRun() {
return { ...run, status: "cancelled", runtimeMode: "native" as const, companyId: issue.companyId,
nativeIssueId: issue.id, errorCode: "cancelled", resultJson: {
cancelledByActorType: "user", cancelledByUserId: "board",
executionCancellation: { state: "requested" },
nativeCancellation: { schema: "paperclip.native-cancellation.v1", runId: run.id,
companyId: issue.companyId, issueId: issue.id, scope: "run",
reasonCode: "cancellation_run_only", dispatched: true, dispatchState: "acknowledged",
intentAuditId: "intent", acknowledgementAuditId: "ack" },
} };
}
describe("intentional native Stop presentation", () => {
const workingIssue = { ...issue, status: "in_progress" };
it("does not offer a failed-attempt Retry for an audited operator Stop that keeps its task in progress", () => {
expect(projectTaskInterruption(workingIssue, { run: operatorStoppedRun(), agent, paused: false, approval: false }))
.toMatchObject({ state: "resolved", canRetry: false, userAction: null });
expect(workingIssue.status).toBe("in_progress");
});
it.each([{ runId: "foreign" }, { companyId: "foreign" }, { issueId: "foreign" },
{ dispatchState: "pending" }, { dispatched: false }, { acknowledgementAuditId: undefined },
{ intentAuditId: 123 }, { acknowledgementAuditId: {} },
{ scope: "subtree" }])("retains failure presentation without the matching operator receipt %j", change => {
const stopped = operatorStoppedRun();
expect(projectTaskInterruption(workingIssue, { run: { ...stopped, resultJson: { ...stopped.resultJson,
nativeCancellation: { ...stopped.resultJson.nativeCancellation, ...change } } }, agent, paused: false, approval: false }))
.toMatchObject({ state: "failed", canRetry: true });
});
it("keeps pause, approval, non-user cancellation and new input authoritative", () => {
const stopped = operatorStoppedRun();
expect(projectTaskInterruption(workingIssue, { run: stopped, agent, paused: true, approval: false }))
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
expect(projectTaskInterruption(workingIssue, { run: stopped, agent, paused: false, approval: true }))
.toMatchObject({ state: "waiting_for_user", userAction: "approval" });
expect(projectTaskInterruption(workingIssue, { run: { ...stopped, resultJson: {
...stopped.resultJson, cancelledByActorType: "system" } }, agent, paused: false, approval: false }))
.toMatchObject({ state: "failed", canRetry: true });
expect(projectTaskInterruption(workingIssue, { run: stopped, agent, paused: false, approval: false,
wait: { id: "new-input", requestedAt: new Date() } }))
.toMatchObject({ state: "working", canRetry: false });
});
});
it("preserves a provider explanation on a failed task without changing retry authority", () => {
const providerFailure = { schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "startup",
category: "rate_limit", message: "API error: 429 This request would exceed your account's rate limit.",
httpStatus: 429, retryability: "unknown" };
const failedRun = { ...run, errorCode: "native_session_interrupted", resultJson: { providerFailure } };
const presentation = projectTaskInterruption(issue, { run: failedRun, agent, paused: false, approval: false });
expect(presentation).toMatchObject({ state: "failed", canRetry: true, providerFailure });
expect(projectTaskInterruption({ ...issue, status: "done" }, { run: failedRun, agent, paused: false, approval: false }))
.not.toHaveProperty("providerFailure");
});
describe("task project-access interruption", () => {
it("explains a project denial without offering a futile retry", () => {
expect(projectTaskInterruption(issue, { run, agent, paused: false, approval: false })).toMatchObject({
state: "waiting_for_user", responsibility: "user", userAction: "project_access", canRetry: false,
projectAccess: { agentId: "agent", projectId: "22222222-2222-4222-8222-222222222222", ready: false },
});
});
it("projects the historical setup denial without requiring another failed run", () => {
expect(projectTaskInterruption(issue, { run: { ...run, errorCode: "setup_failed",
error: "Project is outside this low-trust boundary.", resultJson: null }, agent, paused: false, approval: false }))
.toMatchObject({ userAction: "project_access", canRetry: false });
});
it("offers an explicit retry after the project is added, without changing the failed run", () => {
expect(projectTaskInterruption(issue, { run, agent, projectAccessAllowed: true, paused: false, approval: false }))
.toMatchObject({ state: "failed", canRetry: true, projectAccess: { ready: true } });
expect(run.status).toBe("failed");
});
it("shows an accepted retry as working after access is repaired", () => {
const input = { run, agent, paused: false, approval: false, wait: { id: "request", requestedAt: new Date() } };
expect(projectTaskInterruption(issue, { ...input, projectAccessAllowed: true }))
.toMatchObject({ state: "working", canRetry: false });
expect(projectTaskInterruption(issue, { ...input, projectAccessAllowed: true })).not.toHaveProperty("projectAccess");
expect(projectTaskInterruption(issue, { ...input, projectAccessAllowed: false }))
.toMatchObject({ state: "waiting_for_user", userAction: "project_access", canRetry: false });
});
it("does not reinterpret unrelated failures or an old assignee's denial", () => {
for (const otherRun of [{ ...run, errorCode: "setup_failed", error: "Disk full", resultJson: null },
{ ...run, agentId: "previous-agent" }]) {
expect(projectTaskInterruption(issue, { run: otherRun, agent, paused: false, approval: false }))
.not.toHaveProperty("projectAccess");
}
});
it("keeps approval, pause, running and completed states authoritative", () => {
expect(projectTaskInterruption(issue, { run, agent, paused: false, approval: true })).toMatchObject({ userAction: "approval" });
expect(projectTaskInterruption(issue, { run, agent, paused: true, approval: false })).toMatchObject({ userAction: "resume" });
expect(projectTaskInterruption(issue, { run: { ...run, status: "running" }, agent, paused: false, approval: false }))
.toMatchObject({ state: "working" });
expect(projectTaskInterruption({ ...issue, status: "done" }, { run, agent, paused: false, approval: false }))
.toMatchObject({ state: "resolved" });
});
});
describe("saved-work recovery step", () => {
const now = Date.parse("2026-01-01T12:00:00Z");
const failed = { ...run, errorCode: "setup_failed", resultJson: null };
const system = (phase: string, state: string, nextAttemptAt: number | null) => ({
schema: "paperclip.system-recovery.v1", policyVersion: 1, sourceRunId: "33333333-3333-4333-8333-333333333333", revision: 3,
generationId: "44444444-4444-4444-8444-444444444444", intentId: null, phase, state, attemptId: null, attempts: 2,
retriesUsed: 1, firstFailureAt: now - 60_000, nextAttemptAt, deadlineAt: now + 600_000, providerCooldownAt: null,
failure: null, preservationOperationIds: [], freshAttemptRunId: null });
const action = (evidence: Record<string, unknown>) => ({ evidence }) as never;
it("names the step a scheduled recovery will retry", () => {
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
action: action({ systemRecovery: system("verify_execution", "scheduled", now + 30_000) }) }, now))
.toMatchObject({ state: "retry_scheduled", recoveryStep: "checking_previous_run" });
});
it("names the step a running recovery is on", () => {
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
action: action({ systemRecovery: system("preserve_files", "running", null), systemRecoveryActiveAt: now - 1_000 }) }, now))
.toMatchObject({ state: "working", recoveryStep: "saving_previous_files" });
});
it("names recovery over a pending chat preparation and keeps failed preparation authoritative", () => {
const wait = { id: "retry", requestedAt: new Date(now), sourceRunId: failed.id, preparationStatus: "issued", preparationRetryAt: now + 250 };
const running = action({ systemRecovery: system("preserve_files", "running", null), systemRecoveryActiveAt: now - 1_000 });
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false, action: running }, now))
.toMatchObject({ state: "working", recoveryStep: "saving_previous_files" });
const scheduled = action({ systemRecovery: system("verify_execution", "scheduled", now + 30_000) });
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false, action: scheduled }, now))
.toMatchObject({ state: "retry_scheduled", recoveryStep: "checking_previous_run", canRetry: false });
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false, action: scheduled }, now))
.toMatchObject({ state: "retry_scheduled", canRetry: true });
expect(projectTaskInterruption(issue, { run: failed, agent, wait: { ...wait, preparationStatus: "failed" },
paused: false, approval: false, action: running }, now)).toMatchObject({ state: "failed" });
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false }, now))
.toMatchObject({ state: "retry_scheduled", nextRetryAt: new Date(now + 250).toISOString() });
});
it("does not name a step for an ordinary scheduled retry", () => {
expect(projectTaskInterruption(issue, { run: { ...failed, scheduledRetryAt: new Date(now + 30_000) }, agent,
paused: false, approval: false }, now)).not.toHaveProperty("recoveryStep");
});
});
describe("scheduled resource wait presentation", () => {
const now = Date.parse("2026-01-01T12:00:00Z");
const waiting = { ...run, status: "scheduled_retry", errorCode: null, resultJson: null,
scheduledRetryReason: "workspace_busy", scheduledRetryAt: new Date(now - 60_000) };
it("keeps an overdue workspace wait out of the failed-attempt UI", () => {
expect(projectTaskInterruption(issue, { run: waiting, agent, paused: false, approval: false }, now))
.toMatchObject({ state: "working", canRetry: false });
});
it("keeps failure retries, user gates and failed Retry preparation authoritative", () => {
expect(projectTaskInterruption(issue, { run: { ...waiting, scheduledRetryReason: "transient_failure" },
agent, paused: false, approval: false }, now)).toMatchObject({ state: "failed", canRetry: true });
expect(projectTaskInterruption(issue, { run: waiting, agent, paused: true, approval: false }, now))
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
expect(projectTaskInterruption(issue, { run: waiting, agent, paused: false, approval: false,
retryWait: { id: "retry", sourceRunId: run.id, requestedAt: new Date(now), preparationStatus: "failed" } }, now))
.toMatchObject({ state: "failed", canRetry: true });
});
});
describe("saved Retry held by dependencies", () => {
const failedRun = { ...run, status: "failed", errorCode: "native_session_interrupted", resultJson: {} };
const retryWait = { id: "retry", requestedAt: new Date(0), sourceRunId: "run" };
const project = (retryDependencies: { count: number; blocked: boolean; cancelled: boolean; savingFilesOnly: boolean }) =>
projectTaskInterruption(issue, { run: failedRun, agent, retryWait, paused: false, approval: false, retryDependencies });
it.each([
[{ count: 1, blocked: true, cancelled: false, savingFilesOnly: false }, "Retry or unblock it"],
[{ count: 1, blocked: false, cancelled: true, savingFilesOnly: false }, "Remove or replace that dependency"],
[{ count: 1, blocked: false, cancelled: false, savingFilesOnly: true }, "to save its files"],
[{ count: 2, blocked: false, cancelled: false, savingFilesOnly: false }, "to finish"],
])("names what the saved Retry waits on (%o)", (dependencies, text) => {
const presentation = project(dependencies);
expect(presentation).toMatchObject({ state: "working", canRetry: false, intentId: "retry",
retryWait: { reason: "dependency_blocked" } });
expect(presentation?.retryWait?.message).toContain(text);
});
});
describe("setup command failure evidence", () => {
it("removes a registered secret that crosses the output cut before shortening", async () => {
const { setupCommandFailureResultJson } = await import("./workspace-setup-command-failure.js");
const secret = "registered-private-value-0123456789";
const failure = Object.assign(new Error("setup failed"), { step: "workspace_provision", command: "./setup.sh", exitCode: 1,
// The cut keeps the last 299 characters; it lands inside the secret.
output: `${secret}${"y".repeat(299 - 10)}` });
const recorded = setupCommandFailureResultJson(failure, [secret]);
expect(recorded).toMatchObject({ setupCommandFailure: { step: "workspace_provision", exitCode: 1 } });
const text = JSON.stringify(recorded);
for (let index = 0; index + 8 <= secret.length; index += 1) expect(text).not.toContain(secret.slice(index, index + 8));
});
});
describe("saved result that ran out of finalization retries", () => {
const review = { ...issue, status: "in_review" };
const succeeded = { ...run, status: "succeeded", runtimeMode: "native" as const, errorCode: null, resultJson: null };
it("offers Retry for a succeeded run whose accepted result could not be finalized", () => {
expect(projectTaskInterruption(review, { run: { ...succeeded, nativeFinalizationRetry: true }, agent,
paused: false, approval: false })).toMatchObject({ state: "failed", canRetry: true, observedRunId: "run" });
});
it("shows the saved result being applied after Retry", () => {
expect(projectTaskInterruption(review, { run: { ...succeeded, nativeFinalizationRetryPending: true }, agent,
paused: false, approval: false })).toMatchObject({ state: "working", canRetry: false, recoveryStep: "finishing_recovery" });
});
it("keeps ordinary succeeded and failed runs unchanged", () => {
expect(projectTaskInterruption(review, { run: succeeded, agent, paused: false, approval: false }))
.toMatchObject({ state: "resolved", canRetry: false });
expect(projectTaskInterruption(review, { run: { ...succeeded, status: "failed", errorCode: "adapter_failed" }, agent,
paused: false, approval: false })).toMatchObject({ state: "failed", canRetry: true });
expect(projectTaskInterruption({ ...review, status: "done" }, { run: { ...succeeded, nativeFinalizationRetry: true },
agent, paused: false, approval: false })).toMatchObject({ state: "resolved", canRetry: false });
});
});
// Remote native run shape: the provider turn completed and its work was
// saved, then a same-run recovery was refused by the cleanup gate. The task
// showed "Couldn't finish this attempt" with a Retry the server refuses (409).
describe("native cleanup quarantine presentation", () => {
const quarantined = { ...run, runtimeMode: "native" as const, errorCode: "native_session_cleanup_quarantined",
error: "native_session_cleanup_quarantined: prior session cleanup requires operator recovery; verify its retained process ownership and checkpoint before a controlled restart. Clearing a task session does not resolve this quarantine.",
resultJson: null, finishedAt: new Date("2026-11-10T14:39:31.873Z") };
it("shows the task's blocker wait, not a failed attempt, after a completed turn", () => {
expect(projectTaskInterruption(issue, { run: { ...quarantined, completedTurnCleanup: true }, agent,
waitingOnBlockers: true, paused: false, approval: false })).toMatchObject({ state: "resolved", canRetry: false });
});
it("waits without Retry while an owner settles the gate", () => {
for (const completedTurnCleanup of [true, false]) {
const presentation = projectTaskInterruption(issue, { run: { ...quarantined, completedTurnCleanup }, agent,
cleanupSettling: true, paused: false, approval: false });
expect(presentation).toMatchObject({ state: "failed", canRetry: false, retryWait: { reason: "cleanup_quarantined" } });
expect(presentation?.retryWait?.message).not.toMatch(/operator|admin/i);
}
const preparation = { id: "retry", requestedAt: new Date(0), sourceRunId: quarantined.id, preparationStatus: "failed" };
expect(projectTaskInterruption(issue, { run: quarantined, agent, retryWait: preparation, cleanupSettling: true,
paused: false, approval: false })).toMatchObject({ canRetry: false });
});
it("offers Retry when nothing settles the gate by itself (s9107 t04/t05)", () => {
// An operator-required gate, or one lost with a restarted process, has no
// owner. Waiting would never end, so Retry is the way forward.
for (const completedTurnCleanup of [true, false]) {
const presentation = projectTaskInterruption(issue, { run: { ...quarantined, completedTurnCleanup }, agent,
paused: false, approval: false });
expect(presentation).toMatchObject({ state: "failed", canRetry: true });
expect(presentation).not.toHaveProperty("retryWait");
}
});
it("never leaves a failed task without Retry, a scheduled retry or a settling owner", () => {
// Invariant: a failed presentation without Retry must name a wait that an
// owner is settling. Enumerate the quarantine-related inputs.
const waits = [undefined, ...["cleanup_quarantined", "local_cleanup", "execution_settling"].map(reason =>
({ id: "retry", requestedAt: new Date(0), sourceRunId: quarantined.id, executionWaitReason: reason }))];
for (const completedTurnCleanup of [true, false]) for (const wait of waits) for (const settling of [true, false]) {
const presentation = projectTaskInterruption(issue, { run: { ...quarantined, completedTurnCleanup }, agent,
retryWait: wait, cleanupSettling: settling, paused: false, approval: false });
if (presentation?.state !== "failed" || presentation.canRetry) continue;
expect(presentation.retryWait, JSON.stringify({ completedTurnCleanup, wait: wait?.executionWaitReason, settling }))
.toBeDefined();
if (presentation.retryWait?.reason === "cleanup_quarantined") expect(settling).toBe(true);
}
});
it("keeps Retry for ordinary failed native attempts", () => {
expect(projectTaskInterruption(issue, { run: { ...quarantined, errorCode: "native_session_interrupted" }, agent,
paused: false, approval: false })).toMatchObject({ state: "failed", canRetry: true });
});
});
describe("automatic scheduled retry", () => {
const now = Date.parse("2026-11-12T23:00:00Z");
const scheduled = { ...run, status: "scheduled_retry", runtimeMode: "legacy", errorCode: null, resultJson: null,
scheduledRetryReason: "transient_failure", scheduledRetryAt: new Date(now + 60_000) };
it("shows a pending transient-failure retry as scheduled without a competing Retry", () => {
expect(projectTaskInterruption(issue, { run: scheduled, agent, paused: false, approval: false }, now))
.toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: new Date(now + 60_000).toISOString() });
});
it("offers Retry once the scheduled retry is past due with nothing else pending", () => {
expect(projectTaskInterruption(issue, { run: { ...scheduled, scheduledRetryAt: new Date(now - 10 * 60_000) }, agent,
paused: false, approval: false }, now)).toMatchObject({ state: "failed", canRetry: true });
});
it("offers Retry once the scheduled retries are exhausted", () => {
expect(projectTaskInterruption(issue, { run: { ...scheduled, status: "failed", errorCode: "transient_failure_retry_exhausted",
scheduledRetryAt: new Date(now - 60_000) }, agent, paused: false, approval: false }, now))
.toMatchObject({ state: "failed", canRetry: true });
});
});
describe("hand-off of a run that failed before any provider work", () => {
const now = Date.parse("2026-11-13T15:08:12Z");
const failed = { ...run, runtimeMode: "native" as const, errorCode: "adapter_failed", finishedAt: new Date(now - 3_000),
resultJson: { failureCause: "native_predecessor_retirement_unproven", executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } };
const inProgress = { ...issue, status: "in_progress" };
it("shows the task as working while its fresh-sandbox continuation is being queued", () => {
for (const failureCause of ["native_predecessor_retirement_unproven", "native_failed_before_provider_turn"]) {
expect(projectTaskInterruption(inProgress, { run: { ...failed, resultJson: { ...failed.resultJson, failureCause } },
agent, paused: false, approval: false }, now)).toMatchObject({ state: "working", canRetry: false });
}
});
it("keeps the task working after the fallback is recorded while its continuation is still being scheduled", () => {
const recorded = { ...failed, resultJson: { ...failed.resultJson, predecessorRetirementFallback: { version: 1, providerWorkStarted: false } } };
expect(projectTaskInterruption(inProgress, { run: recorded, agent, paused: false, approval: false }, now))
.toMatchObject({ state: "working", canRetry: false });
});
it("offers Retry when the continuation was refused or never queued", () => {
const refused = { ...failed, resultJson: { ...failed.resultJson, predecessorRetirementFallback: { version: 1, providerWorkStarted: false },
continuationScheduling: "refused" } };
expect(projectTaskInterruption(inProgress, { run: refused, agent, paused: false, approval: false }, now))
.toMatchObject({ state: "failed", canRetry: true });
expect(projectTaskInterruption(inProgress, { run: { ...failed, finishedAt: new Date(now - 31_000) }, agent, paused: false, approval: false }, now))
.toMatchObject({ state: "failed", canRetry: true });
});
});