Files
PaperClipAI/ui/src/components/WorkspaceExportRecovery.control.test.tsx
T
DottaandClaude 889b33da72 fix(native): apply a saved accepted result on Retry instead of a new turn (#20819)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can finish with an accepted result,
and then fail to save its workspace. Saved-work recovery saves the files
again from the retained sandbox, with no provider turn.
> - #20758 made a save that finds the sandbox disk full stop at once as
`sandbox_disk_full` and hand the task to the board with a Retry.
> - When the user frees space and presses Retry, the save now succeeds.
But the Retry then started a new agent turn (`strategy: continue`). That
turn repeats work that already finished and spends provider budget
again. The accepted result was not applied (`replacementDenied:
explicit_user_continuation`).
> - The decision for this case: when a durable record proves that the
run's result was already accepted, Retry applies that saved result and
starts no new turn.
> - This pull request makes Retry admission wait for the run's
saved-work holds and then applies the accepted result through the
existing preserved-result finalization.
> - The benefit is that a Retry after a failed save finishes the task
from the work that was already done, with no repeated provider work or
spend.

## Linked Issues or Issue Description

Refs #20758 (full-disk save rests as `sandbox_disk_full` with a Retry),
Refs #20810 (chaos fault DF1 for a full disk on the final workspace
save)

**What happened?**
A native run had an accepted, succeeded result. Its workspace save
failed because the sandbox disk was full, and the task rested at
`sandbox_disk_full` with a Retry. The user freed space and pressed
Retry. The save succeeded, and then Retry admission started a new run
with `strategy: continue`. The accepted result stayed unapplied (the
finalization coordinator stayed in `terminal_failure`).

Two paths caused this:
1. Retry admission let the Retry through as soon as the saved files were
verified. That can happen before preserved-result finalization runs: the
recovery worker dispatches the Retry right after one hold settles, while
the hold that Retry added for the same run is still scheduled. On master
this is deterministic in that order, and a race otherwise.
2. Preserved-result finalization reads the newest saved-work hold of the
run. While that hold was still scheduled, it declined, and nothing held
the Retry back.

**Expected behavior**
After the save succeeds, and a durable record proves that the run's
result was already accepted, Retry applies that result. The run and task
finish from it. No new `turn.start` and no new provider spend. If the
save fails again with a full disk, the task keeps the
`sandbox_disk_full` rest and a working Retry. If there is no proven
accepted result, Retry keeps the previous behaviour and starts one run.

**Steps to reproduce**
1. Give a native run an accepted, succeeded result whose copy-back
failed (`native_workspace_sync_out_retry_exhausted`).
2. Make the saved-work recovery save fail with
`remote_workspace_sandbox_disk_full`. The hold goes to the board.
3. Free space. Press Retry. Let recovery save the files, then dispatch
the Retry before preserved-result finalization runs.
4. On master, a new run starts with `previousRunId` set to the source
run, and the accepted result is not applied.

## What Changed

- `preserved-result-finalization.ts`: the eligibility checks move into
one read (`readPreservedResult`). They are unchanged: an accepted,
succeeded result of this run, a completed saved-work recovery with a
verified receipt, no Stop, no newer run, no current execution, and no
applied generation. A new read-only
`canResumePreservedResultFinalization` uses the same read without locks.
`resumePreservedResultFinalization` keeps the finalizer's lock order
(coordinator first) and takes the issue and run rows with NOWAIT, so it
cannot deadlock with the finalizer, owner-stop confirmation or Retry
dispatch; a busy row is retried briefly, and the last attempt waits for
its locks. It also writes an activity entry,
`issue.saved_result_finalization_resumed`, when it applies a result.
- `explicit-native-continuation.ts`: Retry admission holds an accepted
Retry (`evaluate`) of a run whose saved files are verified and whose
accepted result is still held, with the wait `finalizing_saved_result`.
It holds while a saved-work hold of that run is still `scheduled`,
`running` or `awaiting_evidence` and the result is otherwise applicable
(no Stop, newer run, other owner or current execution), or once
`canResumePreservedResultFinalization` is true. In every other case
admission is unchanged.
- `heartbeat/run-control.ts` (`resumeTaskRetry`): after dispatch, if
admission held the Retry with `finalizing_saved_result`, it calls
`resumePreservedResultFinalization`. That coalesces the Retry into the
source run with `strategy: finalize_result` and re-arms finalization.
This is one key read after admission; the resumption runs only for a
held Retry. The recovery worker already calls the same function after
each saved hold, so either path finishes the work.
- Tests: `workspace-recovery-disk-full.test.ts` now proves the new Retry
behaviour (red on master), a dispatch that lands while another hold of
the run is still saving (red on master), a Retry while the disk is still
full, and three negative cases that keep one new run.
`workspace-recovery-preparation-gates.test.ts` (s12 and RC-50b) and
`explicit-native-continuation.test.ts` (copy-back + `task_retry`)
encoded the old "new run" result for the same situation. They now expect
the saved result to be applied. A fixture helper `recoverRunHolds` runs
the recovery steps for every hold of a run, as the worker does.
- `ui/src/components/WorkspaceExportRecovery.tsx`: at the
`sandbox_disk_full` rest, the retained-file panel no longer shows its
own "Recover saved files" button and no longer hides the task's Retry.
Before, the API said `canRetry: true` and the note said "Retry saves the
files again", but the board showed only "Recover saved files" (seen in
the DF1 chaos wave). The panel now says to free space and use Retry. A
new UI test covers it. `pnpm check:token-gates` is clean.
- `doc/execution-semantics.md`: documents the Retry behaviour and its
conditions.

## Verification

Red/green (`cd server && npx vitest run
src/services/workspace-recovery-disk-full.test.ts`):

- On master source with the new tests: 3 failed, 3 passed (the 3
negative cases pass on master, as they should). Each failure shows a
second heartbeat run (the new agent turn): `expected [ …(2) ] to deeply
equal [ Array(1) ]`.
- On this branch: 7 passed. A fourth negative case (an operator Stop
while a hold is still settling keeps one new run) was added after
review; it also passes on master.
- Guard removal 1: with the `canResumePreservedResultFinalization` check
replaced by `true`, the negative test "an operator Stop recorded on the
run" fails (`expected [] to have a length of 1 but got +0`: the Retry
waits forever instead of starting its one run).
- Guard removal 2: with the "hold still saving" check removed, the test
"holds a Retry dispatched while another saved-work hold of the run is
still settling" fails (a second run starts).

Related suites pass on this branch: `workspace-recovery-disk-full`,
`workspace-recovery-preparation-gates`, `workspace-recovery-admission`,
`native-exhausted-finalization-retry`, `explicit-native-continuation`
(641 tests).

On the first revision, `task-retry`,
`workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`,
`preserved-native-response-wait.integration`, `deferred-execution-wait`
and `deferred-execution-receipt-work` passed (317 tests). Other retry
and recovery suites were not run locally; required CI `general_tests`
covers them. After the review fixes, the five related suites above
passed again (642 tests).

`npx tsc --noEmit -p server/tsconfig.json` is clean.

`pnpm verify --explain` selects a broad plan (exit 2) because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

## Risks

- Behaviour change: an accepted Retry of a terminal run with an
accepted, succeeded result and verified saved files now applies that
result and starts no run. Before, it started one new run. This is the
intended change. It also covers the s12 and RC-50b recovery-handoff
cases, which used to start a new run.
- A held Retry waits while any saved-work hold of the run is still
saving. The recovery worker drives those holds to `complete` or
`exhausted`. When a hold ends `exhausted` or the records do not prove
the result, the wait ends and the Retry starts one run, as before. A
hold that stays in `awaiting_evidence` keeps the Retry waiting with
"Paperclip is finishing the saved result."
- Upgrade-safe: no persisted shape changes. The checks read existing
records (`native_run_results`, `native_run_finalizations`, the recovery
action evidence, workspace receipts). A record written by an older
server that lacks any required fact keeps the old behaviour. An older
server that reads the new state sees a coalesced Retry and a re-armed
finalization, which it already handles.
- No new work before `turn.start` on the normal path. Admission adds the
eligibility read only when it has already proven an accepted result and
verified saved files for a Retry of that run. `resumeTaskRetry` adds one
key read after admission.
- Lock order: `resumePreservedResultFinalization` keeps the finalizer's
order (coordinator, then issue, then run). Because it can now run beside
Retry dispatch (which locks the issue first), it takes the issue and run
rows with NOWAIT. A busy row ends the attempt, and it retries up to 9
times at 100 ms. The 10th attempt waits for its locks, as this
resumption always did, so a recovered result is never left unapplied.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 10:03:53 -06:00

590 lines
37 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
// @vitest-environment jsdom
import { act } from "react";
import { createRoot, type Root } from "react-dom/client";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { IssueRecoveryAction } from "@paperclipai/shared";
import { ApiError } from "../api/client";
import { queryKeys } from "../lib/queryKeys";
import { writePendingSend } from "../lib/pending-send";
import { WorkspaceExportRecovery } from "./WorkspaceExportRecovery";
const api = vi.hoisted(() => ({ inspect: vi.fn(), recover: vi.fn(), resume: vi.fn(), confirmController: vi.fn(), confirmOwnerGone: vi.fn() }));
vi.mock("../api/issues", () => ({ issuesApi: { recoveryActions: api.inspect, recoverWorkspace: api.recover,
resumeWorkspaceFinalization: api.resume, confirmControllerStopped: api.confirmController, confirmOwnerGone: api.confirmOwnerGone } }));
(globalThis as unknown as { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
const action = {
id: "action-1", companyId: "company-1", sourceIssueId: "issue-1",
cause: "native_workspace_sync_out_retry_exhausted", kind: "active_run_watchdog",
status: "active", ownerType: "system", wakePolicy: null,
evidence: { runId: "run-1", workspaceRecovery: {
schema: "paperclip.workspace-recovery.v1", runId: "run-1", status: "blocked",
stage: "restoring_files", attempt: 6, requestIds: [], retryable: true,
}, systemRecovery: { state: "exhausted", preservationOperationIds: [] } },
} as unknown as IssueRecoveryAction;
function inspection(value = action) {
return { active: value, actions: [value], sourceInspection: {
schema: "paperclip.workspace-source-inspection.v1", actionId: value.id, runId: "run-1",
status: "verified", scope: "recorded_source_identity", observedAt: "2026-11-03T12:00:00Z",
} };
}
describe("governed retained-file recovery", () => {
let root: Root, container: HTMLDivElement, client: QueryClient;
beforeEach(() => {
vi.resetAllMocks(); localStorage.clear();
api.inspect.mockResolvedValue(inspection());
api.recover.mockResolvedValue({ actionId: action.id, status: "queued" });
client = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: Infinity }, mutations: { retry: false } } });
client.setQueryData(queryKeys.auth.session, { user: { id: "board-1" } });
client.setQueryData(queryKeys.access.currentBoardAccess, {
userId: "board-1", companyIds: ["company-1"], source: "session", isInstanceAdmin: false,
memberships: [{ companyId: "company-1", membershipRole: "operator", status: "active" }],
});
container = document.createElement("div"); document.body.append(container); root = createRoot(container);
});
afterEach(async () => {
await act(async () => root.unmount()); container.remove(); client.clear(); localStorage.clear();
vi.restoreAllMocks();
});
async function mount(value: IssueRecoveryAction | null = action) {
const props = { action: value, issueId: "issue-1", companyId: "company-1" };
await act(async () => root.render(<QueryClientProvider client={client}>
<WorkspaceExportRecovery {...props} />
</QueryClientProvider>));
await settle();
}
async function settle() {
await act(async () => { await new Promise(resolve => setTimeout(resolve, 0)); });
}
function button(label: string) { return [...container.querySelectorAll("button")].find(b => b.textContent === label); }
async function click(label: string) {
const target = button(label); expect(target, `missing ${label}`).toBeDefined();
await act(async () => target!.click()); await settle();
}
it("offers the existing preservation operation for a stopped, exhausted system-owned export", async () => {
await mount();
await click("Recover saved files");
expect(api.inspect).toHaveBeenCalledWith("issue-1", { runId: "run-1", actionId: "action-1" });
expect(api.recover).toHaveBeenCalledTimes(1);
expect(api.recover).toHaveBeenCalledWith("issue-1", {
actionId: "action-1", runId: "run-1", idempotencyKey: expect.any(String),
});
expect(container.textContent).toContain("queued");
expect(container.textContent).not.toContain("Saved files recovered");
});
it.each(["access", "source"])("retains loaded recovery controls through a transient %s refresh failure", async kind => {
await mount();
const key = kind === "access" ? queryKeys.access.currentBoardAccess
: queryKeys.issues.recoveryActions("issue-1", "action-1", "run-1");
await act(async () => {
await client.fetchQuery({ queryKey: key, staleTime: 0, queryFn: async () => { throw new ApiError("Restarting", 503, { error: "tenant_app_unavailable" }); }, retry: false }).catch(() => {});
});
await settle();
expect(button("Recover saved files")).toBeDefined();
expect(container.querySelector('[role="alert"]')).toBeNull();
expect(container.textContent).not.toContain("tenant_app_unavailable");
expect(api.recover).not.toHaveBeenCalled();
});
it.each(["access", "source"])("hides cached recovery controls after definitive %s denial", async kind => {
await mount();
const key = kind === "access" ? queryKeys.access.currentBoardAccess
: queryKeys.issues.recoveryActions("issue-1", "action-1", "run-1");
await act(async () => {
await client.fetchQuery({ queryKey: key, staleTime: 0, queryFn: async () => { throw new ApiError("Forbidden", 403, null); }, retry: false }).catch(() => {});
});
await settle();
expect(button("Recover saved files")).toBeUndefined();
expect(container.querySelector('[role="alert"]')).not.toBeNull();
expect(api.recover).not.toHaveBeenCalled();
});
it("requests file preservation on plain HTTP without crypto.randomUUID", async () => {
const originalCrypto = globalThis.crypto;
vi.stubGlobal("crypto", { getRandomValues: originalCrypto.getRandomValues.bind(originalCrypto) });
try {
await mount(); await click("Recover saved files");
expect(api.recover).toHaveBeenCalledExactlyOnceWith("issue-1", {
actionId: "action-1", runId: "run-1", idempotencyKey: expect.stringMatching(
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/),
});
expect(container.textContent).not.toContain("Saved files recovered");
} finally { vi.unstubAllGlobals(); }
});
it.each(["viewer", "suspended", "foreign-company", "other-account"])("does not offer mutation with %s access", async kind => {
client.setQueryData(queryKeys.access.currentBoardAccess, {
userId: kind === "other-account" ? "board-2" : "board-1", source: "session", isInstanceAdmin: false,
companyIds: kind === "foreign-company" ? ["company-2"] : ["company-1"],
memberships: [{ companyId: kind === "foreign-company" ? "company-2" : "company-1",
membershipRole: kind === "viewer" ? "viewer" : "operator", status: kind === "suspended" ? "suspended" : "active" }],
});
await mount();
expect(button("Recover saved files")).toBeUndefined();
expect(api.inspect).not.toHaveBeenCalled();
expect(api.recover).not.toHaveBeenCalled();
});
it.each(["resolved", "cancelled", "foreign-company", "foreign-task", "unsafe-archive"])("does not offer a %s source", async kind => {
await mount({ ...action, ...(kind === "foreign-company" ? { companyId: "company-2" }
: kind === "foreign-task" ? { sourceIssueId: "issue-2" }
: kind === "unsafe-archive" ? { cause: "native_workspace_sync_out_unsafe_archive" }
: { status: kind }) } as IssueRecoveryAction);
expect(button("Recover saved files")).toBeUndefined();
expect(api.recover).not.toHaveBeenCalled();
});
it("rejects a source/action/run binding that changed after the control was rendered", async () => {
api.inspect.mockResolvedValueOnce(inspection()).mockResolvedValueOnce({
...inspection(), sourceInspection: { ...inspection().sourceInspection, runId: "foreign-run" },
});
await mount(); await click("Recover saved files");
expect(api.recover).not.toHaveBeenCalled();
expect(container.textContent).toContain("File recovery changed");
});
it("does not offer an unverified source", async () => {
api.inspect.mockResolvedValue({ ...inspection(), sourceInspection: {
...inspection().sourceInspection, status: "blocked", failureCode: "source_binding_invalid",
} });
await mount();
expect(button("Recover saved files")).toBeUndefined();
expect(container.textContent).toContain("can’t verify the saved files’ recorded source (source_binding_invalid)");
});
// Recovery saved by an older release: a board-owned hold whose
// sandbox predates ownership receipts. The server still records a retryable
// blocked recovery, but the read-only source probe cannot verify ownership.
const legacyHold = { ...action, id: "legacy-hold", ownerType: "board", evidence: { runId: "run-1",
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: "run-1", stage: "verifying_stop",
status: "blocked", attempt: 1, attemptLimit: 6, requestIds: [], retryable: true,
message: "Automatic recovery of saved work stopped after 1 attempts (last failure: allocation_ownership_ambiguous)." },
systemRecovery: { schema: "paperclip.system-recovery.v1", phase: "verify_execution", state: "exhausted", revision: 5,
sourceRunId: "run-1", attempts: 1, retriesUsed: 0, preservationOperationIds: [], freshAttemptRunId: null,
failure: { code: "allocation_ownership_ambiguous", retryClass: "service_required" } },
originalFailureCode: "native_workspace_sync_out_failed", workspaceFinalizeAttempt: 3, previousWorkspaceRecovery: null },
} as unknown as IssueRecoveryAction;
const blockedInspection = (value: IssueRecoveryAction, failureCode = "allocation_ownership_ambiguous") => ({
active: value, actions: [value], sourceInspection: { schema: "paperclip.workspace-source-inspection.v1",
actionId: value.id, runId: "run-1", status: "blocked", failureCode,
scope: "recorded_source_identity", observedAt: "2026-11-07T14:44:46Z" } });
it("hands the next action to the task Retry for an old-release hold whose source cannot be verified", async () => {
api.inspect.mockResolvedValue(blockedInspection(legacyHold));
const owns = vi.fn();
await act(async () => root.render(<QueryClientProvider client={client}>
<WorkspaceExportRecovery action={legacyHold} issueId="issue-1" companyId="company-1" onRecoveryControlChange={owns} />
</QueryClientProvider>));
await settle();
expect(owns).toHaveBeenLastCalledWith(false);
expect(button("Recover saved files")).toBeUndefined();
// No dead-end: no "changed, refresh" alert and no status check that can only repeat it.
expect(button("Check file recovery status")).toBeUndefined();
expect(container.textContent).not.toContain("has changed");
expect(container.textContent).toContain("can’t confirm that this run alone used the sandbox");
expect(api.recover).not.toHaveBeenCalled();
});
it("does not request recovery when the source stops verifying between render and click", async () => {
api.inspect.mockResolvedValueOnce(inspection()).mockResolvedValueOnce({ ...inspection(), sourceInspection: {
...inspection().sourceInspection, status: "blocked", failureCode: "allocation_ownership_ambiguous" } });
await mount();
await click("Recover saved files");
expect(api.recover).not.toHaveBeenCalled();
expect(container.textContent).toContain("can’t confirm that this run alone used the sandbox");
expect(container.textContent).not.toContain("File recovery changed");
expect(button("Recover saved files")).toBeUndefined();
});
it("keeps naming the unverified source after an accepted request failed", async () => {
const key = "accepted-request";
writePendingSend("workspace-file-recovery:company-1:issue-1:legacy-hold:run-1:board-1",
{ idempotencyKey: key, payload: { actionId: "legacy-hold", runId: "run-1" }, createdAt: Date.now() });
const failed = { ...legacyHold, evidence: { ...legacyHold.evidence, workspaceRecovery: {
...(legacyHold.evidence.workspaceRecovery as object), requestIds: [key] } } } as IssueRecoveryAction;
api.inspect.mockResolvedValue(blockedInspection(failed));
await mount(failed);
expect(container.textContent).toContain("File recovery has not finished");
expect(container.textContent).toContain("can’t confirm that this run alone used the sandbox");
// While recovery is running, the panel shows progress instead of the reason.
const running = { ...failed, evidence: { ...failed.evidence, workspaceRecovery: {
...(failed.evidence.workspaceRecovery as object), status: "recovering" } } } as IssueRecoveryAction;
api.inspect.mockResolvedValue(blockedInspection(running));
await act(async () => client.invalidateQueries({ queryKey: queryKeys.issues.recoveryActions("issue-1", "legacy-hold", "run-1") }));
await settle();
expect(container.textContent).toContain("Recovering saved files");
expect(container.textContent).not.toContain("can’t confirm that this run alone used the sandbox");
expect(api.recover).not.toHaveBeenCalled();
});
it("keeps ownership of the next action while the source is verified", async () => {
const owns = vi.fn();
await act(async () => root.render(<QueryClientProvider client={client}>
<WorkspaceExportRecovery action={action} issueId="issue-1" companyId="company-1" onRecoveryControlChange={owns} />
</QueryClientProvider>));
await settle();
expect(owns).toHaveBeenLastCalledWith(true);
expect(button("Recover saved files")).toBeDefined();
});
it("reports recovered files even when the source no longer matches after recovery", async () => {
// Preservation completed, so the workspace moved past the
// recorded source and inspection reports a reference mismatch.
const recovered = { ...legacyHold, ownerType: "system", evidence: { ...legacyHold.evidence,
workspaceRecovery: { ...(legacyHold.evidence.workspaceRecovery as object), status: "recovered" },
systemRecovery: { ...(legacyHold.evidence.systemRecovery as object), state: "complete",
preservationOperationIds: ["preservation-1"] } } } as IssueRecoveryAction;
api.inspect.mockResolvedValue(blockedInspection(recovered, "source_binding_invalid"));
await mount(recovered);
expect(container.textContent).toContain("Saved files recovered");
expect(container.textContent).not.toContain("need recovery");
expect(container.textContent).not.toContain("has changed");
expect(button("Recover saved files")).toBeUndefined();
});
it("reloads the task instead of looping when the recovery record was replaced", async () => {
api.inspect.mockResolvedValue({ active: null, actions: [], sourceInspection: undefined });
const invalidate = vi.spyOn(client, "invalidateQueries");
await mount(legacyHold);
await click("Check file recovery status");
expect(container.textContent).not.toContain("Refresh the task");
expect(invalidate).toHaveBeenCalledWith({ queryKey: queryKeys.issues.detail("issue-1") });
});
it("persists the exact request before POST and prevents a double click", async () => {
let finish!: (value: unknown) => void;
api.recover.mockImplementation((_issue, body) => {
expect(localStorage.getItem("workspace-file-recovery:company-1:issue-1:action-1:run-1:board-1")).toContain(body.idempotencyKey);
return new Promise(resolve => { finish = resolve; });
});
await mount();
const target = button("Recover saved files")!;
await act(async () => { target.click(); target.click(); }); await settle();
expect(api.recover).toHaveBeenCalledTimes(1);
await act(async () => finish({ actionId: action.id, status: "queued" })); await settle();
expect(button("Recover saved files")).toBeUndefined();
expect(container.textContent).not.toContain("Saved files recovered");
});
it("reconciles a lost accepted response through GET without another POST", async () => {
let current = action;
api.inspect.mockImplementation(async () => inspection(current));
api.recover.mockImplementation(async (_issue, body) => {
current = { ...action, evidence: { ...action.evidence, workspaceRecovery: {
...(action.evidence.workspaceRecovery as object), status: "queued", requestIds: [body.idempotencyKey],
} } };
throw Object.assign(new Error("Network reply lost"), { status: 503 });
});
await mount(); await click("Recover saved files");
expect(container.textContent).toContain("could not be confirmed");
await click("Check file recovery status");
expect(api.recover).toHaveBeenCalledTimes(1);
expect(container.textContent).toContain("Recovering saved files");
expect(container.textContent).not.toContain("could not be confirmed");
expect(button("Recover saved files")).toBeUndefined();
});
it("reuses the stored key after remount and a GET that has no acceptance receipt", async () => {
api.recover.mockRejectedValueOnce(Object.assign(new Error("Network unavailable"), { status: 503 }));
await mount(); await click("Recover saved files");
const first = api.recover.mock.calls[0][1].idempotencyKey;
await act(async () => root.render(null));
await mount();
expect(button("Recover saved files")).toBeUndefined();
await click("Check file recovery status");
await click("Recover saved files");
expect(api.recover).toHaveBeenCalledTimes(2);
expect(api.recover.mock.calls[1][1].idempotencyKey).toBe(first);
});
it("does not POST when durable storage is unavailable", async () => {
vi.spyOn(Storage.prototype, "setItem").mockImplementation(() => { throw new Error("storage denied"); });
await mount(); await click("Recover saved files");
expect(api.recover).not.toHaveBeenCalled();
expect(container.textContent).toContain("No request was sent");
});
it("does not send an old account's request after a source read completes", async () => {
let finish!: (value: unknown) => void;
api.inspect.mockResolvedValueOnce(inspection()).mockImplementationOnce(() => new Promise(resolve => { finish = resolve; }));
await mount(); await click("Recover saved files");
await act(async () => client.setQueryData(queryKeys.auth.session, { user: { id: "board-2" } }));
await act(async () => finish(inspection())); await settle();
expect(api.recover).not.toHaveBeenCalled();
});
it("shows a confirmed blocked attempt as failed preservation, not queued or recovered", async () => {
let current = action;
api.inspect.mockImplementation(async () => inspection(current));
api.recover.mockImplementation(async (_issue, body) => {
current = { ...action, evidence: { ...action.evidence, workspaceRecovery: {
...(action.evidence.workspaceRecovery as object), requestIds: [body.idempotencyKey],
} } };
return { actionId: action.id, status: "queued" };
});
await mount(); await click("Recover saved files");
expect(container.textContent).toContain("File recovery has not finished");
expect(container.textContent).not.toContain("File recovery is queued");
expect(container.textContent).not.toContain("Saved files recovered");
expect(api.recover).toHaveBeenCalledTimes(1);
});
it("offers a new explicit request only after the prior accepted attempt is confirmed exhausted", async () => {
let current = action;
const accepted: string[] = [];
api.inspect.mockImplementation(async () => inspection(current));
api.recover.mockImplementation(async (_issue, body) => {
accepted.push(body.idempotencyKey);
current = { ...action, evidence: { ...action.evidence, workspaceRecovery: {
...(action.evidence.workspaceRecovery as object), requestIds: [...accepted],
} } };
return { actionId: action.id, status: "queued" };
});
await mount(); await click("Recover saved files");
expect(container.textContent).toContain("File recovery has not finished");
expect(api.recover).toHaveBeenCalledTimes(1);
await click("Try file recovery again");
expect(api.recover).toHaveBeenCalledTimes(2);
expect(accepted[1]).not.toBe(accepted[0]);
expect(container.textContent).not.toContain("Saved files recovered");
});
it("does not replace the confirmed failed key if another recovery is active at the fresh read", async () => {
let current = action;
api.inspect.mockImplementation(async () => inspection(current));
api.recover.mockImplementation(async (_issue, body) => {
current = { ...action, evidence: { ...action.evidence, workspaceRecovery: {
...(action.evidence.workspaceRecovery as object), requestIds: [body.idempotencyKey],
} } };
return { actionId: action.id, status: "queued" };
});
await mount(); await click("Recover saved files");
const previous = api.recover.mock.calls[0][1].idempotencyKey;
current = { ...current, evidence: { ...current.evidence, workspaceRecovery: {
...(current.evidence.workspaceRecovery as object), status: "recovering",
} } };
await click("Try file recovery again");
expect(api.recover).toHaveBeenCalledTimes(1);
expect(localStorage.getItem("workspace-file-recovery:company-1:issue-1:action-1:run-1:board-1")).toContain(previous);
});
it("hands a full-disk save rest to the task's Retry instead of its own recovery button", async () => {
const diskFull = { ...action, ownerType: "board", evidence: { ...action.evidence, systemRecovery: {
state: "exhausted", preservationOperationIds: [], failure: { code: "export_interrupted",
retryClass: "user_action_required", cause: { label: "sandbox_disk_full", code: "ENOSPC", status: null } } } },
} as unknown as IssueRecoveryAction;
api.inspect.mockResolvedValue(inspection(diskFull));
const owns = vi.fn();
await act(async () => root.render(<QueryClientProvider client={client}>
<WorkspaceExportRecovery action={diskFull} issueId="issue-1" companyId="company-1" onRecoveryControlChange={owns} />
</QueryClientProvider>));
await settle();
expect(button("Recover saved files")).toBeUndefined();
expect(button("Try file recovery again")).toBeUndefined();
expect(owns).toHaveBeenLastCalledWith(false);
expect(owns).not.toHaveBeenCalledWith(true);
expect(container.textContent).toContain("use Retry to save them");
expect(api.recover).not.toHaveBeenCalled();
});
it("only reports recovered files after the durable preservation receipt is present", async () => {
const recovered = { ...action, evidence: { ...action.evidence, workspaceRecovery: {
...(action.evidence.workspaceRecovery as object), status: "recovered",
}, systemRecovery: { state: "complete", preservationOperationIds: [] } } };
api.inspect.mockResolvedValue(inspection(recovered)); await mount();
expect(container.textContent).not.toContain("Saved files recovered");
api.inspect.mockResolvedValue(inspection({ ...recovered, evidence: { ...recovered.evidence,
systemRecovery: { state: "complete", preservationOperationIds: ["host-preservation-operation"] },
} }));
await act(async () => client.invalidateQueries({ queryKey: queryKeys.issues.recoveryActions("issue-1", "action-1", "run-1") }));
await settle();
expect(container.textContent).toContain("Saved files recovered");
expect(api.recover).not.toHaveBeenCalled();
});
// Recovery is automatic. No recovery state may tell a person to fix or inspect
// a sandbox, or name an owner or administrator as the one who must act.
const manualCopy = /administrator|\bowner\b|operator assistance|service assistance|service attention|inspect the (run|failed run)|environment/i;
const withRecovery = (value: IssueRecoveryAction, patch: Record<string, unknown>, system?: Record<string, unknown>) => ({
...value, evidence: { ...value.evidence, workspaceRecovery: { ...(value.evidence.workspaceRecovery as object), ...patch },
...(system ? { systemRecovery: { ...(value.evidence.systemRecovery as object), ...system } } : {}) } }) as IssueRecoveryAction;
const operatorAccess = { userId: "board-1", companyIds: ["company-1"], source: "session", isInstanceAdmin: false,
memberships: [{ companyId: "company-1", membershipRole: "operator", status: "active" }] };
const viewerAccess = { ...operatorAccess, memberships: [{ companyId: "company-1", membershipRole: "viewer", status: "active" }] };
it.each([
["verified and retryable", action, inspection(action), operatorAccess],
["unverified ownership", legacyHold, blockedInspection(legacyHold), operatorAccess],
["unverified binding", legacyHold, blockedInspection(legacyHold, "source_binding_invalid"), operatorAccess],
["not inspected", legacyHold, { ...blockedInspection(legacyHold), sourceInspection: { ...blockedInspection(legacyHold).sourceInspection,
status: "not_inspected", reason: "source_limit" } }, operatorAccess],
["queued", withRecovery(legacyHold, { status: "queued" }), blockedInspection(withRecovery(legacyHold, { status: "queued" })), operatorAccess],
["not retryable", withRecovery(legacyHold, { retryable: false }), blockedInspection(withRecovery(legacyHold, { retryable: false })), operatorAccess],
["recovered", withRecovery(legacyHold, { status: "recovered" }, { state: "complete", preservationOperationIds: ["op-1"] }),
blockedInspection(withRecovery(legacyHold, { status: "recovered" }, { state: "complete", preservationOperationIds: ["op-1"] })), operatorAccess],
["viewer", legacyHold, blockedInspection(legacyHold), viewerAccess],
["copyback owner hold for an operator", { ...action, evidence: { ...action.evidence,
workspaceOwnerRecovery: { state: "operator_required", owner: { token: "token" } } } } as IssueRecoveryAction, inspection(action), operatorAccess],
["board-owned status", { ...action, cause: "native_workspace_finalization_owner_unverified", ownerType: "board",
evidence: { runId: "run-1" } } as unknown as IssueRecoveryAction, inspection(action), operatorAccess],
] as const)("renders no manual sandbox or administrator copy: %s", async (_name, value, response, access) => {
client.setQueryData(queryKeys.access.currentBoardAccess, access);
api.inspect.mockResolvedValue(response);
await mount(value);
expect(container.textContent).not.toMatch(manualCopy);
});
describe("previous server stop confirmation", () => {
const ownerAction = { ...action, cause: "native_workspace_finalization_owner_unverified", ownerType: "board", status: "escalated",
evidence: { runId: "run-1", owner: { token: "owner-token", hostname: "secret-host", pid: 2 },
workspaceOwnerRecovery: { state: "operator_required" } } } as unknown as IssueRecoveryAction;
it("offers one board action that resumes the saved result with the board's stop evidence", async () => {
api.resume.mockResolvedValue({ runId: "run-1", resultId: "result-1", status: "queued" });
client.setQueryData(queryKeys.access.currentBoardAccess, { userId: "board-1", source: "session", isInstanceAdmin: false,
companyIds: ["company-1"], memberships: [{ companyId: "company-1", membershipRole: "owner", status: "active" }] });
await mount(ownerAction);
expect(container.textContent).toContain("can’t verify that it stopped");
expect(container.textContent).not.toContain("secret-host");
await click("Confirm the previous server stopped");
const submit = button("Confirm and resume")!;
expect(submit.disabled).toBe(true);
const textarea = container.querySelector("textarea")!;
await act(async () => {
const setter = Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!;
setter.call(textarea, "The old server process exited and was not restarted.");
textarea.dispatchEvent(new Event("input", { bubbles: true }));
});
await click("Confirm and resume");
expect(api.resume).toHaveBeenCalledExactlyOnceWith("issue-1", { actionId: "action-1", runId: "run-1",
ownerToken: "owner-token", controllerAndCopybackStopped: true,
stopEvidence: "The old server process exited and was not restarted." });
expect(container.textContent).toContain("Restoring saved work…");
});
it("offers the same action for retained saved-work recovery held by a stopped copyback owner", async () => {
api.resume.mockResolvedValue({ runId: "run-1", resultId: null, status: "queued" });
client.setQueryData(queryKeys.access.currentBoardAccess, { userId: "board-1", source: "session", isInstanceAdmin: false,
companyIds: ["company-1"], memberships: [{ companyId: "company-1", membershipRole: "admin", status: "active" }] });
const held = { ...action, evidence: { ...action.evidence,
workspaceOwnerRecovery: { state: "operator_required", owner: { token: "retained-token", hostname: "secret-host", pid: 9 } } } } as IssueRecoveryAction;
await mount(held);
expect(button("Recover saved files")).toBeUndefined();
expect(container.textContent).not.toContain("secret-host");
await click("Confirm the previous server stopped");
const textarea = container.querySelector("textarea")!;
await act(async () => {
const setter = Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!;
setter.call(textarea, "The old server process exited and was not restarted.");
textarea.dispatchEvent(new Event("input", { bubbles: true }));
});
await click("Confirm and resume");
expect(api.resume).toHaveBeenCalledExactlyOnceWith("issue-1", { actionId: "action-1", runId: "run-1",
ownerToken: "retained-token", controllerAndCopybackStopped: true,
stopEvidence: "The old server process exited and was not restarted." });
// Once confirmed or resuming, the ordinary retained recovery controls return.
await mount({ ...held, evidence: { ...held.evidence, workspaceOwnerRecovery: { state: "resuming" } } } as IssueRecoveryAction);
expect(button("Confirm the previous server stopped")).toBeUndefined();
expect(container.querySelector('[data-testid="retained-file-recovery"]')).not.toBeNull();
});
it("confirms a pre-upgrade controller an ephemeral server cannot verify", async () => {
api.confirmController.mockResolvedValue({ runId: "run-1", status: "queued" });
client.setQueryData(queryKeys.access.currentBoardAccess, { userId: "board-1", source: "session", isInstanceAdmin: false,
companyIds: ["company-1"], memberships: [{ companyId: "company-1", membershipRole: "owner", status: "active" }] });
const controllerAction = { ...action, cause: "native_controller_host_unverified", ownerType: "board",
evidence: { runId: "run-1", controller: { bootId: "boot", pid: 7 } } } as unknown as IssueRecoveryAction;
await mount(controllerAction);
expect(container.textContent).toContain("before Paperclip stored which host it ran on");
await click("Confirm the previous server stopped");
const textarea = container.querySelector("textarea")!;
await act(async () => {
const setter = Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!;
setter.call(textarea, "The old server container was replaced and is not running.");
textarea.dispatchEvent(new Event("input", { bubbles: true }));
});
await click("Confirm and resume");
expect(api.confirmController).toHaveBeenCalledExactlyOnceWith("issue-1", { actionId: "action-1", runId: "run-1",
stopEvidence: "The old server container was replaced and is not running." });
expect(api.resume).not.toHaveBeenCalled();
expect(container.textContent).toContain("Controller stop recorded. Checking recovery…");
});
it("confirms the unrecorded server holding a stuck resource, for board owners only", async () => {
api.confirmOwnerGone.mockResolvedValue({ recordId: "lease-1", status: "queued" });
const ownerGoneAction = { ...action, cause: "ownership_owner_unprovable", ownerType: "board",
evidence: { kind: "environment_run_lease_active", recordId: "lease-1", runId: "run-1" } } as unknown as IssueRecoveryAction;
// An operator sees the wait but cannot attest the stop.
await mount(ownerGoneAction);
expect(container.textContent).toContain("before Paperclip stored which server held it");
expect(button("Confirm the previous server stopped")).toBeUndefined();
client.setQueryData(queryKeys.access.currentBoardAccess, { userId: "board-1", source: "session", isInstanceAdmin: false,
companyIds: ["company-1"], memberships: [{ companyId: "company-1", membershipRole: "owner", status: "active" }] });
await mount(ownerGoneAction);
await click("Confirm the previous server stopped");
const textarea = container.querySelector("textarea")!;
await act(async () => {
const setter = Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!;
setter.call(textarea, "The old server container was replaced and is not running.");
textarea.dispatchEvent(new Event("input", { bubbles: true }));
});
await click("Confirm and resume");
expect(api.confirmOwnerGone).toHaveBeenCalledExactlyOnceWith("issue-1", { actionId: "action-1", recordId: "lease-1",
stopEvidence: "The old server container was replaced and is not running." });
expect(container.textContent).toContain("Releasing the sandbox lease…");
});
it("resets an open confirmation when the request is refreshed to a newer record version", async () => {
api.confirmOwnerGone.mockResolvedValue({ recordId: "run-1", status: "queued" });
client.setQueryData(queryKeys.access.currentBoardAccess, { userId: "board-1", source: "session", isInstanceAdmin: false,
companyIds: ["company-1"], memberships: [{ companyId: "company-1", membershipRole: "owner", status: "active" }] });
const at = (version: string) => ({ ...action, cause: "ownership_owner_unprovable", ownerType: "board",
evidence: { kind: "legacy_controller_lease", recordId: "run-1", runId: "run-1", recordUpdatedAt: version } }) as unknown as IssueRecoveryAction;
await mount(at("2026-11-08T10:00:00.000Z"));
await click("Confirm the previous server stopped");
expect(container.querySelector("textarea")).not.toBeNull();
// The record changed and the request now names its new version: the open form starts over.
await mount(at("2026-11-08T10:05:00.000Z"));
expect(container.querySelector("textarea")).toBeNull();
await click("Confirm the previous server stopped");
const textarea = container.querySelector("textarea")!;
await act(async () => {
const setter = Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!;
setter.call(textarea, "The old server container was replaced and is not running.");
textarea.dispatchEvent(new Event("input", { bubbles: true }));
});
await click("Confirm and resume");
expect(api.confirmOwnerGone).toHaveBeenCalledExactlyOnceWith("issue-1", { actionId: "action-1", recordId: "run-1",
stopEvidence: "The old server container was replaced and is not running.", recordVersion: "2026-11-08T10:05:00.000Z" });
});
it("names the stuck record for a kind confirmed by record version, and a generic resource otherwise", async () => {
client.setQueryData(queryKeys.access.currentBoardAccess, { userId: "board-1", source: "session", isInstanceAdmin: false,
companyIds: ["company-1"], memberships: [{ companyId: "company-1", membershipRole: "owner", status: "active" }] });
const ownerGone = (kind: string) => ({ ...action, cause: "ownership_owner_unprovable", ownerType: "board",
evidence: { kind, recordId: "run-1", runId: "run-1", recordUpdatedAt: "2026-11-08T10:00:00.000Z" } }) as unknown as IssueRecoveryAction;
await mount(ownerGone("legacy_controller_lease"));
expect(container.textContent).toContain("This task’s earlier run was recorded before Paperclip stored which server held it");
expect(button("Confirm the previous server stopped")).toBeDefined();
await mount(ownerGone("a_future_kind"));
expect(container.textContent).toContain("A resource from this task’s earlier run was recorded");
});
it("shows status only, without the action, once automatic recovery is resuming or for non-administrators", async () => {
// The default fixture is an operator: it may resolve other recovery but not attest a controller stop.
await mount({ ...ownerAction, evidence: { ...ownerAction.evidence, workspaceOwnerRecovery: { state: "resuming" } } } as IssueRecoveryAction);
expect(button("Confirm the previous server stopped")).toBeUndefined();
client.setQueryData(queryKeys.access.currentBoardAccess, { userId: "board-1", source: "session", isInstanceAdmin: false,
companyIds: ["company-1"], memberships: [{ companyId: "company-1", membershipRole: "operator", status: "active" }] });
await mount(ownerAction);
expect(button("Confirm the previous server stopped")).toBeUndefined();
expect(container.textContent).toContain("waiting to confirm that the previous server stopped");
expect(api.resume).not.toHaveBeenCalled();
});
});
});