Files
PaperClipAI/ui/src/components/WorkspaceExportRecovery.tsx
T
DottaandClaude 889b33da72 fix(native): apply a saved accepted result on Retry instead of a new turn (#20819)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can finish with an accepted result,
and then fail to save its workspace. Saved-work recovery saves the files
again from the retained sandbox, with no provider turn.
> - #20758 made a save that finds the sandbox disk full stop at once as
`sandbox_disk_full` and hand the task to the board with a Retry.
> - When the user frees space and presses Retry, the save now succeeds.
But the Retry then started a new agent turn (`strategy: continue`). That
turn repeats work that already finished and spends provider budget
again. The accepted result was not applied (`replacementDenied:
explicit_user_continuation`).
> - The decision for this case: when a durable record proves that the
run's result was already accepted, Retry applies that saved result and
starts no new turn.
> - This pull request makes Retry admission wait for the run's
saved-work holds and then applies the accepted result through the
existing preserved-result finalization.
> - The benefit is that a Retry after a failed save finishes the task
from the work that was already done, with no repeated provider work or
spend.

## Linked Issues or Issue Description

Refs #20758 (full-disk save rests as `sandbox_disk_full` with a Retry),
Refs #20810 (chaos fault DF1 for a full disk on the final workspace
save)

**What happened?**
A native run had an accepted, succeeded result. Its workspace save
failed because the sandbox disk was full, and the task rested at
`sandbox_disk_full` with a Retry. The user freed space and pressed
Retry. The save succeeded, and then Retry admission started a new run
with `strategy: continue`. The accepted result stayed unapplied (the
finalization coordinator stayed in `terminal_failure`).

Two paths caused this:
1. Retry admission let the Retry through as soon as the saved files were
verified. That can happen before preserved-result finalization runs: the
recovery worker dispatches the Retry right after one hold settles, while
the hold that Retry added for the same run is still scheduled. On master
this is deterministic in that order, and a race otherwise.
2. Preserved-result finalization reads the newest saved-work hold of the
run. While that hold was still scheduled, it declined, and nothing held
the Retry back.

**Expected behavior**
After the save succeeds, and a durable record proves that the run's
result was already accepted, Retry applies that result. The run and task
finish from it. No new `turn.start` and no new provider spend. If the
save fails again with a full disk, the task keeps the
`sandbox_disk_full` rest and a working Retry. If there is no proven
accepted result, Retry keeps the previous behaviour and starts one run.

**Steps to reproduce**
1. Give a native run an accepted, succeeded result whose copy-back
failed (`native_workspace_sync_out_retry_exhausted`).
2. Make the saved-work recovery save fail with
`remote_workspace_sandbox_disk_full`. The hold goes to the board.
3. Free space. Press Retry. Let recovery save the files, then dispatch
the Retry before preserved-result finalization runs.
4. On master, a new run starts with `previousRunId` set to the source
run, and the accepted result is not applied.

## What Changed

- `preserved-result-finalization.ts`: the eligibility checks move into
one read (`readPreservedResult`). They are unchanged: an accepted,
succeeded result of this run, a completed saved-work recovery with a
verified receipt, no Stop, no newer run, no current execution, and no
applied generation. A new read-only
`canResumePreservedResultFinalization` uses the same read without locks.
`resumePreservedResultFinalization` keeps the finalizer's lock order
(coordinator first) and takes the issue and run rows with NOWAIT, so it
cannot deadlock with the finalizer, owner-stop confirmation or Retry
dispatch; a busy row is retried briefly, and the last attempt waits for
its locks. It also writes an activity entry,
`issue.saved_result_finalization_resumed`, when it applies a result.
- `explicit-native-continuation.ts`: Retry admission holds an accepted
Retry (`evaluate`) of a run whose saved files are verified and whose
accepted result is still held, with the wait `finalizing_saved_result`.
It holds while a saved-work hold of that run is still `scheduled`,
`running` or `awaiting_evidence` and the result is otherwise applicable
(no Stop, newer run, other owner or current execution), or once
`canResumePreservedResultFinalization` is true. In every other case
admission is unchanged.
- `heartbeat/run-control.ts` (`resumeTaskRetry`): after dispatch, if
admission held the Retry with `finalizing_saved_result`, it calls
`resumePreservedResultFinalization`. That coalesces the Retry into the
source run with `strategy: finalize_result` and re-arms finalization.
This is one key read after admission; the resumption runs only for a
held Retry. The recovery worker already calls the same function after
each saved hold, so either path finishes the work.
- Tests: `workspace-recovery-disk-full.test.ts` now proves the new Retry
behaviour (red on master), a dispatch that lands while another hold of
the run is still saving (red on master), a Retry while the disk is still
full, and three negative cases that keep one new run.
`workspace-recovery-preparation-gates.test.ts` (s12 and RC-50b) and
`explicit-native-continuation.test.ts` (copy-back + `task_retry`)
encoded the old "new run" result for the same situation. They now expect
the saved result to be applied. A fixture helper `recoverRunHolds` runs
the recovery steps for every hold of a run, as the worker does.
- `ui/src/components/WorkspaceExportRecovery.tsx`: at the
`sandbox_disk_full` rest, the retained-file panel no longer shows its
own "Recover saved files" button and no longer hides the task's Retry.
Before, the API said `canRetry: true` and the note said "Retry saves the
files again", but the board showed only "Recover saved files" (seen in
the DF1 chaos wave). The panel now says to free space and use Retry. A
new UI test covers it. `pnpm check:token-gates` is clean.
- `doc/execution-semantics.md`: documents the Retry behaviour and its
conditions.

## Verification

Red/green (`cd server && npx vitest run
src/services/workspace-recovery-disk-full.test.ts`):

- On master source with the new tests: 3 failed, 3 passed (the 3
negative cases pass on master, as they should). Each failure shows a
second heartbeat run (the new agent turn): `expected [ …(2) ] to deeply
equal [ Array(1) ]`.
- On this branch: 7 passed. A fourth negative case (an operator Stop
while a hold is still settling keeps one new run) was added after
review; it also passes on master.
- Guard removal 1: with the `canResumePreservedResultFinalization` check
replaced by `true`, the negative test "an operator Stop recorded on the
run" fails (`expected [] to have a length of 1 but got +0`: the Retry
waits forever instead of starting its one run).
- Guard removal 2: with the "hold still saving" check removed, the test
"holds a Retry dispatched while another saved-work hold of the run is
still settling" fails (a second run starts).

Related suites pass on this branch: `workspace-recovery-disk-full`,
`workspace-recovery-preparation-gates`, `workspace-recovery-admission`,
`native-exhausted-finalization-retry`, `explicit-native-continuation`
(641 tests).

On the first revision, `task-retry`,
`workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`,
`preserved-native-response-wait.integration`, `deferred-execution-wait`
and `deferred-execution-receipt-work` passed (317 tests). Other retry
and recovery suites were not run locally; required CI `general_tests`
covers them. After the review fixes, the five related suites above
passed again (642 tests).

`npx tsc --noEmit -p server/tsconfig.json` is clean.

`pnpm verify --explain` selects a broad plan (exit 2) because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

## Risks

- Behaviour change: an accepted Retry of a terminal run with an
accepted, succeeded result and verified saved files now applies that
result and starts no run. Before, it started one new run. This is the
intended change. It also covers the s12 and RC-50b recovery-handoff
cases, which used to start a new run.
- A held Retry waits while any saved-work hold of the run is still
saving. The recovery worker drives those holds to `complete` or
`exhausted`. When a hold ends `exhausted` or the records do not prove
the result, the wait ends and the Retry starts one run, as before. A
hold that stays in `awaiting_evidence` keeps the Retry waiting with
"Paperclip is finishing the saved result."
- Upgrade-safe: no persisted shape changes. The checks read existing
records (`native_run_results`, `native_run_finalizations`, the recovery
action evidence, workspace receipts). A record written by an older
server that lacks any required fact keeps the old behaviour. An older
server that reads the new state sees a coalesced Retry and a re-armed
finalization, which it already handles.
- No new work before `turn.start` on the normal path. Admission adds the
eligibility read only when it has already proven an accepted result and
verified saved files for a Retry of that run. `resumeTaskRetry` adds one
key read after admission.
- Lock order: `resumePreservedResultFinalization` keeps the finalizer's
order (coordinator, then issue, then run). Because it can now run beside
Retry dispatch (which locks the issue first), it takes the issue and run
rows with NOWAIT. A busy row ends the attempt, and it retries up to 9
times at 100 ms. The 10th attempt waits for its locks, as this
resumption always did, so a recovered result is never left unapplied.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 10:03:53 -06:00

396 lines
28 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { useLayoutEffect, useRef, useState } from "react";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import type { IssueRecoveryAction, WorkspaceSourceInspection } from "@paperclipai/shared";
import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared";
import { isTransientError } from "../api/errors";
import { useQueryView } from "./QueryView";
import { accessApi } from "../api/access";
import { issuesApi } from "../api/issues";
import { useDraftOwner, useDraftOwnerReader } from "../hooks/useDraftOwner";
import { canBoardConfirmOwnerStop, canBoardResolveRecoveryAction, isRetainedWorkspaceExportCause } from "../lib/recovery-reconcile";
import { queryKeys } from "../lib/queryKeys";
import { classifySendFailure, pendingSendKeyFor, readPendingSend, writePendingSend } from "../lib/pending-send";
import { asObject, cn } from "../lib/utils";
import { createUuid } from "../lib/uuid";
import { Button } from "./ui/button";
import { Textarea } from "./ui/textarea";
/** Reports whether this surface owns the task's next recovery action. The task's
* generic Retry stays hidden only while this surface offers its own control. */
function ReportRecoveryControl({ owns, onChange }: { owns: boolean; onChange?: (owns: boolean) => void }) {
useLayoutEffect(() => {
onChange?.(owns);
return () => onChange?.(false);
}, [owns, onChange]);
return null;
}
const CONTROLLER_HOST_UNVERIFIED = "native_controller_host_unverified";
const OWNER_UNPROVABLE = "ownership_owner_unprovable";
/** What the stuck record is, by ownership kind, for the board's confirmation. */
const OWNER_GONE_RESOURCE: Record<string, { what: string; releasing: string }> = {
environment_run_lease_active: { what: "A sandbox lease from this task’s earlier run", releasing: "Releasing the sandbox lease…" },
legacy_controller_lease: { what: "This task’s earlier run", releasing: "Recovering the run…" },
environment_teardown_claim: { what: "A sandbox cleanup from this task’s earlier run", releasing: "Resuming the sandbox cleanup…" },
sandbox_creation_intent: { what: "A sandbox this task’s earlier run was creating", releasing: "Reconciling the sandbox…" },
workspace_runtime_control_operation: { what: "A workspace service change from this task’s earlier run", releasing: "Releasing the workspace…" },
native_workspace_finalize_attempt: { what: "A workspace save from this task’s earlier run", releasing: "Releasing the workspace save…" },
native_finalizer_lease: { what: "The result handling of this task’s earlier run", releasing: "Resuming the result handling…" },
native_cleanup_lease: { what: "The cleanup of this task’s earlier run", releasing: "Resuming the cleanup…" },
};
const OWNER_GONE_DEFAULT = { what: "A resource from this task’s earlier run", releasing: "Releasing the resource…" };
/** Causes whose one action is the board's previous-server stop confirmation;
* the generic recovery card must not offer other resolutions for them. */
export function isPreviousServerStopCause(cause: string | null | undefined) {
return cause === "native_workspace_finalization_owner_unverified" || cause === CONTROLLER_HOST_UNVERIFIED
|| cause === OWNER_UNPROVABLE;
}
/** Requests preservation; the existing worker owns stop, copyback and verification. */
export function WorkspaceExportRecovery({ action, className, issueId, companyId, showStatus = true, onRecoveryControlChange }: {
className?: string; action: IssueRecoveryAction | null; issueId?: string; companyId?: string; showStatus?: boolean;
onRecoveryControlChange?: (owns: boolean) => void;
}) {
// A restart killed the copyback owner of retained saved-work recovery. Recovery
// and any accepted Retry wait on its receipt; the confirmation is the one action.
const retainedOwnerToken = action && issueId && companyId && isRetainedExport(action, issueId, companyId)
? retainedOwnerStopToken(action) : null;
if (action && issueId && companyId && retainedOwnerToken) {
return <><ReportRecoveryControl owns onChange={onRecoveryControlChange} />
<PreviousServerStopConfirmation key={`${action.id}:${String(action.evidence.runId)}:${retainedOwnerToken}`}
action={action} ownerToken={retainedOwnerToken} issueId={issueId} companyId={companyId} className={className} /></>;
}
if (action && issueId && companyId && isRetainedExport(action, issueId, companyId)) {
return <RetainedFilesRecovery key={`${companyId}:${issueId}:${action.id}:${action.evidence.runId}`}
action={action} issueId={issueId} companyId={companyId} className={className}
onRecoveryControlChange={onRecoveryControlChange} />;
}
// An ephemeral server cannot judge a pre-upgrade controller row by PID.
if (action?.cause === CONTROLLER_HOST_UNVERIFIED && issueId && companyId
&& isControllerStopRequired(action, issueId, companyId)) {
return <><ReportRecoveryControl owns onChange={onRecoveryControlChange} />
<PreviousServerStopConfirmation key={`${action.id}:${String(action.evidence.runId)}`}
action={action} issueId={issueId} companyId={companyId} className={className}
confirmStop={stopEvidence => issuesApi.confirmControllerStopped(issueId, {
actionId: action.id, runId: action.evidence.runId as string, stopEvidence })}
copy={{ waiting: "This run’s previous server was recorded before Paperclip stored which host it ran on, so Paperclip can’t verify that it stopped. The run’s saved state is kept.",
question: "How did you confirm that the previous server stopped?",
placeholder: "For example: the old server container was replaced and is no longer running.",
resuming: "Controller stop recorded. Checking recovery…" }} /></>;
}
// A resource held by a server recorded before Paperclip stored which server held it.
if (action?.cause === OWNER_UNPROVABLE && issueId && companyId && isOwnerGoneRequired(action, issueId, companyId)) {
const resource = OWNER_GONE_RESOURCE[String(action.evidence.kind)] ?? OWNER_GONE_DEFAULT;
return <><ReportRecoveryControl owns onChange={onRecoveryControlChange} />
{/* Keyed by the record version: a request refreshed to a newer version resets an open form, so
the board never confirms a version it was not shown. */}
<PreviousServerStopConfirmation key={`${action.id}:${String(action.evidence.recordId)}:${String(action.evidence.recordUpdatedAt ?? "")}`}
action={action} issueId={issueId} companyId={companyId} className={className}
confirmStop={stopEvidence => issuesApi.confirmOwnerGone(issueId, {
actionId: action.id, recordId: action.evidence.recordId as string, stopEvidence,
// The version shown here: a request refreshed since must be confirmed again.
...(typeof action.evidence.recordUpdatedAt === "string" ? { recordVersion: action.evidence.recordUpdatedAt } : {}) })}
copy={{ waiting: `${resource.what} was recorded before Paperclip stored which server held it, so Paperclip can’t verify that the server stopped. It stays reserved until then.`,
question: "How did you confirm that the previous server stopped?",
placeholder: "For example: the old server container was replaced and is no longer running.",
resuming: resource.releasing }} /></>;
}
const ownerRecovery = action?.cause === "native_workspace_finalization_owner_unverified";
// The task's interruption names this wait; the confirmation is its one action.
if (ownerRecovery && issueId && companyId && isOwnerStopRequired(action, issueId, companyId)) {
return <><ReportRecoveryControl owns onChange={onRecoveryControlChange} />
<PreviousServerStopConfirmation key={`${action.id}:${String(action.evidence.runId)}`}
action={action} ownerToken={asObject(action.evidence.owner).token as string}
issueId={issueId} companyId={companyId} className={className} /></>;
}
if (!showStatus) return <ReportRecoveryControl owns={false} onChange={onRecoveryControlChange} />;
if (!action || (!ownerRecovery && !isNativeWorkspaceExportRepairCause(action.cause)) || action.ownerType !== "board"
|| !["active", "escalated"].includes(action.status) || typeof action.evidence.runId !== "string")
return <ReportRecoveryControl owns={false} onChange={onRecoveryControlChange} />;
const queued = !!action.evidence.workspaceOwnerStop || action.wakePolicy?.kind === "resume_native_run";
const automatic = queued || (ownerRecovery && asObject(action.evidence.workspaceOwnerRecovery).state === "resuming");
return <><ReportRecoveryControl owns={false} onChange={onRecoveryControlChange} />
<p role="status" className={cn("text-sm text-muted-foreground", className)}>
{automatic ? "Restoring saved work…" : "Saved work is not recovered yet. The agent’s result is saved."}
</p></>;
}
function isRetainedExport(action: IssueRecoveryAction, issueId: string, companyId: string) {
return action.companyId === companyId && action.sourceIssueId === issueId
&& action.kind === "active_run_watchdog" && ["active", "escalated"].includes(action.status)
&& ["board", "system"].includes(action.ownerType)
&& isRetainedWorkspaceExportCause(action.cause)
&& typeof action.evidence.runId === "string" && action.evidence.runId.length > 0;
}
/** Saved-work recovery ended because the sandbox disk was still full (#20758). */
function isSandboxDiskFullRest(action: IssueRecoveryAction) {
const system = asObject(action.evidence.systemRecovery);
const failure = asObject(system.failure);
return system.state === "exhausted" && failure.retryClass === "user_action_required"
&& asObject(failure.cause).label === "sandbox_disk_full";
}
function isControllerStopRequired(action: IssueRecoveryAction, issueId: string, companyId: string) {
return action.companyId === companyId && action.sourceIssueId === issueId
&& action.ownerType === "board" && ["active", "escalated"].includes(action.status)
&& typeof action.evidence.runId === "string" && !action.evidence.controllerStop;
}
function isOwnerGoneRequired(action: IssueRecoveryAction, issueId: string, companyId: string) {
return action.companyId === companyId && action.sourceIssueId === issueId
&& action.ownerType === "board" && ["active", "escalated"].includes(action.status)
&& typeof action.evidence.recordId === "string" && !action.evidence.ownerGone;
}
function isOwnerStopRequired(action: IssueRecoveryAction, issueId: string, companyId: string) {
return action.companyId === companyId && action.sourceIssueId === issueId
&& action.ownerType === "board" && ["active", "escalated"].includes(action.status)
&& typeof action.evidence.runId === "string" && typeof asObject(action.evidence.owner).token === "string"
&& !action.evidence.workspaceOwnerStop && asObject(action.evidence.workspaceOwnerRecovery).state !== "resuming";
}
/** The exact receipt #19721 records when retained recovery waits on a stopped copyback owner. */
function retainedOwnerStopToken(action: IssueRecoveryAction): string | null {
const hold = asObject(action.evidence.workspaceOwnerRecovery);
const token = asObject(hold.owner).token;
return hold.state === "operator_required" && !action.evidence.workspaceOwnerStop
&& typeof action.evidence.runId === "string" && typeof token === "string" && token.length > 0 ? token : null;
}
/** A previous server that can no longer report cannot confirm its own stop.
* The board confirms it once; the saved result then finishes without another agent turn. */
export function PreviousServerStopConfirmation({ action, runId: givenRunId, formId: givenFormId, ownerToken, issueId, companyId, className, confirmStop, copy }: {
action?: IssueRecoveryAction; runId?: string; formId?: string; issueId: string; companyId: string; className?: string;
/** Saved-work copyback owner; used when no custom confirmation is given. */
ownerToken?: string;
confirmStop?: (stopEvidence: string) => Promise<unknown>;
copy?: { waiting: string; question: string; placeholder: string; resuming: string; confirm?: string };
}) {
const runId = givenRunId ?? action!.evidence.runId as string;
const formId = givenFormId ?? action!.id;
const client = useQueryClient();
const [open, setOpen] = useState(false);
const [evidence, setEvidence] = useState("");
const access = useQuery({ queryKey: queryKeys.access.currentBoardAccess,
queryFn: () => accessApi.getCurrentBoardAccess() });
const accessView = useQueryView(access);
const canConfirm = (accessView.kind === "ready" || accessView.kind === "stale") && canBoardConfirmOwnerStop(companyId, access.data);
const confirm = useMutation({ retry: false, // query-error-ok: an explicit board confirmation; the error is shown below.
mutationFn: () => confirmStop ? confirmStop(evidence.trim())
: issuesApi.resumeWorkspaceFinalization(issueId, { actionId: action!.id, runId, ownerToken: ownerToken!,
controllerAndCopybackStopped: true, stopEvidence: evidence.trim() }),
onSuccess: () => {
setOpen(false);
void client.invalidateQueries({ queryKey: queryKeys.issues.detail(issueId) });
void client.invalidateQueries({ queryKey: queryKeys.issues.runs(issueId) });
void client.invalidateQueries({ queryKey: queryKeys.runDetail(runId) });
} });
const ready = evidence.trim().length >= 20;
return <div className={cn("flex flex-col gap-2 text-sm", className)} data-testid="previous-server-stop">
<p role="status">{confirm.isSuccess ? copy?.resuming ?? "Restoring saved work…"
: copy?.waiting ?? "The previous server stopped while it was saving this task’s files, and Paperclip can’t verify that it stopped. The agent’s result is saved."}</p>
{confirm.isSuccess ? null : !canConfirm
? <p className="text-xs text-muted-foreground">{accessView.kind === "loading" || accessView.kind === "reconnecting"
? "Checking access…" : "Paperclip is waiting to confirm that the previous server stopped."}</p>
: !open ? <Button variant="outline" size="sm" className="self-start" onClick={() => setOpen(true)}>
{copy?.confirm ?? "Confirm the previous server stopped"}</Button>
: <form className="flex flex-col gap-2" onSubmit={event => { event.preventDefault(); if (ready && !confirm.isPending) confirm.mutate(); }}>
<label htmlFor={`previous-server-stop-${formId}`} className="text-xs text-muted-foreground">
{copy?.question ?? "How did you confirm that the previous server and its file copy stopped?"}</label>
<Textarea id={`previous-server-stop-${formId}`} value={evidence} onChange={event => setEvidence(event.target.value)}
placeholder={copy?.placeholder ?? "For example: the old server process exited and was not restarted."} maxLength={12000} rows={3} />
<div className="flex items-center justify-between gap-2">
<Button type="button" variant="ghost" size="sm" disabled={confirm.isPending} onClick={() => setOpen(false)}>Cancel</Button>
<Button type="submit" size="sm" disabled={!ready || confirm.isPending}>
{confirm.isPending ? "Confirming…" : "Confirm and resume"}</Button>
</div>
</form>}
{confirm.error ? <p role="alert" className="text-sm text-destructive">
{confirm.error instanceof Error ? confirm.error.message : "The confirmation was not accepted. Refresh the task and try again."}</p> : null}
</div>;
}
type RecoveryInput = { actionId: string; runId: string };
/** Plain wording for a source inspection that did not verify. The code stays visible for support. */
function unverifiedSourceText(inspected: WorkspaceSourceInspection | undefined): string {
if (!inspected || inspected.status === "verified") return "";
if (inspected.status === "not_inspected")
return "Paperclip can’t check the saved files’ source here because the run recorded too many sources.";
const code = inspected.sourceVerification?.reason ?? inspected.failureCode;
return code === "allocation_ownership_ambiguous"
? "Paperclip can’t confirm that this run alone used the sandbox that holds the saved files."
: `Paperclip can’t verify the saved files’ recorded source (${code}).`;
}
function RetainedFilesRecovery({ action, issueId, companyId, className, onRecoveryControlChange }: {
action: IssueRecoveryAction; issueId: string; companyId: string; className?: string;
onRecoveryControlChange?: (owns: boolean) => void;
}) {
const runId = action.evidence.runId as string;
const owner = useDraftOwner();
const readOwner = useDraftOwnerReader();
const client = useQueryClient();
const busy = useRef(false);
const [acknowledged, setAcknowledged] = useState(false);
const [checkedUnconfirmed, setCheckedUnconfirmed] = useState(false);
const [notice, setNotice] = useState<string | null>(null);
const access = useQuery({ queryKey: queryKeys.access.currentBoardAccess,
queryFn: () => accessApi.getCurrentBoardAccess() });
const accessView = useQueryView(access);
const accessReady = accessView.kind === "ready" || accessView.kind === "stale";
const canMutate = Boolean(accessReady && owner && (access.data?.source === "local_implicit"
|| owner !== "signed-out" && access.data?.userId === owner)
&& canBoardResolveRecoveryAction(companyId, access.data));
const source = useQuery({ queryKey: queryKeys.issues.recoveryActions(issueId, action.id, runId),
queryFn: () => issuesApi.recoveryActions(issueId, { runId, actionId: action.id }), enabled: canMutate });
const sourceView = useQueryView(source);
const sourceReady = sourceView.kind === "ready" || sourceView.kind === "stale";
const storageKey = owner ? pendingSendKeyFor(`workspace-file-recovery:${companyId}:${issueId}:${action.id}:${runId}`, owner) : null;
const matchesInput = (value: unknown): value is RecoveryInput => {
const input = asObject(value);
return input.actionId === action.id && input.runId === runId;
};
const pending = storageKey ? readPendingSend(storageKey, matchesInput) : null;
// The recovery record is bound by its own identity. Source inspection is a
// separate read-only probe: it gates only the physical file-recovery request.
// It legitimately reports "blocked" after files were recovered, or when a
// sandbox predates ownership receipts; the recorded state stays authoritative.
function boundAction(data: typeof source.data) {
const current = data?.actions.find(item => item.id === action.id);
const inspected = data?.sourceInspection;
const state = asObject(current?.evidence.workspaceRecovery);
return current && isRetainedExport(current, issueId, companyId) && current.evidence.runId === runId
&& state.runId === runId && state.schema === "paperclip.workspace-recovery.v1"
&& inspected?.actionId === action.id && inspected.runId === runId ? current : null;
}
const sourceVerified = (data: typeof source.data) => Boolean(boundAction(data)) && data?.sourceInspection?.status === "verified";
const current = sourceReady ? boundAction(source.data) : null;
const verified = sourceReady && sourceVerified(source.data);
const recovery = asObject(current?.evidence.workspaceRecovery);
const system = asObject(current?.evidence.systemRecovery);
const serverAcknowledged = Boolean(pending && Array.isArray(recovery.requestIds) && recovery.requestIds.includes(pending.idempotencyKey));
const restoring = ["queued", "recovering"].includes(String(recovery.status));
const saved = recovery.status === "recovered" && system.state === "complete"
&& Array.isArray(system.preservationOperationIds) && system.preservationOperationIds.length > 0
&& system.preservationOperationIds.every(id => typeof id === "string" && id.length > 0);
const retryable = recovery.status === "blocked" && recovery.retryable === true;
const invalidate = () => {
void client.invalidateQueries({ queryKey: queryKeys.issues.detail(issueId) });
void client.invalidateQueries({ queryKey: queryKeys.issues.runs(issueId) });
void client.invalidateQueries({ queryKey: queryKeys.runDetail(runId) });
};
const request = useMutation({ retry: false, mutationFn: async (restartFailedAttempt: boolean) => { // query-error-ok: recovery requests require an explicit action and receipt reconciliation.
if (busy.current) return;
busy.current = true;
setNotice(null);
setCheckedUnconfirmed(false);
setAcknowledged(false);
try {
const observed = await source.refetch();
const fresh = boundAction(observed.data);
const state = asObject(fresh?.evidence.workspaceRecovery);
if (observed.error || !fresh || state.status !== "blocked" || state.retryable !== true)
throw new Error("File recovery changed. Inspect its current status before requesting recovery.");
if (!sourceVerified(observed.data))
throw new Error(unverifiedSourceText(observed.data?.sourceInspection) || "The saved files’ source could not be verified.");
if (!canMutate || !owner || readOwner() !== owner || !storageKey)
throw new Error("Your account changed. Refresh before requesting file recovery.");
const previous = readPendingSend(storageKey, matchesInput);
if (restartFailedAttempt && (!previous || !Array.isArray(state.requestIds)
|| !state.requestIds.includes(previous.idempotencyKey)
|| asObject(fresh.evidence.systemRecovery).state !== "exhausted"))
throw new Error("The previous failed recovery is not confirmed. Check its status before trying again.");
// Unknown acceptance reuses its identity. A new bounded request requires
// an explicit operator click and a freshly confirmed exhausted attempt.
const record = (!restartFailedAttempt && previous) || {
idempotencyKey: createUuid(), payload: { actionId: action.id, runId }, createdAt: Date.now(),
};
if (!writePendingSend(storageKey, record))
throw new Error("The recovery request could not be saved in this browser. No request was sent.");
try {
const result = await issuesApi.recoverWorkspace(issueId, { ...record.payload, idempotencyKey: record.idempotencyKey });
if (result.actionId !== action.id || !["queued", "recovering", "blocked", "recovered"].includes(result.status))
throw new Error("The recovery acknowledgement could not be confirmed.");
setAcknowledged(true);
setCheckedUnconfirmed(false);
invalidate();
await source.refetch();
} catch (error) {
setNotice(classifySendFailure(error) === "rejected"
? `Recovery request was not confirmed: ${error instanceof Error ? error.message : "Inspect its status."}`
: "The recovery request could not be confirmed. Check its status before trying again; the saved request will be reused.");
}
} catch (error) {
setNotice(error instanceof Error ? error.message : "Unable to inspect file recovery.");
} finally { busy.current = false; }
} });
const check = useMutation({ retry: false, mutationFn: async () => { // query-error-ok: operator checks run once; the source read uses the shared query retry policy.
const observed = await source.refetch();
if (observed.error) throw observed.error;
const fresh = boundAction(observed.data);
if (!fresh) {
// The task's current recovery replaced this record. Reload the task so it
// shows that recovery, instead of asking for a refresh that changes nothing.
invalidate();
throw new Error("This file recovery was replaced. Showing the task’s current recovery.");
}
const state = asObject(fresh.evidence.workspaceRecovery);
const ids = state.requestIds;
const unconfirmed = Boolean(pending && Array.isArray(ids) && !ids.includes(pending.idempotencyKey));
setCheckedUnconfirmed(unconfirmed);
setNotice(unconfirmed ? "No recovery request is confirmed yet. Trying again will reuse the saved request." : null);
invalidate();
} });
const checkError = isTransientError(check.error) ? null : check.error;
const failedReceipt = serverAcknowledged && recovery.status === "blocked";
const confirmedFailedAttempt = failedReceipt && retryable && system.state === "exhausted";
const waiting = !failedReceipt && !saved && (restoring || acknowledged || serverAcknowledged);
const unconfirmed = Boolean(pending && !acknowledged && !serverAcknowledged);
const sourceLoading = sourceView.kind === "loading" || sourceView.kind === "reconnecting";
// A save that rested on a full sandbox disk is resumed by the task's Retry,
// which saves the files again and then applies the saved result. Its note
// names Retry, so this panel offers no separate control for it.
const diskFullRest = isSandboxDiskFullRest(current ?? action);
// A verified source makes this panel the governed recovery path. Otherwise the
// task's own Retry (renewed, re-verified recovery) remains the next action.
const ownsRecoveryControl = canMutate && !diskFullRest && (sourceLoading || Boolean(current && verified && !saved));
// `waiting` already excludes a failed accepted request, so the reason stays
// visible after a failed attempt and hides only while recovery is running.
const unverified = Boolean(current && !verified && !saved && !waiting);
return <div className={cn("flex flex-col gap-2 text-sm", className)} data-testid="retained-file-recovery">
<ReportRecoveryControl owns={ownsRecoveryControl} onChange={onRecoveryControlChange} />
<p role="status">{saved ? "Saved files recovered. Task recovery is still being checked."
: diskFullRest && !waiting ? "The sandbox disk was full when Paperclip saved the files. Free space, then use Retry to save them and finish the task."
: failedReceipt ? "File recovery has not finished. The saved source remains protected."
: restoring ? "Recovering saved files. Work remains paused until recovery is verified."
: waiting ? "File recovery is queued. Waiting for verified saved files."
: "The saved files need recovery before work can continue."}</p>
{canMutate ? <>
{current && verified && retryable && !waiting && !diskFullRest && (!pending || unconfirmed && checkedUnconfirmed || confirmedFailedAttempt) ? <Button variant="outline" size="sm"
className="self-start" disabled={request.isPending || check.isPending}
onClick={() => request.mutate(confirmedFailedAttempt)}>{request.isPending ? "Requesting file recovery…"
: confirmedFailedAttempt ? "Try file recovery again" : "Recover saved files"}</Button> : null}
{waiting || unconfirmed || failedReceipt || source.error || sourceReady && !current ? <Button variant="ghost" size="sm" className="self-start"
disabled={request.isPending || check.isPending} onClick={() => check.mutate()}>
{check.isPending ? "Checking file recovery…" : "Check file recovery status"}
</Button> : null}
{sourceView.kind === "loading" || sourceView.kind === "reconnecting" ? <p className="text-xs text-muted-foreground">Inspecting the recorded file source…</p> : null}
{sourceView.kind === "error" || checkError || notice ? <p role="alert" className="text-sm text-destructive">
{notice ?? (checkError instanceof Error ? checkError.message : "File recovery could not be inspected. Check its status again.")}
</p> : null}
{current && recovery.status === "blocked" && (serverAcknowledged || recovery.retryable !== true)
? <p role="alert">File recovery has not finished. The saved files remain protected and are not applied.</p> : null}
{unverified ? <p role="status" className="text-muted-foreground">
{unverifiedSourceText(source.data?.sourceInspection)} Paperclip does not apply files it can’t verify.
</p> : null}
{sourceReady && !current ? <p role="alert">This file recovery was replaced. Check its status to show the task’s current recovery.</p> : null}
</> : accessView.kind === "error" ? <p role="alert" className="text-sm text-destructive">Task access could not be checked. Refresh before requesting file recovery.</p>
: accessView.kind === "loading" || accessView.kind === "reconnecting" || !owner ? <p role="status" className="text-xs text-muted-foreground">Checking file recovery access…</p>
: <p className="text-xs text-muted-foreground">A board member with task access can request file recovery.</p>}
</div>;
}