fix(ui): keep a Retry for retained-file holds whose source cannot be verified (#19840)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a native run finishes but its files cannot be saved, the task
stays blocked behind a retained-file recovery hold.
> - The server records that hold as blocked and retryable, and the task
notice tells the user to "Retry to try again".
> - The task page hides its own Retry for these holds and relies on the
retained-file recovery panel instead.
> - That panel only binds the recovery record when a read-only source
inspection returns `verified`.
> - Inspection can return `blocked` for normal reasons: a sandbox that
predates ownership receipts (state saved by an older release), or a
workspace that moved on after its files were recovered.
> - Then neither surface offers an action, and "Check file recovery
status" always answers "File recovery changed. Refresh the task", also
after a reload.
> - This pull request binds the record by its own identity, uses
inspection only to gate the physical file-recovery request, and keeps
the task Retry visible whenever the panel has no control of its own.

## Linked Issues or Issue Description

Refs #18848 (the change that added the retained-file recovery panel and
hid the generic Retry for these holds).

**What happened?**

A blocked task with an active
`native_workspace_sync_out_retry_exhausted` hold showed "The saved files
need recovery before work can continue", "The recorded source is
unavailable or has changed", and "Saving the agent's files failed after
automatic retries. Retry to try again". It showed no Retry and no
recover button. "Check file recovery status" always answered "File
recovery changed. Refresh the task to inspect the current recovery", and
a reload did not change it. The hold was saved by an earlier release:
`systemRecovery` exhausted with `allocation_ownership_ambiguous` /
`service_required`, and `workspaceRecovery` blocked with `retryable:
true`. The source inspection endpoint returned `status: "blocked"` for
it. A second task whose files were already recovered showed the same
"need recovery" and "has changed" text, because its inspection reported
a reference mismatch after recovery.

**Expected behavior**

A retryable hold always shows a working Retry or recover control. A
notice that says "Retry" has a Retry button. Status checks never loop on
"changed, refresh". Recovered files show as recovered. When the source
cannot be verified, the page names the reason and who must inspect it.

**Steps to reproduce**

1. Use an active retained-export hold whose `workspaceRecovery` is
blocked and retryable, and whose `systemRecovery` is exhausted.
2. Make the source inspection (`GET
/api/issues/:id/recovery-actions?inspectSourceRunId=…&inspectSourceActionId=…`)
return `status: "blocked"`.
3. Open the task as a board operator.
4. Before this change, the page shows no Retry and no recover button.
"Check file recovery status" reports "File recovery changed. Refresh the
task".

**Paperclip version or commit**

Observed on build `8812c19`. The regressions fail on master `0f5ce5050e`
before the production edits.

## What Changed

- `WorkspaceExportRecovery`: bind the recovery record by action, run and
schema identity, not by inspection status. Inspection status now gates
only the physical "Recover saved files" request (and the request
re-checks it before POST).
- `WorkspaceExportRecovery`: report through `onRecoveryControlChange`
whether the panel offers its own control. It reports `true` only while
the source is loading or verified and the files are not yet recovered,
or while it shows the previous-server stop confirmation.
- `IssueDetail`: hide the task Retry for retained-export holds only
while the panel reports that it owns the control. Otherwise the existing
task Retry (`POST /issues/:id/retry`) shows. The server renews a bounded
recovery generation and re-verifies the source.
- Show recovered files as recovered even when inspection no longer
matches after recovery.
- Name an unverifiable source in plain words (with the code for other
reasons). State that Paperclip does not apply files it can't verify.
- Recovery text never asks a person to inspect a sandbox or environment,
and never names an owner or administrator as the one who must act. This
covers the retained-file panel, the run source inspector, the copyback
wait, and the native cleanup quarantine message. The execution blocker
and the recovery action read model describe a spent recovery budget from
its current state, so text saved by older releases ("If Retry keeps
failing, inspect the run and its environment") is not shown.
- Task Retry uses the existing `createUuid()` helper, so it works on
plain-HTTP previews without `crypto.randomUUID`. An accepted Retry also
refreshes the inspected file-recovery state. After a failed accepted
request, the unverified-source reason stays visible; it hides only while
recovery runs.
- When the recovery record was replaced, "Check file recovery status"
reloads the task detail instead of telling the user to refresh.
- Tests: component regressions for an old-release hold, a recovered hold
with a mismatched source, verified ownership, and a replaced record. A
page regression proves Retry is shown and calls `retryTask` for
unverified and stale sources. A server safety test proves that Retry for
the old-release hold shape is accepted, opens one new bounded
generation, queues file recovery again, keeps the hold active, and is
idempotent.

## Verification

- RED (master `0f5ce5050e` production files, new tests): 5 component
tests fail. The failure output shows the exact staging text: "Check file
recovery status", "File recovery changed. Refresh the task to inspect
the current recovery.", "The recorded source is unavailable or has
changed." 2 page tests (unverified and stale) fail with `expected [
'Check file recovery status', … ] to include 'Retry'`.
- GREEN: `cd ui && npx vitest run
src/components/WorkspaceExportRecovery.control.test.tsx
src/components/WorkspaceExportRecovery.test.tsx
src/components/RunWorkspaceRecoverySurface.test.tsx
src/components/ExecutionBlockerNotice.test.tsx
src/pages/IssueDetail.test.tsx` passes 320 tests.
- Server: `cd server && npx vitest run src/services/task-retry.test.ts`
passes 81 tests, including the new old-release safety case. This is
safety evidence, not a red: the server path was already correct, and the
defect was in the UI.
- Full UI suite: 11564 pass. 4 tests in `IssuesList.test.tsx` and
`CompanyEnvironments.test.tsx` failed under full-suite load and pass in
isolation (106/106). They are unrelated to this change.
- `pnpm check:token-gates` is clean. `node scripts/run-tsc.mjs -p
ui/tsconfig.json --noEmit` reports no errors in the touched files. The
remaining errors are local artifacts that were not built (plugin SDK UI
entry, generated Storybook catalog).
- Staging was read only (GET requests). The inspect request did not loop
in a fresh tab: two calls in 30 seconds. Most of the tab's request
volume is periodic company issue-list refetches, which this change does
not touch.

## Risks

- Low risk. This is a UI-only behavior change. Server authorization,
source verification, stop receipts, and the bounded recovery policy are
unchanged and remain authoritative.
- The task Retry is now visible for retained-export holds whose source
does not verify. Retry renews one recovery generation. The worker
re-runs the same source-identity checks, so Retry cannot bypass an
ownership refusal. If the refusal is permanent (for example an ambiguous
sandbox allocation), the hold returns to exhausted and the page names
the reason. An automatic fallback for such sources is follow-up work.
- Wording: a parametrized UI test renders every recovery state and
asserts that no administrator, owner, or "inspect the environment" copy
appears. A server test proves that a stored old-release message is
replaced by state-derived text. Both failed before the change.
- Previous-release story: the old-release hold shape (board-owned,
`systemRecovery` revision 5 exhausted with `service_required`,
`workspaceRecovery` blocked and retryable) is covered by component,
page, and server tests. No persisted state shape changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (repository search, file edits, shell, test
execution, and a read-only browser).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-07 10:24:03 -06:00
1 parent 630411af53
commit 824b25654c
18 files changed
+356 -58

No files matched your search

+10 -1
View File
@@ -17,6 +17,7 @@ import type {
} from "@paperclipai/shared";
import { isNativeWorkspaceFinalizationOperationActive } from "./workspace-operations.js";
import { exhaustedSystemRecoveryAction, readSystemRecovery } from "./system-recovery-state.js";
const ACTIVE_RECOVERY_ACTION_STATUSES = ["active", "escalated"] as const satisfies readonly IssueRecoveryActionStatus[];
const MAX_UPSERT_RETRIES = 3;
@@ -87,6 +88,14 @@ export type ResolveIssueRecoveryActionInput = {
resolutionNote?: string | null;
};
/** A spent saved-work recovery is described from its current state. Text saved
* by an older release could ask a person to inspect the run's environment. */
function readModelNextAction(row: IssueRecoveryActionRow): string {
const system = readSystemRecovery(row.evidence);
return system?.state === "exhausted" && row.evidence?.workspaceRecovery
? exhaustedSystemRecoveryAction(system) : row.nextAction;
}
function toReadModel(row: IssueRecoveryActionRow): IssueRecoveryAction {
return {
id: row.id,
@@ -103,7 +112,7 @@ function toReadModel(row: IssueRecoveryActionRow): IssueRecoveryAction {
cause: row.cause,
fingerprint: row.fingerprint,
evidence: row.evidence,
nextAction: row.nextAction,
nextAction: readModelNextAction(row),
wakePolicy: row.wakePolicy,
monitorPolicy: row.monitorPolicy,
attemptCount: row.attemptCount,
@@ -7984,7 +7984,7 @@ const NATIVE_STARTUP_MODEL_REJECTED_ACTION =
"The provider could not start the selected model before any provider work began. Choose a supported model for this agent, then retry the task. Automatic retries are stopped.";
const NATIVE_CLEANUP_OPERATOR_RECOVERY_MESSAGE =
"Send a new message to continue after Paperclip verifies that the previous provider and its tools have stopped. If cleanup cannot be verified, inspect the run and its environment. Clearing a task session does not resolve this quarantine. Automatic retries are stopped.";
"Send a new message to continue after Paperclip verifies that the previous provider and its tools have stopped. Clearing a task session does not resolve this quarantine. Automatic retries are stopped.";
const PROVIDER_DURABLE_EVENT_TYPES = NATIVE_PROVIDER_DURABLE_EVENT_TYPES;
@@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto";
import { describe, expect, it } from "vitest";
import { createSystemRecovery, failSystemRecovery, readSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "./system-recovery-state.js";
import { exhaustedSystemRecoveryAction } from "./system-recovery.js";
import { workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js";
describe("durable system recovery budget", () => {
it("makes six attempts with exponential delays that survive serialization and exhaustion", () => {
@@ -43,6 +44,25 @@ describe("durable system recovery budget", () => {
{ now: 0, cause: { label: "/home/daytona/secret", code: null, status: null } });
expect(unsafe.failure).not.toHaveProperty("cause");
});
it("never asks a person to inspect a sandbox, even for text saved by an older release", () => {
const manual = /administrator|\bowner\b|board member|inspect the run|environment/i;
const exhausted = failSystemRecovery(createSystemRecovery(randomUUID()),
new SystemRecoveryFailure("allocation_ownership_ambiguous", "service_required"));
const copyback = failSystemRecovery(createSystemRecovery(randomUUID()),
new SystemRecoveryFailure("copyback_stop_unconfirmed", "recheck_evidence"));
for (const state of [exhausted, { ...copyback, state: "exhausted" as const }])
expect(exhaustedSystemRecoveryAction(state)).not.toMatch(manual);
const runId = exhausted.sourceRunId;
const display = workspaceRecoveryDisplay({ runId, systemRecovery: exhausted, workspaceRecovery: {
schema: "paperclip.workspace-recovery.v1", runId, status: "blocked", stage: "verifying_stop", attempt: 1,
attemptLimit: 6, requestIds: [], retryable: true,
message: "Automatic recovery of saved work stopped after 1 attempts (last failure: allocation_ownership_ambiguous). "
+ "Retry recovery to export and merge from the retained sandbox again; no provider turn is submitted. "
+ "If Retry keeps failing, inspect the run and its environment." } }, runId);
expect(display?.message).toContain("Retry to try again");
expect(display?.message).not.toMatch(manual);
});
it("names the last failure and its cause in the board handoff", () => {
const state = reserveSystemRecoveryAttempt(createSystemRecovery(randomUUID()), 0)!;
const withCause = failSystemRecovery(state, new SystemRecoveryFailure("provider_unavailable", "retry_transient"),
@@ -126,3 +126,16 @@ export function failSystemRecovery(
evidenceFingerprint: options.evidenceFingerprint ?? createHash("sha256").update(`${state.sourceRunId}:${failure.code}`).digest("hex") },
};
}
/** User-facing text for a spent recovery budget. It names what happens next and
* never asks a person to inspect a sandbox or environment. The code stays for support. */
export function exhaustedSystemRecoveryAction(state: SystemRecoveryState): string {
// Another Retry cannot release a copyback owner that stopped without reporting.
if (state.failure?.code === "copyback_stop_unconfirmed")
return "Recovery of saved work is waiting for the previous server's file copy to stop. "
+ "Recovery resumes once Paperclip can confirm that it stopped. No provider turn is submitted.";
const cause = state.failure?.cause ? `, cause: ${state.failure.cause.label}` : "";
const reason = state.failure?.code ? ` (last failure: ${state.failure.code}${cause})` : "";
return `Automatic recovery of saved work stopped after ${state.attempts} attempts${reason}. `
+ "Retry to try again; the agent will not repeat its work.";
}
+2 -13
View File
@@ -3,7 +3,7 @@ import { and, eq, sql } from "drizzle-orm";
import { agents, issues, heartbeatRuns, issueComments, issueRecoveryActions, type Db } from "@paperclipai/db";
import { writeExecutionControl } from "./execution-control-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { createSystemRecovery, readSystemRecovery, type SystemRecoveryState } from "./system-recovery-state.js";
import { createSystemRecovery, exhaustedSystemRecoveryAction, readSystemRecovery, type SystemRecoveryState } from "./system-recovery-state.js";
import { workspaceRecoverySourceEligible, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
export async function adoptSystemRecovery(db: Db, source: WorkspaceRecoverySourceIdentity) {
@@ -47,18 +47,7 @@ export async function adoptSystemRecovery(db: Db, source: WorkspaceRecoverySourc
});
}
export function exhaustedSystemRecoveryAction(state: SystemRecoveryState): string {
// Another Retry cannot release a copyback owner that stopped without reporting.
if (state.failure?.code === "copyback_stop_unconfirmed")
return "Automatic recovery of saved work is waiting for the previous server's file copy to stop, and Paperclip cannot verify "
+ "that it stopped. A board member who manages this company's runtime can confirm that it stopped; recovery then "
+ "resumes. No provider turn is submitted.";
const cause = state.failure?.cause ? `, cause: ${state.failure.cause.label}` : "";
const reason = state.failure?.code ? ` (last failure: ${state.failure.code}${cause})` : "";
return `Automatic recovery of saved work stopped after ${state.attempts} attempts${reason}. `
+ "Retry recovery to export and merge from the retained sandbox again; no provider turn is submitted. "
+ "If Retry keeps failing, inspect the run and its environment.";
}
export { exhaustedSystemRecoveryAction };
/** CAS protects a newer manual generation from an old worker's late write.
* This is bookkeeping authority only; physical writes still require ownership. */
+34
View File
@@ -678,6 +678,40 @@ describe("durable task Retry admission", () => {
expect(wakes.find(row => row.id === first.requestId)?.status).toBe("failed");
expect(wakes.filter(row => row.status === "deferred_issue_execution")).toHaveLength(1);
});
it("accepts Retry for an old-release retained-export hold whose source ownership could not be verified", async () => {
// State saved by an earlier release: a board-owned export hold whose only
// attempt ended with a permanent ownership refusal. The task UI now offers
// Retry for it; Retry must open one new bounded generation that re-verifies
// the source, keep the hold in place, and never resolve or bypass it.
const input = await seed();
const previous = { ...failSystemRecovery({ ...createSystemRecovery(input.observedRunId), revision: 4 },
new SystemRecoveryFailure("allocation_ownership_ambiguous", "service_required")), revision: 5 };
expect(previous.state).toBe("exhausted");
const [action] = await db.insert(issueRecoveryActions).values({
companyId: input.companyId, sourceIssueId: input.issueId, kind: "active_run_watchdog", status: "active",
ownerType: "board", cause: "native_workspace_sync_out_retry_exhausted", fingerprint: randomUUID(),
nextAction: "Automatic recovery of saved work stopped after 1 attempts (last failure: allocation_ownership_ambiguous).",
evidence: { runId: input.observedRunId, systemRecovery: previous,
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: input.observedRunId, stage: "verifying_stop",
status: "blocked", attempt: 1, attemptLimit: 6, requestIds: [], retryable: true, message: "Retry recovery." },
coordinatorAttempt: 1, originalFailureCode: "native_workspace_sync_out_failed",
workspaceFinalizeAttempt: 3, previousWorkspaceRecovery: null },
}).returning();
const receipt = await requestTaskRetry(input);
expect(receipt).toMatchObject({ accepted: true, stale: false, observedRunId: input.observedRunId });
const [updated] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id));
expect(updated.status).toBe("active");
expect(updated.resolvedAt).toBeNull();
expect(readSystemRecovery(updated.evidence)).toMatchObject({ generationId: receipt.requestId, sourceRunId: input.observedRunId,
attempts: 0, failure: null, revision: 6, freshAttemptRunId: null, preservationOperationIds: [] });
// File recovery is queued again for the worker, which re-verifies the source.
expect(updated.evidence.workspaceRecovery).toMatchObject({ runId: input.observedRunId, status: "queued",
stage: "verifying_stop", attemptLimit: 6, retryable: true });
// A replayed click is idempotent and does not open another generation.
expect((await requestTaskRetry(input)).requestId).toBe(receipt.requestId);
const [replayed] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id));
expect(readSystemRecovery(replayed.evidence)?.revision).toBe(6);
});
it("binds a key to its company, task and actor across all statuses", async () => {
const input = await seed();
const receipt = await requestTaskRetry(input);
@@ -5,6 +5,7 @@ import { z } from "zod";
import { readNativeWorkspaceSyncReference, type NativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
import { readMigratedLegacyWorkspaceSource } from "./legacy-workspace-recovery-migration.js";
import { readSnapshot } from "./workspace-recovery-snapshot.js";
import { exhaustedSystemRecoveryAction, readSystemRecovery } from "./system-recovery-state.js";
export const workspaceRecoverySchema = z.object({
schema: z.literal("paperclip.workspace-recovery.v1"),
@@ -58,10 +59,17 @@ export function workspaceRecoveryDisplay(evidence: Record<string, unknown>, runI
canRetry: state.status === "blocked" && state.retryable === true,
message: state.status === "queued" || state.status === "recovering" ? "Recovering saved work…"
: state.status === "recovered" ? "Saved work recovered. Send a message to continue."
: state.message ?? "Paperclip couldn’t safely recover this task. This task needs service attention.",
: blockedWorkspaceRecoveryMessage(evidence) ?? state.message ?? "Recovery of this task's saved work has not finished.",
};
}
/** Text saved by an older release can name a manual step. A spent budget is
* described from its current state instead of the stored text. */
export function blockedWorkspaceRecoveryMessage(evidence: Record<string, unknown> | null | undefined): string | null {
const system = readSystemRecovery(evidence ?? undefined);
return system?.state === "exhausted" ? exhaustedSystemRecoveryAction(system) : null;
}
async function verifiedNativeCopybackReceipt(db: Db, input: {
companyId: string; issueId: string; runId: string; reference: NativeWorkspaceSyncReference;
}) {
@@ -508,7 +508,7 @@ describe("automatic retained workspace recovery", () => {
const stable = (row: typeof after) => ({ ...row, updatedAt: null, ownerType: null, nextAction: null,
evidence: { ...row.evidence, workspaceRecovery: { ...(row.evidence.workspaceRecovery as object), message: null } } });
if (spent) {
expect(after).toMatchObject({ ownerType: "board", nextAction: expect.stringMatching(/Retry recovery/) });
expect(after).toMatchObject({ ownerType: "board", nextAction: expect.stringMatching(/Retry to try again/) });
expect(stable(after)).toEqual(stable(before));
} else expect(after).toEqual(before);
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].leasePolicy).toBe("reuse_by_environment");
@@ -474,7 +474,7 @@ describe("automatic retained workspace recovery", () => {
expect(exhausted.nextAction).toMatch(/stopped after 6 attempts/);
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
expect(blocker?.workspaceRecovery).toMatchObject({ status: "blocked", canRetry: true });
expect(blocker?.nextAction).toMatch(/Retry recovery/);
expect(blocker?.nextAction).toMatch(/Retry to try again/);
expect(blocker?.nextAction).not.toMatch(/send a (new )?message/i);
// Nothing physical or provider-facing ran after the budget was spent.
expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
@@ -501,7 +501,7 @@ describe("automatic retained workspace recovery", () => {
const handed = await action(f.actionId);
expect(handed).toMatchObject({ status: "active", ownerType: "board" });
expect(readSystemRecovery(handed.evidence)).toEqual(spent);
expect((await getExecutionBlocker(db, f.companyId, f.issueId))?.nextAction).toMatch(/Retry recovery/);
expect((await getExecutionBlocker(db, f.companyId, f.issueId))?.nextAction).toMatch(/Retry to try again/);
expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
// Repeat sweeps are idempotent and still grant no attempt.
await recoverRetainedWorkspace(f.input);
@@ -51,7 +51,7 @@ export function RunWorkspaceSourceInspection({ issueId, runId, companyId }: { is
<p className="font-mono text-muted-foreground">Action {current.actionId}</p>
<p className="text-muted-foreground">This read-only observation checks recorded identity. It does not confirm provider termination, local files, or successful preservation.</p>
{current.status === "verified" ? <p>Recorded-source identity verified.</p>
: current.status === "not_inspected" ? <p>Source not inspected: this record exceeds the inspection limit. Service attention is required.</p>
: current.status === "not_inspected" ? <p>Source not inspected: this record exceeds the inspection limit.</p>
: <p>Blocked: <code className="font-mono">{current.sourceVerification?.reason ?? current.failureCode}</code>
{current.sourceVerification?.observedLeasePolicy ? <> · lease policy <code className="font-mono">{current.sourceVerification.observedLeasePolicy}</code></> : null}</p>}
<p className="font-mono text-muted-foreground">Observed {formatDateTime(current.observedAt)}</p>
@@ -6,6 +6,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { IssueRecoveryAction } from "@paperclipai/shared";
import { ApiError } from "../api/client";
import { queryKeys } from "../lib/queryKeys";
import { writePendingSend } from "../lib/pending-send";
import { WorkspaceExportRecovery } from "./WorkspaceExportRecovery";
const api = vi.hoisted(() => ({ inspect: vi.fn(), recover: vi.fn(), resume: vi.fn() }));
@@ -153,7 +154,106 @@ describe("governed retained-file recovery", () => {
} });
await mount();
expect(button("Recover saved files")).toBeUndefined();
expect(container.textContent).toContain("source is unavailable or has changed");
expect(container.textContent).toContain("can’t verify the saved files’ recorded source (source_binding_invalid)");
});
// Recovery saved by an older release: a board-owned hold whose
// sandbox predates ownership receipts. The server still records a retryable
// blocked recovery, but the read-only source probe cannot verify ownership.
const legacyHold = { ...action, id: "legacy-hold", ownerType: "board", evidence: { runId: "run-1",
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: "run-1", stage: "verifying_stop",
status: "blocked", attempt: 1, attemptLimit: 6, requestIds: [], retryable: true,
message: "Automatic recovery of saved work stopped after 1 attempts (last failure: allocation_ownership_ambiguous)." },
systemRecovery: { schema: "paperclip.system-recovery.v1", phase: "verify_execution", state: "exhausted", revision: 5,
sourceRunId: "run-1", attempts: 1, retriesUsed: 0, preservationOperationIds: [], freshAttemptRunId: null,
failure: { code: "allocation_ownership_ambiguous", retryClass: "service_required" } },
originalFailureCode: "native_workspace_sync_out_failed", workspaceFinalizeAttempt: 3, previousWorkspaceRecovery: null },
} as unknown as IssueRecoveryAction;
const blockedInspection = (value: IssueRecoveryAction, failureCode = "allocation_ownership_ambiguous") => ({
active: value, actions: [value], sourceInspection: { schema: "paperclip.workspace-source-inspection.v1",
actionId: value.id, runId: "run-1", status: "blocked", failureCode,
scope: "recorded_source_identity", observedAt: "2026-11-07T14:44:46Z" } });
it("hands the next action to the task Retry for an old-release hold whose source cannot be verified", async () => {
api.inspect.mockResolvedValue(blockedInspection(legacyHold));
const owns = vi.fn();
await act(async () => root.render(<QueryClientProvider client={client}>
<WorkspaceExportRecovery action={legacyHold} issueId="issue-1" companyId="company-1" onRecoveryControlChange={owns} />
</QueryClientProvider>));
await settle();
expect(owns).toHaveBeenLastCalledWith(false);
expect(button("Recover saved files")).toBeUndefined();
// No dead-end: no "changed, refresh" alert and no status check that can only repeat it.
expect(button("Check file recovery status")).toBeUndefined();
expect(container.textContent).not.toContain("has changed");
expect(container.textContent).toContain("can’t confirm that this run alone used the sandbox");
expect(api.recover).not.toHaveBeenCalled();
});
it("does not request recovery when the source stops verifying between render and click", async () => {
api.inspect.mockResolvedValueOnce(inspection()).mockResolvedValueOnce({ ...inspection(), sourceInspection: {
...inspection().sourceInspection, status: "blocked", failureCode: "allocation_ownership_ambiguous" } });
await mount();
await click("Recover saved files");
expect(api.recover).not.toHaveBeenCalled();
expect(container.textContent).toContain("can’t confirm that this run alone used the sandbox");
expect(container.textContent).not.toContain("File recovery changed");
expect(button("Recover saved files")).toBeUndefined();
});
it("keeps naming the unverified source after an accepted request failed", async () => {
const key = "accepted-request";
writePendingSend("workspace-file-recovery:company-1:issue-1:legacy-hold:run-1:board-1",
{ idempotencyKey: key, payload: { actionId: "legacy-hold", runId: "run-1" }, createdAt: Date.now() });
const failed = { ...legacyHold, evidence: { ...legacyHold.evidence, workspaceRecovery: {
...(legacyHold.evidence.workspaceRecovery as object), requestIds: [key] } } } as IssueRecoveryAction;
api.inspect.mockResolvedValue(blockedInspection(failed));
await mount(failed);
expect(container.textContent).toContain("File recovery has not finished");
expect(container.textContent).toContain("can’t confirm that this run alone used the sandbox");
// While recovery is running, the panel shows progress instead of the reason.
const running = { ...failed, evidence: { ...failed.evidence, workspaceRecovery: {
...(failed.evidence.workspaceRecovery as object), status: "recovering" } } } as IssueRecoveryAction;
api.inspect.mockResolvedValue(blockedInspection(running));
await act(async () => client.invalidateQueries({ queryKey: queryKeys.issues.recoveryActions("issue-1", "legacy-hold", "run-1") }));
await settle();
expect(container.textContent).toContain("Recovering saved files");
expect(container.textContent).not.toContain("can’t confirm that this run alone used the sandbox");
expect(api.recover).not.toHaveBeenCalled();
});
it("keeps ownership of the next action while the source is verified", async () => {
const owns = vi.fn();
await act(async () => root.render(<QueryClientProvider client={client}>
<WorkspaceExportRecovery action={action} issueId="issue-1" companyId="company-1" onRecoveryControlChange={owns} />
</QueryClientProvider>));
await settle();
expect(owns).toHaveBeenLastCalledWith(true);
expect(button("Recover saved files")).toBeDefined();
});
it("reports recovered files even when the source no longer matches after recovery", async () => {
// Preservation completed, so the workspace moved past the
// recorded source and inspection reports a reference mismatch.
const recovered = { ...legacyHold, ownerType: "system", evidence: { ...legacyHold.evidence,
workspaceRecovery: { ...(legacyHold.evidence.workspaceRecovery as object), status: "recovered" },
systemRecovery: { ...(legacyHold.evidence.systemRecovery as object), state: "complete",
preservationOperationIds: ["preservation-1"] } } } as IssueRecoveryAction;
api.inspect.mockResolvedValue(blockedInspection(recovered, "source_binding_invalid"));
await mount(recovered);
expect(container.textContent).toContain("Saved files recovered");
expect(container.textContent).not.toContain("need recovery");
expect(container.textContent).not.toContain("has changed");
expect(button("Recover saved files")).toBeUndefined();
});
it("reloads the task instead of looping when the recovery record was replaced", async () => {
api.inspect.mockResolvedValue({ active: null, actions: [], sourceInspection: undefined });
const invalidate = vi.spyOn(client, "invalidateQueries");
await mount(legacyHold);
await click("Check file recovery status");
expect(container.textContent).not.toContain("Refresh the task");
expect(invalidate).toHaveBeenCalledWith({ queryKey: queryKeys.issues.detail("issue-1") });
});
it("persists the exact request before POST and prevents a double click", async () => {
@@ -288,6 +388,37 @@ describe("governed retained-file recovery", () => {
expect(api.recover).not.toHaveBeenCalled();
});
// Recovery is automatic. No recovery state may tell a person to fix or inspect
// a sandbox, or name an owner or administrator as the one who must act.
const manualCopy = /administrator|\bowner\b|operator assistance|service assistance|service attention|inspect the (run|failed run)|environment/i;
const withRecovery = (value: IssueRecoveryAction, patch: Record<string, unknown>, system?: Record<string, unknown>) => ({
...value, evidence: { ...value.evidence, workspaceRecovery: { ...(value.evidence.workspaceRecovery as object), ...patch },
...(system ? { systemRecovery: { ...(value.evidence.systemRecovery as object), ...system } } : {}) } }) as IssueRecoveryAction;
const operatorAccess = { userId: "board-1", companyIds: ["company-1"], source: "session", isInstanceAdmin: false,
memberships: [{ companyId: "company-1", membershipRole: "operator", status: "active" }] };
const viewerAccess = { ...operatorAccess, memberships: [{ companyId: "company-1", membershipRole: "viewer", status: "active" }] };
it.each([
["verified and retryable", action, inspection(action), operatorAccess],
["unverified ownership", legacyHold, blockedInspection(legacyHold), operatorAccess],
["unverified binding", legacyHold, blockedInspection(legacyHold, "source_binding_invalid"), operatorAccess],
["not inspected", legacyHold, { ...blockedInspection(legacyHold), sourceInspection: { ...blockedInspection(legacyHold).sourceInspection,
status: "not_inspected", reason: "source_limit" } }, operatorAccess],
["queued", withRecovery(legacyHold, { status: "queued" }), blockedInspection(withRecovery(legacyHold, { status: "queued" })), operatorAccess],
["not retryable", withRecovery(legacyHold, { retryable: false }), blockedInspection(withRecovery(legacyHold, { retryable: false })), operatorAccess],
["recovered", withRecovery(legacyHold, { status: "recovered" }, { state: "complete", preservationOperationIds: ["op-1"] }),
blockedInspection(withRecovery(legacyHold, { status: "recovered" }, { state: "complete", preservationOperationIds: ["op-1"] })), operatorAccess],
["viewer", legacyHold, blockedInspection(legacyHold), viewerAccess],
["copyback owner hold for an operator", { ...action, evidence: { ...action.evidence,
workspaceOwnerRecovery: { state: "operator_required", owner: { token: "token" } } } } as IssueRecoveryAction, inspection(action), operatorAccess],
["board-owned status", { ...action, cause: "native_workspace_finalization_owner_unverified", ownerType: "board",
evidence: { runId: "run-1" } } as unknown as IssueRecoveryAction, inspection(action), operatorAccess],
] as const)("renders no manual sandbox or administrator copy: %s", async (_name, value, response, access) => {
client.setQueryData(queryKeys.access.currentBoardAccess, access);
api.inspect.mockResolvedValue(response);
await mount(value);
expect(container.textContent).not.toMatch(manualCopy);
});
describe("previous server stop confirmation", () => {
const ownerAction = { ...action, cause: "native_workspace_finalization_owner_unverified", ownerType: "board", status: "escalated",
evidence: { runId: "run-1", owner: { token: "owner-token", hostname: "secret-host", pid: 2 },
@@ -350,7 +481,7 @@ describe("governed retained-file recovery", () => {
companyIds: ["company-1"], memberships: [{ companyId: "company-1", membershipRole: "operator", status: "active" }] });
await mount(ownerAction);
expect(button("Confirm the previous server stopped")).toBeUndefined();
expect(container.textContent).toContain("owner or administrator can confirm");
expect(container.textContent).toContain("waiting to confirm that the previous server stopped");
expect(api.resume).not.toHaveBeenCalled();
});
});
@@ -20,16 +20,16 @@ describe("WorkspaceExportRecovery", () => {
});
it("does not mistake platform identity for confirmed recovery", async () => {
await mount({ ...action, evidence: { ...action.evidence, owner: { ...(action.evidence.owner as object), worker: { provider: "railway", deploymentId: "old" } } } });
expect(container.textContent).toContain("operator assistance");
expect(container.textContent).toContain("Saved work is not recovered yet");
});
it("states that an operator is needed for legacy receipts", async () => {
await mount(action);
expect(container.textContent).toContain("Workspace recovery needs operator assistance");
expect(container.textContent).toContain("Saved work is not recovered yet");
});
it("keeps export repair out of the task", async () => {
const cause = "native_workspace_sync_out_retry_exhausted";
await mount({ ...action, cause });
expect(container.textContent).toContain("operator assistance");
expect(container.textContent).toContain("Saved work is not recovered yet");
});
it("removes the status when recovery resolves, including updates in place", async () => {
await mount({ ...action, evidence: { ...action.evidence, workspaceOwnerStop: {} } });
+73 -21
View File
@@ -1,6 +1,6 @@
import { useRef, useState } from "react";
import { useLayoutEffect, useRef, useState } from "react";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import type { IssueRecoveryAction } from "@paperclipai/shared";
import type { IssueRecoveryAction, WorkspaceSourceInspection } from "@paperclipai/shared";
import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared";
import { isTransientError } from "../api/errors";
import { useQueryView } from "./QueryView";
@@ -15,37 +15,53 @@ import { createUuid } from "../lib/uuid";
import { Button } from "./ui/button";
import { Textarea } from "./ui/textarea";
/** Reports whether this surface owns the task's next recovery action. The task's
* generic Retry stays hidden only while this surface offers its own control. */
function ReportRecoveryControl({ owns, onChange }: { owns: boolean; onChange?: (owns: boolean) => void }) {
useLayoutEffect(() => {
onChange?.(owns);
return () => onChange?.(false);
}, [owns, onChange]);
return null;
}
/** Requests preservation; the existing worker owns stop, copyback and verification. */
export function WorkspaceExportRecovery({ action, className, issueId, companyId, showStatus = true }: {
export function WorkspaceExportRecovery({ action, className, issueId, companyId, showStatus = true, onRecoveryControlChange }: {
className?: string; action: IssueRecoveryAction | null; issueId?: string; companyId?: string; showStatus?: boolean;
onRecoveryControlChange?: (owns: boolean) => void;
}) {
// A restart killed the copyback owner of retained saved-work recovery. Recovery
// and any accepted Retry wait on its receipt; the confirmation is the one action.
const retainedOwnerToken = action && issueId && companyId && isRetainedExport(action, issueId, companyId)
? retainedOwnerStopToken(action) : null;
if (action && issueId && companyId && retainedOwnerToken) {
return <PreviousServerStopConfirmation key={`${action.id}:${String(action.evidence.runId)}:${retainedOwnerToken}`}
action={action} ownerToken={retainedOwnerToken} issueId={issueId} companyId={companyId} className={className} />;
return <><ReportRecoveryControl owns onChange={onRecoveryControlChange} />
<PreviousServerStopConfirmation key={`${action.id}:${String(action.evidence.runId)}:${retainedOwnerToken}`}
action={action} ownerToken={retainedOwnerToken} issueId={issueId} companyId={companyId} className={className} /></>;
}
if (action && issueId && companyId && isRetainedExport(action, issueId, companyId)) {
return <RetainedFilesRecovery key={`${companyId}:${issueId}:${action.id}:${action.evidence.runId}`}
action={action} issueId={issueId} companyId={companyId} className={className} />;
action={action} issueId={issueId} companyId={companyId} className={className}
onRecoveryControlChange={onRecoveryControlChange} />;
}
const ownerRecovery = action?.cause === "native_workspace_finalization_owner_unverified";
// The task's interruption names this wait; the confirmation is its one action.
if (ownerRecovery && issueId && companyId && isOwnerStopRequired(action, issueId, companyId)) {
return <PreviousServerStopConfirmation key={`${action.id}:${String(action.evidence.runId)}`}
action={action} ownerToken={asObject(action.evidence.owner).token as string}
issueId={issueId} companyId={companyId} className={className} />;
return <><ReportRecoveryControl owns onChange={onRecoveryControlChange} />
<PreviousServerStopConfirmation key={`${action.id}:${String(action.evidence.runId)}`}
action={action} ownerToken={asObject(action.evidence.owner).token as string}
issueId={issueId} companyId={companyId} className={className} /></>;
}
if (!showStatus) return null;
if (!showStatus) return <ReportRecoveryControl owns={false} onChange={onRecoveryControlChange} />;
if (!action || (!ownerRecovery && !isNativeWorkspaceExportRepairCause(action.cause)) || action.ownerType !== "board"
|| !["active", "escalated"].includes(action.status) || typeof action.evidence.runId !== "string") return null;
|| !["active", "escalated"].includes(action.status) || typeof action.evidence.runId !== "string")
return <ReportRecoveryControl owns={false} onChange={onRecoveryControlChange} />;
const queued = !!action.evidence.workspaceOwnerStop || action.wakePolicy?.kind === "resume_native_run";
const automatic = queued || (ownerRecovery && asObject(action.evidence.workspaceOwnerRecovery).state === "resuming");
return <p role="status" className={cn("text-sm text-muted-foreground", className)}>
{automatic ? "Restoring saved work…" : "Workspace recovery needs operator assistance. The agent’s result is saved."}
</p>;
return <><ReportRecoveryControl owns={false} onChange={onRecoveryControlChange} />
<p role="status" className={cn("text-sm text-muted-foreground", className)}>
{automatic ? "Restoring saved work…" : "Saved work is not recovered yet. The agent’s result is saved."}
</p></>;
}
function isRetainedExport(action: IssueRecoveryAction, issueId: string, companyId: string) {
@@ -99,7 +115,7 @@ function PreviousServerStopConfirmation({ action, ownerToken, issueId, companyId
: "The previous server stopped while it was saving this task’s files, and Paperclip can’t verify that it stopped. The agent’s result is saved."}</p>
{confirm.isSuccess ? null : !canConfirm
? <p className="text-xs text-muted-foreground">{accessView.kind === "loading" || accessView.kind === "reconnecting"
? "Checking access…" : "A company owner or administrator can confirm that the previous server stopped."}</p>
? "Checking access…" : "Paperclip is waiting to confirm that the previous server stopped."}</p>
: !open ? <Button variant="outline" size="sm" className="self-start" onClick={() => setOpen(true)}>
Confirm the previous server stopped</Button>
: <form className="flex flex-col gap-2" onSubmit={event => { event.preventDefault(); if (ready && !confirm.isPending) confirm.mutate(); }}>
@@ -120,8 +136,20 @@ function PreviousServerStopConfirmation({ action, ownerToken, issueId, companyId
type RecoveryInput = { actionId: string; runId: string };
function RetainedFilesRecovery({ action, issueId, companyId, className }: {
/** Plain wording for a source inspection that did not verify. The code stays visible for support. */
function unverifiedSourceText(inspected: WorkspaceSourceInspection | undefined): string {
if (!inspected || inspected.status === "verified") return "";
if (inspected.status === "not_inspected")
return "Paperclip can’t check the saved files’ source here because the run recorded too many sources.";
const code = inspected.sourceVerification?.reason ?? inspected.failureCode;
return code === "allocation_ownership_ambiguous"
? "Paperclip can’t confirm that this run alone used the sandbox that holds the saved files."
: `Paperclip can’t verify the saved files’ recorded source (${code}).`;
}
function RetainedFilesRecovery({ action, issueId, companyId, className, onRecoveryControlChange }: {
action: IssueRecoveryAction; issueId: string; companyId: string; className?: string;
onRecoveryControlChange?: (owns: boolean) => void;
}) {
const runId = action.evidence.runId as string;
const owner = useDraftOwner();
@@ -148,15 +176,21 @@ function RetainedFilesRecovery({ action, issueId, companyId, className }: {
return input.actionId === action.id && input.runId === runId;
};
const pending = storageKey ? readPendingSend(storageKey, matchesInput) : null;
// The recovery record is bound by its own identity. Source inspection is a
// separate read-only probe: it gates only the physical file-recovery request.
// It legitimately reports "blocked" after files were recovered, or when a
// sandbox predates ownership receipts; the recorded state stays authoritative.
function boundAction(data: typeof source.data) {
const current = data?.actions.find(item => item.id === action.id);
const inspected = data?.sourceInspection;
const state = asObject(current?.evidence.workspaceRecovery);
return current && isRetainedExport(current, issueId, companyId) && current.evidence.runId === runId
&& state.runId === runId && state.schema === "paperclip.workspace-recovery.v1"
&& inspected?.actionId === action.id && inspected.runId === runId && inspected.status === "verified" ? current : null;
&& inspected?.actionId === action.id && inspected.runId === runId ? current : null;
}
const sourceVerified = (data: typeof source.data) => Boolean(boundAction(data)) && data?.sourceInspection?.status === "verified";
const current = sourceReady ? boundAction(source.data) : null;
const verified = sourceReady && sourceVerified(source.data);
const recovery = asObject(current?.evidence.workspaceRecovery);
const system = asObject(current?.evidence.systemRecovery);
const serverAcknowledged = Boolean(pending && Array.isArray(recovery.requestIds) && recovery.requestIds.includes(pending.idempotencyKey));
@@ -182,6 +216,8 @@ function RetainedFilesRecovery({ action, issueId, companyId, className }: {
const state = asObject(fresh?.evidence.workspaceRecovery);
if (observed.error || !fresh || state.status !== "blocked" || state.retryable !== true)
throw new Error("File recovery changed. Inspect its current status before requesting recovery.");
if (!sourceVerified(observed.data))
throw new Error(unverifiedSourceText(observed.data?.sourceInspection) || "The saved files’ source could not be verified.");
if (!canMutate || !owner || readOwner() !== owner || !storageKey)
throw new Error("Your account changed. Refresh before requesting file recovery.");
const previous = readPendingSend(storageKey, matchesInput);
@@ -217,7 +253,12 @@ function RetainedFilesRecovery({ action, issueId, companyId, className }: {
const observed = await source.refetch();
if (observed.error) throw observed.error;
const fresh = boundAction(observed.data);
if (!fresh) throw new Error("File recovery changed. Refresh the task to inspect the current recovery.");
if (!fresh) {
// The task's current recovery replaced this record. Reload the task so it
// shows that recovery, instead of asking for a refresh that changes nothing.
invalidate();
throw new Error("This file recovery was replaced. Showing the task’s current recovery.");
}
const state = asObject(fresh.evidence.workspaceRecovery);
const ids = state.requestIds;
const unconfirmed = Boolean(pending && Array.isArray(ids) && !ids.includes(pending.idempotencyKey));
@@ -230,14 +271,22 @@ function RetainedFilesRecovery({ action, issueId, companyId, className }: {
const confirmedFailedAttempt = failedReceipt && retryable && system.state === "exhausted";
const waiting = !failedReceipt && !saved && (restoring || acknowledged || serverAcknowledged);
const unconfirmed = Boolean(pending && !acknowledged && !serverAcknowledged);
const sourceLoading = sourceView.kind === "loading" || sourceView.kind === "reconnecting";
// A verified source makes this panel the governed recovery path. Otherwise the
// task's own Retry (renewed, re-verified recovery) remains the next action.
const ownsRecoveryControl = canMutate && (sourceLoading || Boolean(current && verified && !saved));
// `waiting` already excludes a failed accepted request, so the reason stays
// visible after a failed attempt and hides only while recovery is running.
const unverified = Boolean(current && !verified && !saved && !waiting);
return <div className={cn("flex flex-col gap-2 text-sm", className)} data-testid="retained-file-recovery">
<ReportRecoveryControl owns={ownsRecoveryControl} onChange={onRecoveryControlChange} />
<p role="status">{saved ? "Saved files recovered. Task recovery is still being checked."
: failedReceipt ? "File recovery has not finished. The saved source remains protected."
: restoring ? "Recovering saved files. Work remains paused until recovery is verified."
: waiting ? "File recovery is queued. Waiting for verified saved files."
: "The saved files need recovery before work can continue."}</p>
{canMutate ? <>
{current && retryable && !waiting && (!pending || unconfirmed && checkedUnconfirmed || confirmedFailedAttempt) ? <Button variant="outline" size="sm"
{current && verified && retryable && !waiting && (!pending || unconfirmed && checkedUnconfirmed || confirmedFailedAttempt) ? <Button variant="outline" size="sm"
className="self-start" disabled={request.isPending || check.isPending}
onClick={() => request.mutate(confirmedFailedAttempt)}>{request.isPending ? "Requesting file recovery…"
: confirmedFailedAttempt ? "Try file recovery again" : "Recover saved files"}</Button> : null}
@@ -250,8 +299,11 @@ function RetainedFilesRecovery({ action, issueId, companyId, className }: {
{notice ?? (checkError instanceof Error ? checkError.message : "File recovery could not be inspected. Check its status again.")}
</p> : null}
{current && recovery.status === "blocked" && (serverAcknowledged || recovery.retryable !== true)
? <p role="alert">File recovery has not finished. The saved source remains protected; service assistance is needed.</p> : null}
{sourceReady && !current ? <p role="alert">The recorded source is unavailable or has changed. Inspect recovery before continuing.</p> : null}
? <p role="alert">File recovery has not finished. The saved files remain protected and are not applied.</p> : null}
{unverified ? <p role="status" className="text-muted-foreground">
{unverifiedSourceText(source.data?.sourceInspection)} Paperclip does not apply files it can’t verify.
</p> : null}
{sourceReady && !current ? <p role="alert">This file recovery was replaced. Check its status to show the task’s current recovery.</p> : null}
</> : accessView.kind === "error" ? <p role="alert" className="text-sm text-destructive">Task access could not be checked. Refresh before requesting file recovery.</p>
: accessView.kind === "loading" || accessView.kind === "reconnecting" || !owner ? <p role="status" className="text-xs text-muted-foreground">Checking file recovery access…</p>
: <p className="text-xs text-muted-foreground">A board member with task access can request file recovery.</p>}
+9
View File
@@ -39,6 +39,15 @@ describe("optimistic task Retry", () => {
expect(container.textContent).not.toContain("Thinking");
expect(container.textContent).not.toContain("Taking longer");
});
it("refreshes the inspected file recovery after an accepted Retry", async () => {
vi.mocked(issuesApi.retryTask).mockResolvedValue({ requestId: "retry", accepted: true, stale: false,
observedRunId: "failed-run", presentation: null } as Awaited<ReturnType<typeof issuesApi.retryTask>>);
const key = queryKeys.issues.recoveryActions("task", "hold", "failed-run");
client.setQueryData(key, { active: null, actions: [] });
await render();
await act(async () => container.querySelector("button")!.click());
expect(client.getQueryState(key)?.isInvalidated).toBe(true);
});
it("paints Thinking before admission resolves and shares the same request across double clicks", async () => {
let settle!: (value: Awaited<ReturnType<typeof issuesApi.retryTask>>) => void;
vi.mocked(issuesApi.retryTask).mockImplementation(() => new Promise(resolve => { settle = resolve; }));
+5 -2
View File
@@ -4,6 +4,7 @@ import type { Issue } from "@paperclipai/shared";
import { ApiError } from "../api/client";
import { issuesApi } from "../api/issues";
import { queryKeys } from "../lib/queryKeys";
import { createUuid } from "../lib/uuid";
type RetryRequest = { issueId: string; identifier?: string | null; idempotencyKey: string; observedRunId: string | null };
@@ -24,8 +25,10 @@ export function useTaskRetry(issue: Issue | undefined) {
{
for (const ref of refs) queryClient.setQueryData<Issue>(queryKeys.issues.detail(ref), current =>
current ? { ...current, interruption: result.presentation } : current);
// Retry can queue retained-file recovery; refresh its inspected state too.
await Promise.all(refs.flatMap(ref => [queryKeys.issues.detail(ref), queryKeys.issues.runs(ref),
queryKeys.issues.liveRuns(ref), queryKeys.issues.activeRun(ref)]).map(queryKey =>
queryKeys.issues.liveRuns(ref), queryKeys.issues.activeRun(ref),
queryKeys.issues.allRecoveryActions(ref)]).map(queryKey =>
queryClient.invalidateQueries({ queryKey })));
}
if (isCurrent && !request.current) { setOptimistic(false); setTakingLonger(false); }
@@ -73,7 +76,7 @@ export function useTaskRetry(issue: Issue | undefined) {
if (!issue) return;
setReconcile(null);
mutation.reset();
request.current ??= { issueId: issue.id, identifier: issue.identifier, idempotencyKey: crypto.randomUUID(),
request.current ??= { issueId: issue.id, identifier: issue.identifier, idempotencyKey: createUuid(),
observedRunId: issue.interruption?.observedRunId ?? issue.executionBlocker?.runId ?? null };
setOptimistic(true);
setTakingLonger(false);
+1
View File
@@ -310,6 +310,7 @@ export const queryKeys = {
issues: {
recoveryActions: (id: string, actionId: string | null, runId: string | null) =>
["issues", id, "recovery-actions", actionId, runId] as const,
allRecoveryActions: (id: string) => ["issues", id, "recovery-actions"] as const,
list: (companyId: string) => ["issues", companyId] as const,
onboardingFirstTask: (companyId: string) =>
["issues", companyId, "onboarding-first-task"] as const,
+34 -9
View File
@@ -3756,7 +3756,12 @@ describe("IssueDetail", () => {
mockIssuesApi.update.mockReset();
});
it.each(["verified", "unverified", "stale", "denied"] as const)("keeps native export Retry closed with %s preservation access or source", async mode => {
it.each(["verified", "unverified", "unverified-plain-http", "stale", "denied"] as const)("offers exactly one native export recovery control with %s preservation access or source", async mode => {
// Plain-HTTP LAN and tailnet previews have no crypto.randomUUID.
const originalCrypto = globalThis.crypto;
if (mode === "unverified-plain-http")
vi.stubGlobal("crypto", { getRandomValues: originalCrypto.getRandomValues.bind(originalCrypto) });
try {
const action = {
id: "file-recovery", companyId: "company-1", sourceIssueId: "issue-1",
kind: "active_run_watchdog", cause: "native_workspace_sync_out_retry_exhausted",
@@ -3765,9 +3770,11 @@ describe("IssueDetail", () => {
stage: "restoring_files", requestIds: [], retryable: true,
}, systemRecovery: { state: "exhausted", preservationOperationIds: [] } },
} as unknown as NonNullable<Issue["activeRecoveryAction"]>;
const issue = createIssue({ status: "blocked", assigneeAgentId: "agent-1", activeRecoveryAction: action,
interruption: { intentId: "held-input", responsibility: "system", state: "failed", nextRetryAt: null,
canRetry: true, userAction: null, sourceRunIds: ["stopped-run"], observedRunId: "stopped-run" },
const interruption: NonNullable<Issue["interruption"]> = { intentId: "held-input", responsibility: "system", state: "failed", nextRetryAt: null,
canRetry: true, userAction: null, sourceRunIds: ["stopped-run"], observedRunId: "stopped-run",
retryWait: { reason: "saved_work_recovery_failed",
message: "Saving the agent's files failed after automatic retries. Retry to try again; the agent will not repeat its work." } };
const issue = createIssue({ status: "blocked", assigneeAgentId: "agent-1", activeRecoveryAction: action, interruption,
executionBlocker: { runId: "stopped-run", cause: "native_workspace_sync_out_retry_exhausted", canRetry: false, workspaceRepairRequired: true,
workspaceRecovery: { status: "blocked", stage: "restoring_files", canRetry: true } } as Issue["executionBlocker"],
});
@@ -3779,18 +3786,35 @@ describe("IssueDetail", () => {
});
mockIssuesApi.recoveryActions.mockResolvedValue({ active: action, actions: [action], sourceInspection: {
actionId: mode === "stale" ? "other-action" : action.id, runId: "stopped-run",
status: mode === "unverified" ? "blocked" : "verified", scope: "recorded_source_identity",
status: mode.startsWith("unverified") ? "blocked" : "verified", scope: "recorded_source_identity",
} });
mockIssuesApi.recoverWorkspace.mockResolvedValue({ actionId: action.id, status: "queued" });
mockIssuesApi.retryTask.mockResolvedValue({ requestId: "export-retry", accepted: true, stale: false, presentation: interruption });
await act(async () => renderIssueView());
if (mode !== "verified") {
await waitForAssertion(() => expect(container.textContent).toContain(mode === "denied"
? "A board member with task access" : "The recorded source is unavailable or has changed"));
expect([...container.querySelectorAll("button")].some(button => ["Recover saved files", "Retry"].includes(button.textContent ?? ""))).toBe(false);
const labels = () => [...container.querySelectorAll("button")].map(button => button.textContent ?? "");
if (mode === "denied") {
await waitForAssertion(() => expect(container.textContent).toContain("A board member with task access"));
expect(labels().some(label => ["Recover saved files", "Retry"].includes(label))).toBe(false);
expect(mockIssuesApi.recoverWorkspace).not.toHaveBeenCalled();
expect(mockIssuesApi.retryTask).not.toHaveBeenCalled();
return;
}
if (mode !== "verified") {
// The notice says "Retry to try again"; an unverifiable source must not
// leave it without a Retry. The server re-verifies the source.
await waitForAssertion(() => expect(labels()).toContain("Retry"));
expect(labels()).not.toContain("Recover saved files");
expect(container.textContent).not.toContain("Refresh the task");
const retry = [...container.querySelectorAll("button")].find(button => button.textContent === "Retry")!;
await act(async () => retry.click());
await waitForAssertion(() => expect(mockIssuesApi.retryTask).toHaveBeenCalledExactlyOnceWith(issue.id, {
idempotencyKey: expect.stringMatching(/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/),
observedRunId: "stopped-run",
}));
expect(mockIssuesApi.recoverWorkspace).not.toHaveBeenCalled();
expect(mockIssuesApi.resolveRecoveryAction).not.toHaveBeenCalled();
return;
}
await waitForAssertion(() => expect([...container.querySelectorAll("button")]
.some(button => button.textContent === "Recover saved files")).toBe(true));
expect([...container.querySelectorAll("button")].some(button => button.textContent === "Retry")).toBe(false);
@@ -3803,6 +3827,7 @@ describe("IssueDetail", () => {
expect(mockIssuesApi.update).not.toHaveBeenCalled();
expect(issue.executionBlocker?.workspaceRepairRequired).toBe(true);
expect(container.textContent).not.toContain("Saved files recovered");
} finally { vi.unstubAllGlobals(); }
});
it("keeps the existing Retry path for exhausted legacy workspace restoration", async () => {
+5 -1
View File
@@ -3757,6 +3757,9 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS
selectedCompanyId,
boardAccess,
);
// Retained-file recovery reports whether it offers its own control. The task's
// Retry stays visible whenever it does not, so a retryable hold never dead-ends.
const [retainedRecoveryOwnsRetry, setRetainedRecoveryOwnsRetry] = useState(true);
// The break-glass override reconcile is `runtime:manage`-gated server-side, not gated on the
// recovery-resolution permission — so hide its affordance behind the matching client check.
const canManageBoardRuntime = canBoardManageRuntime(
@@ -8181,13 +8184,14 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS
className={shellSectionClass}
issueId={issue.id} companyId={issue.companyId}
showStatus={!issue.interruption && !issue.executionBlocker?.workspaceRecovery}
onRecoveryControlChange={setRetainedRecoveryOwnsRetry}
action={issue.activeRecoveryAction ?? null} />
)}
{resolvedDetailTab === "chat" && (
<ExecutionBlockerNotice className={shellSectionClass} presentation={issue.interruption}
issueId={issue.id} agentName={agentMap.get(issue.assigneeAgentId ?? "")?.name} projectName={issue.project?.name}
canRetry={canResolveBoardRecoveryAction && !(issue.executionBlocker?.workspaceRepairRequired
&& isRetainedWorkspaceExportCause(issue.executionBlocker.cause))} onRetry={taskRetry.retry}
&& isRetainedWorkspaceExportCause(issue.executionBlocker.cause) && retainedRecoveryOwnsRetry)} onRetry={taskRetry.retry}
optimistic={taskRetry.optimistic} takingLonger={taskRetry.takingLonger} error={taskRetry.error} />
)}