From 824b25654ce6bcbe9c3b13dab37aca0c2d60247b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 7 Nov 2026 10:24:03 -0600 Subject: [PATCH] fix(ui): keep a Retry for retained-file holds whose source cannot be verified (#19840) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a native run finishes but its files cannot be saved, the task stays blocked behind a retained-file recovery hold. > - The server records that hold as blocked and retryable, and the task notice tells the user to "Retry to try again". > - The task page hides its own Retry for these holds and relies on the retained-file recovery panel instead. > - That panel only binds the recovery record when a read-only source inspection returns `verified`. > - Inspection can return `blocked` for normal reasons: a sandbox that predates ownership receipts (state saved by an older release), or a workspace that moved on after its files were recovered. > - Then neither surface offers an action, and "Check file recovery status" always answers "File recovery changed. Refresh the task", also after a reload. > - This pull request binds the record by its own identity, uses inspection only to gate the physical file-recovery request, and keeps the task Retry visible whenever the panel has no control of its own. ## Linked Issues or Issue Description Refs #18848 (the change that added the retained-file recovery panel and hid the generic Retry for these holds). **What happened?** A blocked task with an active `native_workspace_sync_out_retry_exhausted` hold showed "The saved files need recovery before work can continue", "The recorded source is unavailable or has changed", and "Saving the agent's files failed after automatic retries. Retry to try again". It showed no Retry and no recover button. "Check file recovery status" always answered "File recovery changed. Refresh the task to inspect the current recovery", and a reload did not change it. The hold was saved by an earlier release: `systemRecovery` exhausted with `allocation_ownership_ambiguous` / `service_required`, and `workspaceRecovery` blocked with `retryable: true`. The source inspection endpoint returned `status: "blocked"` for it. A second task whose files were already recovered showed the same "need recovery" and "has changed" text, because its inspection reported a reference mismatch after recovery. **Expected behavior** A retryable hold always shows a working Retry or recover control. A notice that says "Retry" has a Retry button. Status checks never loop on "changed, refresh". Recovered files show as recovered. When the source cannot be verified, the page names the reason and who must inspect it. **Steps to reproduce** 1. Use an active retained-export hold whose `workspaceRecovery` is blocked and retryable, and whose `systemRecovery` is exhausted. 2. Make the source inspection (`GET /api/issues/:id/recovery-actions?inspectSourceRunId=…&inspectSourceActionId=…`) return `status: "blocked"`. 3. Open the task as a board operator. 4. Before this change, the page shows no Retry and no recover button. "Check file recovery status" reports "File recovery changed. Refresh the task". **Paperclip version or commit** Observed on build `8812c19`. The regressions fail on master `0f5ce5050e` before the production edits. ## What Changed - `WorkspaceExportRecovery`: bind the recovery record by action, run and schema identity, not by inspection status. Inspection status now gates only the physical "Recover saved files" request (and the request re-checks it before POST). - `WorkspaceExportRecovery`: report through `onRecoveryControlChange` whether the panel offers its own control. It reports `true` only while the source is loading or verified and the files are not yet recovered, or while it shows the previous-server stop confirmation. - `IssueDetail`: hide the task Retry for retained-export holds only while the panel reports that it owns the control. Otherwise the existing task Retry (`POST /issues/:id/retry`) shows. The server renews a bounded recovery generation and re-verifies the source. - Show recovered files as recovered even when inspection no longer matches after recovery. - Name an unverifiable source in plain words (with the code for other reasons). State that Paperclip does not apply files it can't verify. - Recovery text never asks a person to inspect a sandbox or environment, and never names an owner or administrator as the one who must act. This covers the retained-file panel, the run source inspector, the copyback wait, and the native cleanup quarantine message. The execution blocker and the recovery action read model describe a spent recovery budget from its current state, so text saved by older releases ("If Retry keeps failing, inspect the run and its environment") is not shown. - Task Retry uses the existing `createUuid()` helper, so it works on plain-HTTP previews without `crypto.randomUUID`. An accepted Retry also refreshes the inspected file-recovery state. After a failed accepted request, the unverified-source reason stays visible; it hides only while recovery runs. - When the recovery record was replaced, "Check file recovery status" reloads the task detail instead of telling the user to refresh. - Tests: component regressions for an old-release hold, a recovered hold with a mismatched source, verified ownership, and a replaced record. A page regression proves Retry is shown and calls `retryTask` for unverified and stale sources. A server safety test proves that Retry for the old-release hold shape is accepted, opens one new bounded generation, queues file recovery again, keeps the hold active, and is idempotent. ## Verification - RED (master `0f5ce5050e` production files, new tests): 5 component tests fail. The failure output shows the exact staging text: "Check file recovery status", "File recovery changed. Refresh the task to inspect the current recovery.", "The recorded source is unavailable or has changed." 2 page tests (unverified and stale) fail with `expected [ 'Check file recovery status', … ] to include 'Retry'`. - GREEN: `cd ui && npx vitest run src/components/WorkspaceExportRecovery.control.test.tsx src/components/WorkspaceExportRecovery.test.tsx src/components/RunWorkspaceRecoverySurface.test.tsx src/components/ExecutionBlockerNotice.test.tsx src/pages/IssueDetail.test.tsx` passes 320 tests. - Server: `cd server && npx vitest run src/services/task-retry.test.ts` passes 81 tests, including the new old-release safety case. This is safety evidence, not a red: the server path was already correct, and the defect was in the UI. - Full UI suite: 11564 pass. 4 tests in `IssuesList.test.tsx` and `CompanyEnvironments.test.tsx` failed under full-suite load and pass in isolation (106/106). They are unrelated to this change. - `pnpm check:token-gates` is clean. `node scripts/run-tsc.mjs -p ui/tsconfig.json --noEmit` reports no errors in the touched files. The remaining errors are local artifacts that were not built (plugin SDK UI entry, generated Storybook catalog). - Staging was read only (GET requests). The inspect request did not loop in a fresh tab: two calls in 30 seconds. Most of the tab's request volume is periodic company issue-list refetches, which this change does not touch. ## Risks - Low risk. This is a UI-only behavior change. Server authorization, source verification, stop receipts, and the bounded recovery policy are unchanged and remain authoritative. - The task Retry is now visible for retained-export holds whose source does not verify. Retry renews one recovery generation. The worker re-runs the same source-identity checks, so Retry cannot bypass an ownership refusal. If the refusal is permanent (for example an ambiguous sandbox allocation), the hold returns to exhausted and the page names the reason. An automatic fallback for such sources is follow-up work. - Wording: a parametrized UI test renders every recovery state and asserts that no administrator, owner, or "inspect the environment" copy appears. A server test proves that a stored old-release message is replaced by state-derived text. Both failed before the change. - Previous-release story: the old-release hold shape (board-owned, `systemRecovery` revision 5 exhausted with `service_required`, `workspaceRecovery` blocked and retryable) is covered by component, page, and server tests. No persisted state shape changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (repository search, file edits, shell, test execution, and a read-only browser). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- server/src/services/issue-recovery-actions.ts | 11 +- .../native-runtime/native-session-executor.ts | 2 +- .../services/system-recovery-state.test.ts | 20 +++ server/src/services/system-recovery-state.ts | 13 ++ server/src/services/system-recovery.ts | 15 +- server/src/services/task-retry.test.ts | 34 +++++ .../services/workspace-recovery-receipt.ts | 10 +- .../workspace-recovery-retry-copyback.test.ts | 2 +- .../workspace-recovery-warm-handoff.test.ts | 4 +- .../RunWorkspaceSourceInspection.tsx | 2 +- .../WorkspaceExportRecovery.control.test.tsx | 135 +++++++++++++++++- .../WorkspaceExportRecovery.test.tsx | 6 +- ui/src/components/WorkspaceExportRecovery.tsx | 94 +++++++++--- ui/src/hooks/useTaskRetry.test.tsx | 9 ++ ui/src/hooks/useTaskRetry.ts | 7 +- ui/src/lib/queryKeys.ts | 1 + ui/src/pages/IssueDetail.test.tsx | 43 ++++-- ui/src/pages/IssueDetail.tsx | 6 +- 18 files changed, 356 insertions(+), 58 deletions(-) diff --git a/server/src/services/issue-recovery-actions.ts b/server/src/services/issue-recovery-actions.ts index f274af2e15c..bddd3c5ec96 100644 --- a/server/src/services/issue-recovery-actions.ts +++ b/server/src/services/issue-recovery-actions.ts @@ -17,6 +17,7 @@ import type { } from "@paperclipai/shared"; import { isNativeWorkspaceFinalizationOperationActive } from "./workspace-operations.js"; +import { exhaustedSystemRecoveryAction, readSystemRecovery } from "./system-recovery-state.js"; const ACTIVE_RECOVERY_ACTION_STATUSES = ["active", "escalated"] as const satisfies readonly IssueRecoveryActionStatus[]; const MAX_UPSERT_RETRIES = 3; @@ -87,6 +88,14 @@ export type ResolveIssueRecoveryActionInput = { resolutionNote?: string | null; }; +/** A spent saved-work recovery is described from its current state. Text saved + * by an older release could ask a person to inspect the run's environment. */ +function readModelNextAction(row: IssueRecoveryActionRow): string { + const system = readSystemRecovery(row.evidence); + return system?.state === "exhausted" && row.evidence?.workspaceRecovery + ? exhaustedSystemRecoveryAction(system) : row.nextAction; +} + function toReadModel(row: IssueRecoveryActionRow): IssueRecoveryAction { return { id: row.id, @@ -103,7 +112,7 @@ function toReadModel(row: IssueRecoveryActionRow): IssueRecoveryAction { cause: row.cause, fingerprint: row.fingerprint, evidence: row.evidence, - nextAction: row.nextAction, + nextAction: readModelNextAction(row), wakePolicy: row.wakePolicy, monitorPolicy: row.monitorPolicy, attemptCount: row.attemptCount, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index dd5765d3eca..2531fe72169 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -7984,7 +7984,7 @@ const NATIVE_STARTUP_MODEL_REJECTED_ACTION = "The provider could not start the selected model before any provider work began. Choose a supported model for this agent, then retry the task. Automatic retries are stopped."; const NATIVE_CLEANUP_OPERATOR_RECOVERY_MESSAGE = - "Send a new message to continue after Paperclip verifies that the previous provider and its tools have stopped. If cleanup cannot be verified, inspect the run and its environment. Clearing a task session does not resolve this quarantine. Automatic retries are stopped."; + "Send a new message to continue after Paperclip verifies that the previous provider and its tools have stopped. Clearing a task session does not resolve this quarantine. Automatic retries are stopped."; const PROVIDER_DURABLE_EVENT_TYPES = NATIVE_PROVIDER_DURABLE_EVENT_TYPES; diff --git a/server/src/services/system-recovery-state.test.ts b/server/src/services/system-recovery-state.test.ts index 04059e8e678..e04ba8ec10a 100644 --- a/server/src/services/system-recovery-state.test.ts +++ b/server/src/services/system-recovery-state.test.ts @@ -2,6 +2,7 @@ import { randomUUID } from "node:crypto"; import { describe, expect, it } from "vitest"; import { createSystemRecovery, failSystemRecovery, readSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "./system-recovery-state.js"; import { exhaustedSystemRecoveryAction } from "./system-recovery.js"; +import { workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js"; describe("durable system recovery budget", () => { it("makes six attempts with exponential delays that survive serialization and exhaustion", () => { @@ -43,6 +44,25 @@ describe("durable system recovery budget", () => { { now: 0, cause: { label: "/home/daytona/secret", code: null, status: null } }); expect(unsafe.failure).not.toHaveProperty("cause"); }); + it("never asks a person to inspect a sandbox, even for text saved by an older release", () => { + const manual = /administrator|\bowner\b|board member|inspect the run|environment/i; + const exhausted = failSystemRecovery(createSystemRecovery(randomUUID()), + new SystemRecoveryFailure("allocation_ownership_ambiguous", "service_required")); + const copyback = failSystemRecovery(createSystemRecovery(randomUUID()), + new SystemRecoveryFailure("copyback_stop_unconfirmed", "recheck_evidence")); + for (const state of [exhausted, { ...copyback, state: "exhausted" as const }]) + expect(exhaustedSystemRecoveryAction(state)).not.toMatch(manual); + const runId = exhausted.sourceRunId; + const display = workspaceRecoveryDisplay({ runId, systemRecovery: exhausted, workspaceRecovery: { + schema: "paperclip.workspace-recovery.v1", runId, status: "blocked", stage: "verifying_stop", attempt: 1, + attemptLimit: 6, requestIds: [], retryable: true, + message: "Automatic recovery of saved work stopped after 1 attempts (last failure: allocation_ownership_ambiguous). " + + "Retry recovery to export and merge from the retained sandbox again; no provider turn is submitted. " + + "If Retry keeps failing, inspect the run and its environment." } }, runId); + expect(display?.message).toContain("Retry to try again"); + expect(display?.message).not.toMatch(manual); + }); + it("names the last failure and its cause in the board handoff", () => { const state = reserveSystemRecoveryAttempt(createSystemRecovery(randomUUID()), 0)!; const withCause = failSystemRecovery(state, new SystemRecoveryFailure("provider_unavailable", "retry_transient"), diff --git a/server/src/services/system-recovery-state.ts b/server/src/services/system-recovery-state.ts index 411f2cc43f2..703bf0fa9b6 100644 --- a/server/src/services/system-recovery-state.ts +++ b/server/src/services/system-recovery-state.ts @@ -126,3 +126,16 @@ export function failSystemRecovery( evidenceFingerprint: options.evidenceFingerprint ?? createHash("sha256").update(`${state.sourceRunId}:${failure.code}`).digest("hex") }, }; } + +/** User-facing text for a spent recovery budget. It names what happens next and + * never asks a person to inspect a sandbox or environment. The code stays for support. */ +export function exhaustedSystemRecoveryAction(state: SystemRecoveryState): string { + // Another Retry cannot release a copyback owner that stopped without reporting. + if (state.failure?.code === "copyback_stop_unconfirmed") + return "Recovery of saved work is waiting for the previous server's file copy to stop. " + + "Recovery resumes once Paperclip can confirm that it stopped. No provider turn is submitted."; + const cause = state.failure?.cause ? `, cause: ${state.failure.cause.label}` : ""; + const reason = state.failure?.code ? ` (last failure: ${state.failure.code}${cause})` : ""; + return `Automatic recovery of saved work stopped after ${state.attempts} attempts${reason}. ` + + "Retry to try again; the agent will not repeat its work."; +} diff --git a/server/src/services/system-recovery.ts b/server/src/services/system-recovery.ts index 3edd496a1d3..d816e32c049 100644 --- a/server/src/services/system-recovery.ts +++ b/server/src/services/system-recovery.ts @@ -3,7 +3,7 @@ import { and, eq, sql } from "drizzle-orm"; import { agents, issues, heartbeatRuns, issueComments, issueRecoveryActions, type Db } from "@paperclipai/db"; import { writeExecutionControl } from "./execution-control-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; -import { createSystemRecovery, readSystemRecovery, type SystemRecoveryState } from "./system-recovery-state.js"; +import { createSystemRecovery, exhaustedSystemRecoveryAction, readSystemRecovery, type SystemRecoveryState } from "./system-recovery-state.js"; import { workspaceRecoverySourceEligible, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js"; export async function adoptSystemRecovery(db: Db, source: WorkspaceRecoverySourceIdentity) { @@ -47,18 +47,7 @@ export async function adoptSystemRecovery(db: Db, source: WorkspaceRecoverySourc }); } -export function exhaustedSystemRecoveryAction(state: SystemRecoveryState): string { - // Another Retry cannot release a copyback owner that stopped without reporting. - if (state.failure?.code === "copyback_stop_unconfirmed") - return "Automatic recovery of saved work is waiting for the previous server's file copy to stop, and Paperclip cannot verify " - + "that it stopped. A board member who manages this company's runtime can confirm that it stopped; recovery then " - + "resumes. No provider turn is submitted."; - const cause = state.failure?.cause ? `, cause: ${state.failure.cause.label}` : ""; - const reason = state.failure?.code ? ` (last failure: ${state.failure.code}${cause})` : ""; - return `Automatic recovery of saved work stopped after ${state.attempts} attempts${reason}. ` - + "Retry recovery to export and merge from the retained sandbox again; no provider turn is submitted. " - + "If Retry keeps failing, inspect the run and its environment."; -} +export { exhaustedSystemRecoveryAction }; /** CAS protects a newer manual generation from an old worker's late write. * This is bookkeeping authority only; physical writes still require ownership. */ diff --git a/server/src/services/task-retry.test.ts b/server/src/services/task-retry.test.ts index b2b2c42a25e..cbc567f6aa3 100644 --- a/server/src/services/task-retry.test.ts +++ b/server/src/services/task-retry.test.ts @@ -678,6 +678,40 @@ describe("durable task Retry admission", () => { expect(wakes.find(row => row.id === first.requestId)?.status).toBe("failed"); expect(wakes.filter(row => row.status === "deferred_issue_execution")).toHaveLength(1); }); + it("accepts Retry for an old-release retained-export hold whose source ownership could not be verified", async () => { + // State saved by an earlier release: a board-owned export hold whose only + // attempt ended with a permanent ownership refusal. The task UI now offers + // Retry for it; Retry must open one new bounded generation that re-verifies + // the source, keep the hold in place, and never resolve or bypass it. + const input = await seed(); + const previous = { ...failSystemRecovery({ ...createSystemRecovery(input.observedRunId), revision: 4 }, + new SystemRecoveryFailure("allocation_ownership_ambiguous", "service_required")), revision: 5 }; + expect(previous.state).toBe("exhausted"); + const [action] = await db.insert(issueRecoveryActions).values({ + companyId: input.companyId, sourceIssueId: input.issueId, kind: "active_run_watchdog", status: "active", + ownerType: "board", cause: "native_workspace_sync_out_retry_exhausted", fingerprint: randomUUID(), + nextAction: "Automatic recovery of saved work stopped after 1 attempts (last failure: allocation_ownership_ambiguous).", + evidence: { runId: input.observedRunId, systemRecovery: previous, + workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: input.observedRunId, stage: "verifying_stop", + status: "blocked", attempt: 1, attemptLimit: 6, requestIds: [], retryable: true, message: "Retry recovery." }, + coordinatorAttempt: 1, originalFailureCode: "native_workspace_sync_out_failed", + workspaceFinalizeAttempt: 3, previousWorkspaceRecovery: null }, + }).returning(); + const receipt = await requestTaskRetry(input); + expect(receipt).toMatchObject({ accepted: true, stale: false, observedRunId: input.observedRunId }); + const [updated] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)); + expect(updated.status).toBe("active"); + expect(updated.resolvedAt).toBeNull(); + expect(readSystemRecovery(updated.evidence)).toMatchObject({ generationId: receipt.requestId, sourceRunId: input.observedRunId, + attempts: 0, failure: null, revision: 6, freshAttemptRunId: null, preservationOperationIds: [] }); + // File recovery is queued again for the worker, which re-verifies the source. + expect(updated.evidence.workspaceRecovery).toMatchObject({ runId: input.observedRunId, status: "queued", + stage: "verifying_stop", attemptLimit: 6, retryable: true }); + // A replayed click is idempotent and does not open another generation. + expect((await requestTaskRetry(input)).requestId).toBe(receipt.requestId); + const [replayed] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)); + expect(readSystemRecovery(replayed.evidence)?.revision).toBe(6); + }); it("binds a key to its company, task and actor across all statuses", async () => { const input = await seed(); const receipt = await requestTaskRetry(input); diff --git a/server/src/services/workspace-recovery-receipt.ts b/server/src/services/workspace-recovery-receipt.ts index f0dda4daff1..1126d87aa97 100644 --- a/server/src/services/workspace-recovery-receipt.ts +++ b/server/src/services/workspace-recovery-receipt.ts @@ -5,6 +5,7 @@ import { z } from "zod"; import { readNativeWorkspaceSyncReference, type NativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js"; import { readMigratedLegacyWorkspaceSource } from "./legacy-workspace-recovery-migration.js"; import { readSnapshot } from "./workspace-recovery-snapshot.js"; +import { exhaustedSystemRecoveryAction, readSystemRecovery } from "./system-recovery-state.js"; export const workspaceRecoverySchema = z.object({ schema: z.literal("paperclip.workspace-recovery.v1"), @@ -58,10 +59,17 @@ export function workspaceRecoveryDisplay(evidence: Record, runI canRetry: state.status === "blocked" && state.retryable === true, message: state.status === "queued" || state.status === "recovering" ? "Recovering saved work…" : state.status === "recovered" ? "Saved work recovered. Send a message to continue." - : state.message ?? "Paperclip couldn’t safely recover this task. This task needs service attention.", + : blockedWorkspaceRecoveryMessage(evidence) ?? state.message ?? "Recovery of this task's saved work has not finished.", }; } +/** Text saved by an older release can name a manual step. A spent budget is + * described from its current state instead of the stored text. */ +export function blockedWorkspaceRecoveryMessage(evidence: Record | null | undefined): string | null { + const system = readSystemRecovery(evidence ?? undefined); + return system?.state === "exhausted" ? exhaustedSystemRecoveryAction(system) : null; +} + async function verifiedNativeCopybackReceipt(db: Db, input: { companyId: string; issueId: string; runId: string; reference: NativeWorkspaceSyncReference; }) { diff --git a/server/src/services/workspace-recovery-retry-copyback.test.ts b/server/src/services/workspace-recovery-retry-copyback.test.ts index 101ae89912b..cb0a5895304 100644 --- a/server/src/services/workspace-recovery-retry-copyback.test.ts +++ b/server/src/services/workspace-recovery-retry-copyback.test.ts @@ -508,7 +508,7 @@ describe("automatic retained workspace recovery", () => { const stable = (row: typeof after) => ({ ...row, updatedAt: null, ownerType: null, nextAction: null, evidence: { ...row.evidence, workspaceRecovery: { ...(row.evidence.workspaceRecovery as object), message: null } } }); if (spent) { - expect(after).toMatchObject({ ownerType: "board", nextAction: expect.stringMatching(/Retry recovery/) }); + expect(after).toMatchObject({ ownerType: "board", nextAction: expect.stringMatching(/Retry to try again/) }); expect(stable(after)).toEqual(stable(before)); } else expect(after).toEqual(before); expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].leasePolicy).toBe("reuse_by_environment"); diff --git a/server/src/services/workspace-recovery-warm-handoff.test.ts b/server/src/services/workspace-recovery-warm-handoff.test.ts index eb4b7e51706..bfe5a9c1534 100644 --- a/server/src/services/workspace-recovery-warm-handoff.test.ts +++ b/server/src/services/workspace-recovery-warm-handoff.test.ts @@ -474,7 +474,7 @@ describe("automatic retained workspace recovery", () => { expect(exhausted.nextAction).toMatch(/stopped after 6 attempts/); const blocker = await getExecutionBlocker(db, f.companyId, f.issueId); expect(blocker?.workspaceRecovery).toMatchObject({ status: "blocked", canRetry: true }); - expect(blocker?.nextAction).toMatch(/Retry recovery/); + expect(blocker?.nextAction).toMatch(/Retry to try again/); expect(blocker?.nextAction).not.toMatch(/send a (new )?message/i); // Nothing physical or provider-facing ran after the budget was spent. expect(f.runtime.resumeRunLease).not.toHaveBeenCalled(); @@ -501,7 +501,7 @@ describe("automatic retained workspace recovery", () => { const handed = await action(f.actionId); expect(handed).toMatchObject({ status: "active", ownerType: "board" }); expect(readSystemRecovery(handed.evidence)).toEqual(spent); - expect((await getExecutionBlocker(db, f.companyId, f.issueId))?.nextAction).toMatch(/Retry recovery/); + expect((await getExecutionBlocker(db, f.companyId, f.issueId))?.nextAction).toMatch(/Retry to try again/); expect(f.runtime.resumeRunLease).not.toHaveBeenCalled(); // Repeat sweeps are idempotent and still grant no attempt. await recoverRetainedWorkspace(f.input); diff --git a/ui/src/components/RunWorkspaceSourceInspection.tsx b/ui/src/components/RunWorkspaceSourceInspection.tsx index 0ba5812fd62..c9347a633dd 100644 --- a/ui/src/components/RunWorkspaceSourceInspection.tsx +++ b/ui/src/components/RunWorkspaceSourceInspection.tsx @@ -51,7 +51,7 @@ export function RunWorkspaceSourceInspection({ issueId, runId, companyId }: { is

Action {current.actionId}

This read-only observation checks recorded identity. It does not confirm provider termination, local files, or successful preservation.

{current.status === "verified" ?

Recorded-source identity verified.

- : current.status === "not_inspected" ?

Source not inspected: this record exceeds the inspection limit. Service attention is required.

+ : current.status === "not_inspected" ?

Source not inspected: this record exceeds the inspection limit.

:

Blocked: {current.sourceVerification?.reason ?? current.failureCode} {current.sourceVerification?.observedLeasePolicy ? <> · lease policy {current.sourceVerification.observedLeasePolicy} : null}

}

Observed {formatDateTime(current.observedAt)}

diff --git a/ui/src/components/WorkspaceExportRecovery.control.test.tsx b/ui/src/components/WorkspaceExportRecovery.control.test.tsx index 0a10fc87c7f..49ef3c4cb01 100644 --- a/ui/src/components/WorkspaceExportRecovery.control.test.tsx +++ b/ui/src/components/WorkspaceExportRecovery.control.test.tsx @@ -6,6 +6,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import type { IssueRecoveryAction } from "@paperclipai/shared"; import { ApiError } from "../api/client"; import { queryKeys } from "../lib/queryKeys"; +import { writePendingSend } from "../lib/pending-send"; import { WorkspaceExportRecovery } from "./WorkspaceExportRecovery"; const api = vi.hoisted(() => ({ inspect: vi.fn(), recover: vi.fn(), resume: vi.fn() })); @@ -153,7 +154,106 @@ describe("governed retained-file recovery", () => { } }); await mount(); expect(button("Recover saved files")).toBeUndefined(); - expect(container.textContent).toContain("source is unavailable or has changed"); + expect(container.textContent).toContain("can’t verify the saved files’ recorded source (source_binding_invalid)"); + }); + + // Recovery saved by an older release: a board-owned hold whose + // sandbox predates ownership receipts. The server still records a retryable + // blocked recovery, but the read-only source probe cannot verify ownership. + const legacyHold = { ...action, id: "legacy-hold", ownerType: "board", evidence: { runId: "run-1", + workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: "run-1", stage: "verifying_stop", + status: "blocked", attempt: 1, attemptLimit: 6, requestIds: [], retryable: true, + message: "Automatic recovery of saved work stopped after 1 attempts (last failure: allocation_ownership_ambiguous)." }, + systemRecovery: { schema: "paperclip.system-recovery.v1", phase: "verify_execution", state: "exhausted", revision: 5, + sourceRunId: "run-1", attempts: 1, retriesUsed: 0, preservationOperationIds: [], freshAttemptRunId: null, + failure: { code: "allocation_ownership_ambiguous", retryClass: "service_required" } }, + originalFailureCode: "native_workspace_sync_out_failed", workspaceFinalizeAttempt: 3, previousWorkspaceRecovery: null }, + } as unknown as IssueRecoveryAction; + const blockedInspection = (value: IssueRecoveryAction, failureCode = "allocation_ownership_ambiguous") => ({ + active: value, actions: [value], sourceInspection: { schema: "paperclip.workspace-source-inspection.v1", + actionId: value.id, runId: "run-1", status: "blocked", failureCode, + scope: "recorded_source_identity", observedAt: "2026-11-07T14:44:46Z" } }); + + it("hands the next action to the task Retry for an old-release hold whose source cannot be verified", async () => { + api.inspect.mockResolvedValue(blockedInspection(legacyHold)); + const owns = vi.fn(); + await act(async () => root.render( + + )); + await settle(); + expect(owns).toHaveBeenLastCalledWith(false); + expect(button("Recover saved files")).toBeUndefined(); + // No dead-end: no "changed, refresh" alert and no status check that can only repeat it. + expect(button("Check file recovery status")).toBeUndefined(); + expect(container.textContent).not.toContain("has changed"); + expect(container.textContent).toContain("can’t confirm that this run alone used the sandbox"); + expect(api.recover).not.toHaveBeenCalled(); + }); + + it("does not request recovery when the source stops verifying between render and click", async () => { + api.inspect.mockResolvedValueOnce(inspection()).mockResolvedValueOnce({ ...inspection(), sourceInspection: { + ...inspection().sourceInspection, status: "blocked", failureCode: "allocation_ownership_ambiguous" } }); + await mount(); + await click("Recover saved files"); + expect(api.recover).not.toHaveBeenCalled(); + expect(container.textContent).toContain("can’t confirm that this run alone used the sandbox"); + expect(container.textContent).not.toContain("File recovery changed"); + expect(button("Recover saved files")).toBeUndefined(); + }); + + it("keeps naming the unverified source after an accepted request failed", async () => { + const key = "accepted-request"; + writePendingSend("workspace-file-recovery:company-1:issue-1:legacy-hold:run-1:board-1", + { idempotencyKey: key, payload: { actionId: "legacy-hold", runId: "run-1" }, createdAt: Date.now() }); + const failed = { ...legacyHold, evidence: { ...legacyHold.evidence, workspaceRecovery: { + ...(legacyHold.evidence.workspaceRecovery as object), requestIds: [key] } } } as IssueRecoveryAction; + api.inspect.mockResolvedValue(blockedInspection(failed)); + await mount(failed); + expect(container.textContent).toContain("File recovery has not finished"); + expect(container.textContent).toContain("can’t confirm that this run alone used the sandbox"); + // While recovery is running, the panel shows progress instead of the reason. + const running = { ...failed, evidence: { ...failed.evidence, workspaceRecovery: { + ...(failed.evidence.workspaceRecovery as object), status: "recovering" } } } as IssueRecoveryAction; + api.inspect.mockResolvedValue(blockedInspection(running)); + await act(async () => client.invalidateQueries({ queryKey: queryKeys.issues.recoveryActions("issue-1", "legacy-hold", "run-1") })); + await settle(); + expect(container.textContent).toContain("Recovering saved files"); + expect(container.textContent).not.toContain("can’t confirm that this run alone used the sandbox"); + expect(api.recover).not.toHaveBeenCalled(); + }); + + it("keeps ownership of the next action while the source is verified", async () => { + const owns = vi.fn(); + await act(async () => root.render( + + )); + await settle(); + expect(owns).toHaveBeenLastCalledWith(true); + expect(button("Recover saved files")).toBeDefined(); + }); + + it("reports recovered files even when the source no longer matches after recovery", async () => { + // Preservation completed, so the workspace moved past the + // recorded source and inspection reports a reference mismatch. + const recovered = { ...legacyHold, ownerType: "system", evidence: { ...legacyHold.evidence, + workspaceRecovery: { ...(legacyHold.evidence.workspaceRecovery as object), status: "recovered" }, + systemRecovery: { ...(legacyHold.evidence.systemRecovery as object), state: "complete", + preservationOperationIds: ["preservation-1"] } } } as IssueRecoveryAction; + api.inspect.mockResolvedValue(blockedInspection(recovered, "source_binding_invalid")); + await mount(recovered); + expect(container.textContent).toContain("Saved files recovered"); + expect(container.textContent).not.toContain("need recovery"); + expect(container.textContent).not.toContain("has changed"); + expect(button("Recover saved files")).toBeUndefined(); + }); + + it("reloads the task instead of looping when the recovery record was replaced", async () => { + api.inspect.mockResolvedValue({ active: null, actions: [], sourceInspection: undefined }); + const invalidate = vi.spyOn(client, "invalidateQueries"); + await mount(legacyHold); + await click("Check file recovery status"); + expect(container.textContent).not.toContain("Refresh the task"); + expect(invalidate).toHaveBeenCalledWith({ queryKey: queryKeys.issues.detail("issue-1") }); }); it("persists the exact request before POST and prevents a double click", async () => { @@ -288,6 +388,37 @@ describe("governed retained-file recovery", () => { expect(api.recover).not.toHaveBeenCalled(); }); + // Recovery is automatic. No recovery state may tell a person to fix or inspect + // a sandbox, or name an owner or administrator as the one who must act. + const manualCopy = /administrator|\bowner\b|operator assistance|service assistance|service attention|inspect the (run|failed run)|environment/i; + const withRecovery = (value: IssueRecoveryAction, patch: Record, system?: Record) => ({ + ...value, evidence: { ...value.evidence, workspaceRecovery: { ...(value.evidence.workspaceRecovery as object), ...patch }, + ...(system ? { systemRecovery: { ...(value.evidence.systemRecovery as object), ...system } } : {}) } }) as IssueRecoveryAction; + const operatorAccess = { userId: "board-1", companyIds: ["company-1"], source: "session", isInstanceAdmin: false, + memberships: [{ companyId: "company-1", membershipRole: "operator", status: "active" }] }; + const viewerAccess = { ...operatorAccess, memberships: [{ companyId: "company-1", membershipRole: "viewer", status: "active" }] }; + it.each([ + ["verified and retryable", action, inspection(action), operatorAccess], + ["unverified ownership", legacyHold, blockedInspection(legacyHold), operatorAccess], + ["unverified binding", legacyHold, blockedInspection(legacyHold, "source_binding_invalid"), operatorAccess], + ["not inspected", legacyHold, { ...blockedInspection(legacyHold), sourceInspection: { ...blockedInspection(legacyHold).sourceInspection, + status: "not_inspected", reason: "source_limit" } }, operatorAccess], + ["queued", withRecovery(legacyHold, { status: "queued" }), blockedInspection(withRecovery(legacyHold, { status: "queued" })), operatorAccess], + ["not retryable", withRecovery(legacyHold, { retryable: false }), blockedInspection(withRecovery(legacyHold, { retryable: false })), operatorAccess], + ["recovered", withRecovery(legacyHold, { status: "recovered" }, { state: "complete", preservationOperationIds: ["op-1"] }), + blockedInspection(withRecovery(legacyHold, { status: "recovered" }, { state: "complete", preservationOperationIds: ["op-1"] })), operatorAccess], + ["viewer", legacyHold, blockedInspection(legacyHold), viewerAccess], + ["copyback owner hold for an operator", { ...action, evidence: { ...action.evidence, + workspaceOwnerRecovery: { state: "operator_required", owner: { token: "token" } } } } as IssueRecoveryAction, inspection(action), operatorAccess], + ["board-owned status", { ...action, cause: "native_workspace_finalization_owner_unverified", ownerType: "board", + evidence: { runId: "run-1" } } as unknown as IssueRecoveryAction, inspection(action), operatorAccess], + ] as const)("renders no manual sandbox or administrator copy: %s", async (_name, value, response, access) => { + client.setQueryData(queryKeys.access.currentBoardAccess, access); + api.inspect.mockResolvedValue(response); + await mount(value); + expect(container.textContent).not.toMatch(manualCopy); + }); + describe("previous server stop confirmation", () => { const ownerAction = { ...action, cause: "native_workspace_finalization_owner_unverified", ownerType: "board", status: "escalated", evidence: { runId: "run-1", owner: { token: "owner-token", hostname: "secret-host", pid: 2 }, @@ -350,7 +481,7 @@ describe("governed retained-file recovery", () => { companyIds: ["company-1"], memberships: [{ companyId: "company-1", membershipRole: "operator", status: "active" }] }); await mount(ownerAction); expect(button("Confirm the previous server stopped")).toBeUndefined(); - expect(container.textContent).toContain("owner or administrator can confirm"); + expect(container.textContent).toContain("waiting to confirm that the previous server stopped"); expect(api.resume).not.toHaveBeenCalled(); }); }); diff --git a/ui/src/components/WorkspaceExportRecovery.test.tsx b/ui/src/components/WorkspaceExportRecovery.test.tsx index 28a2409d2f1..3c909c334b5 100644 --- a/ui/src/components/WorkspaceExportRecovery.test.tsx +++ b/ui/src/components/WorkspaceExportRecovery.test.tsx @@ -20,16 +20,16 @@ describe("WorkspaceExportRecovery", () => { }); it("does not mistake platform identity for confirmed recovery", async () => { await mount({ ...action, evidence: { ...action.evidence, owner: { ...(action.evidence.owner as object), worker: { provider: "railway", deploymentId: "old" } } } }); - expect(container.textContent).toContain("operator assistance"); + expect(container.textContent).toContain("Saved work is not recovered yet"); }); it("states that an operator is needed for legacy receipts", async () => { await mount(action); - expect(container.textContent).toContain("Workspace recovery needs operator assistance"); + expect(container.textContent).toContain("Saved work is not recovered yet"); }); it("keeps export repair out of the task", async () => { const cause = "native_workspace_sync_out_retry_exhausted"; await mount({ ...action, cause }); - expect(container.textContent).toContain("operator assistance"); + expect(container.textContent).toContain("Saved work is not recovered yet"); }); it("removes the status when recovery resolves, including updates in place", async () => { await mount({ ...action, evidence: { ...action.evidence, workspaceOwnerStop: {} } }); diff --git a/ui/src/components/WorkspaceExportRecovery.tsx b/ui/src/components/WorkspaceExportRecovery.tsx index bf48de41db2..e66e04707ad 100644 --- a/ui/src/components/WorkspaceExportRecovery.tsx +++ b/ui/src/components/WorkspaceExportRecovery.tsx @@ -1,6 +1,6 @@ -import { useRef, useState } from "react"; +import { useLayoutEffect, useRef, useState } from "react"; import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; -import type { IssueRecoveryAction } from "@paperclipai/shared"; +import type { IssueRecoveryAction, WorkspaceSourceInspection } from "@paperclipai/shared"; import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared"; import { isTransientError } from "../api/errors"; import { useQueryView } from "./QueryView"; @@ -15,37 +15,53 @@ import { createUuid } from "../lib/uuid"; import { Button } from "./ui/button"; import { Textarea } from "./ui/textarea"; +/** Reports whether this surface owns the task's next recovery action. The task's + * generic Retry stays hidden only while this surface offers its own control. */ +function ReportRecoveryControl({ owns, onChange }: { owns: boolean; onChange?: (owns: boolean) => void }) { + useLayoutEffect(() => { + onChange?.(owns); + return () => onChange?.(false); + }, [owns, onChange]); + return null; +} + /** Requests preservation; the existing worker owns stop, copyback and verification. */ -export function WorkspaceExportRecovery({ action, className, issueId, companyId, showStatus = true }: { +export function WorkspaceExportRecovery({ action, className, issueId, companyId, showStatus = true, onRecoveryControlChange }: { className?: string; action: IssueRecoveryAction | null; issueId?: string; companyId?: string; showStatus?: boolean; + onRecoveryControlChange?: (owns: boolean) => void; }) { // A restart killed the copyback owner of retained saved-work recovery. Recovery // and any accepted Retry wait on its receipt; the confirmation is the one action. const retainedOwnerToken = action && issueId && companyId && isRetainedExport(action, issueId, companyId) ? retainedOwnerStopToken(action) : null; if (action && issueId && companyId && retainedOwnerToken) { - return ; + return <> + ; } if (action && issueId && companyId && isRetainedExport(action, issueId, companyId)) { return ; + action={action} issueId={issueId} companyId={companyId} className={className} + onRecoveryControlChange={onRecoveryControlChange} />; } const ownerRecovery = action?.cause === "native_workspace_finalization_owner_unverified"; // The task's interruption names this wait; the confirmation is its one action. if (ownerRecovery && issueId && companyId && isOwnerStopRequired(action, issueId, companyId)) { - return ; + return <> + ; } - if (!showStatus) return null; + if (!showStatus) return ; if (!action || (!ownerRecovery && !isNativeWorkspaceExportRepairCause(action.cause)) || action.ownerType !== "board" - || !["active", "escalated"].includes(action.status) || typeof action.evidence.runId !== "string") return null; + || !["active", "escalated"].includes(action.status) || typeof action.evidence.runId !== "string") + return ; const queued = !!action.evidence.workspaceOwnerStop || action.wakePolicy?.kind === "resume_native_run"; const automatic = queued || (ownerRecovery && asObject(action.evidence.workspaceOwnerRecovery).state === "resuming"); - return

- {automatic ? "Restoring saved work…" : "Workspace recovery needs operator assistance. The agent’s result is saved."} -

; + return <> +

+ {automatic ? "Restoring saved work…" : "Saved work is not recovered yet. The agent’s result is saved."} +

; } function isRetainedExport(action: IssueRecoveryAction, issueId: string, companyId: string) { @@ -99,7 +115,7 @@ function PreviousServerStopConfirmation({ action, ownerToken, issueId, companyId : "The previous server stopped while it was saving this task’s files, and Paperclip can’t verify that it stopped. The agent’s result is saved."}

{confirm.isSuccess ? null : !canConfirm ?

{accessView.kind === "loading" || accessView.kind === "reconnecting" - ? "Checking access…" : "A company owner or administrator can confirm that the previous server stopped."}

+ ? "Checking access…" : "Paperclip is waiting to confirm that the previous server stopped."}

: !open ? :
{ event.preventDefault(); if (ready && !confirm.isPending) confirm.mutate(); }}> @@ -120,8 +136,20 @@ function PreviousServerStopConfirmation({ action, ownerToken, issueId, companyId type RecoveryInput = { actionId: string; runId: string }; -function RetainedFilesRecovery({ action, issueId, companyId, className }: { +/** Plain wording for a source inspection that did not verify. The code stays visible for support. */ +function unverifiedSourceText(inspected: WorkspaceSourceInspection | undefined): string { + if (!inspected || inspected.status === "verified") return ""; + if (inspected.status === "not_inspected") + return "Paperclip can’t check the saved files’ source here because the run recorded too many sources."; + const code = inspected.sourceVerification?.reason ?? inspected.failureCode; + return code === "allocation_ownership_ambiguous" + ? "Paperclip can’t confirm that this run alone used the sandbox that holds the saved files." + : `Paperclip can’t verify the saved files’ recorded source (${code}).`; +} + +function RetainedFilesRecovery({ action, issueId, companyId, className, onRecoveryControlChange }: { action: IssueRecoveryAction; issueId: string; companyId: string; className?: string; + onRecoveryControlChange?: (owns: boolean) => void; }) { const runId = action.evidence.runId as string; const owner = useDraftOwner(); @@ -148,15 +176,21 @@ function RetainedFilesRecovery({ action, issueId, companyId, className }: { return input.actionId === action.id && input.runId === runId; }; const pending = storageKey ? readPendingSend(storageKey, matchesInput) : null; + // The recovery record is bound by its own identity. Source inspection is a + // separate read-only probe: it gates only the physical file-recovery request. + // It legitimately reports "blocked" after files were recovered, or when a + // sandbox predates ownership receipts; the recorded state stays authoritative. function boundAction(data: typeof source.data) { const current = data?.actions.find(item => item.id === action.id); const inspected = data?.sourceInspection; const state = asObject(current?.evidence.workspaceRecovery); return current && isRetainedExport(current, issueId, companyId) && current.evidence.runId === runId && state.runId === runId && state.schema === "paperclip.workspace-recovery.v1" - && inspected?.actionId === action.id && inspected.runId === runId && inspected.status === "verified" ? current : null; + && inspected?.actionId === action.id && inspected.runId === runId ? current : null; } + const sourceVerified = (data: typeof source.data) => Boolean(boundAction(data)) && data?.sourceInspection?.status === "verified"; const current = sourceReady ? boundAction(source.data) : null; + const verified = sourceReady && sourceVerified(source.data); const recovery = asObject(current?.evidence.workspaceRecovery); const system = asObject(current?.evidence.systemRecovery); const serverAcknowledged = Boolean(pending && Array.isArray(recovery.requestIds) && recovery.requestIds.includes(pending.idempotencyKey)); @@ -182,6 +216,8 @@ function RetainedFilesRecovery({ action, issueId, companyId, className }: { const state = asObject(fresh?.evidence.workspaceRecovery); if (observed.error || !fresh || state.status !== "blocked" || state.retryable !== true) throw new Error("File recovery changed. Inspect its current status before requesting recovery."); + if (!sourceVerified(observed.data)) + throw new Error(unverifiedSourceText(observed.data?.sourceInspection) || "The saved files’ source could not be verified."); if (!canMutate || !owner || readOwner() !== owner || !storageKey) throw new Error("Your account changed. Refresh before requesting file recovery."); const previous = readPendingSend(storageKey, matchesInput); @@ -217,7 +253,12 @@ function RetainedFilesRecovery({ action, issueId, companyId, className }: { const observed = await source.refetch(); if (observed.error) throw observed.error; const fresh = boundAction(observed.data); - if (!fresh) throw new Error("File recovery changed. Refresh the task to inspect the current recovery."); + if (!fresh) { + // The task's current recovery replaced this record. Reload the task so it + // shows that recovery, instead of asking for a refresh that changes nothing. + invalidate(); + throw new Error("This file recovery was replaced. Showing the task’s current recovery."); + } const state = asObject(fresh.evidence.workspaceRecovery); const ids = state.requestIds; const unconfirmed = Boolean(pending && Array.isArray(ids) && !ids.includes(pending.idempotencyKey)); @@ -230,14 +271,22 @@ function RetainedFilesRecovery({ action, issueId, companyId, className }: { const confirmedFailedAttempt = failedReceipt && retryable && system.state === "exhausted"; const waiting = !failedReceipt && !saved && (restoring || acknowledged || serverAcknowledged); const unconfirmed = Boolean(pending && !acknowledged && !serverAcknowledged); + const sourceLoading = sourceView.kind === "loading" || sourceView.kind === "reconnecting"; + // A verified source makes this panel the governed recovery path. Otherwise the + // task's own Retry (renewed, re-verified recovery) remains the next action. + const ownsRecoveryControl = canMutate && (sourceLoading || Boolean(current && verified && !saved)); + // `waiting` already excludes a failed accepted request, so the reason stays + // visible after a failed attempt and hides only while recovery is running. + const unverified = Boolean(current && !verified && !saved && !waiting); return
+

{saved ? "Saved files recovered. Task recovery is still being checked." : failedReceipt ? "File recovery has not finished. The saved source remains protected." : restoring ? "Recovering saved files. Work remains paused until recovery is verified." : waiting ? "File recovery is queued. Waiting for verified saved files." : "The saved files need recovery before work can continue."}

{canMutate ? <> - {current && retryable && !waiting && (!pending || unconfirmed && checkedUnconfirmed || confirmedFailedAttempt) ? : null} @@ -250,8 +299,11 @@ function RetainedFilesRecovery({ action, issueId, companyId, className }: { {notice ?? (checkError instanceof Error ? checkError.message : "File recovery could not be inspected. Check its status again.")}

: null} {current && recovery.status === "blocked" && (serverAcknowledged || recovery.retryable !== true) - ?

File recovery has not finished. The saved source remains protected; service assistance is needed.

: null} - {sourceReady && !current ?

The recorded source is unavailable or has changed. Inspect recovery before continuing.

: null} + ?

File recovery has not finished. The saved files remain protected and are not applied.

: null} + {unverified ?

+ {unverifiedSourceText(source.data?.sourceInspection)} Paperclip does not apply files it can’t verify. +

: null} + {sourceReady && !current ?

This file recovery was replaced. Check its status to show the task’s current recovery.

: null} : accessView.kind === "error" ?

Task access could not be checked. Refresh before requesting file recovery.

: accessView.kind === "loading" || accessView.kind === "reconnecting" || !owner ?

Checking file recovery access…

:

A board member with task access can request file recovery.

} diff --git a/ui/src/hooks/useTaskRetry.test.tsx b/ui/src/hooks/useTaskRetry.test.tsx index 591ea35c92b..22e5e6b93d7 100644 --- a/ui/src/hooks/useTaskRetry.test.tsx +++ b/ui/src/hooks/useTaskRetry.test.tsx @@ -39,6 +39,15 @@ describe("optimistic task Retry", () => { expect(container.textContent).not.toContain("Thinking"); expect(container.textContent).not.toContain("Taking longer"); }); + it("refreshes the inspected file recovery after an accepted Retry", async () => { + vi.mocked(issuesApi.retryTask).mockResolvedValue({ requestId: "retry", accepted: true, stale: false, + observedRunId: "failed-run", presentation: null } as Awaited>); + const key = queryKeys.issues.recoveryActions("task", "hold", "failed-run"); + client.setQueryData(key, { active: null, actions: [] }); + await render(); + await act(async () => container.querySelector("button")!.click()); + expect(client.getQueryState(key)?.isInvalidated).toBe(true); + }); it("paints Thinking before admission resolves and shares the same request across double clicks", async () => { let settle!: (value: Awaited>) => void; vi.mocked(issuesApi.retryTask).mockImplementation(() => new Promise(resolve => { settle = resolve; })); diff --git a/ui/src/hooks/useTaskRetry.ts b/ui/src/hooks/useTaskRetry.ts index 09a0f9c6761..c2235078c06 100644 --- a/ui/src/hooks/useTaskRetry.ts +++ b/ui/src/hooks/useTaskRetry.ts @@ -4,6 +4,7 @@ import type { Issue } from "@paperclipai/shared"; import { ApiError } from "../api/client"; import { issuesApi } from "../api/issues"; import { queryKeys } from "../lib/queryKeys"; +import { createUuid } from "../lib/uuid"; type RetryRequest = { issueId: string; identifier?: string | null; idempotencyKey: string; observedRunId: string | null }; @@ -24,8 +25,10 @@ export function useTaskRetry(issue: Issue | undefined) { { for (const ref of refs) queryClient.setQueryData(queryKeys.issues.detail(ref), current => current ? { ...current, interruption: result.presentation } : current); + // Retry can queue retained-file recovery; refresh its inspected state too. await Promise.all(refs.flatMap(ref => [queryKeys.issues.detail(ref), queryKeys.issues.runs(ref), - queryKeys.issues.liveRuns(ref), queryKeys.issues.activeRun(ref)]).map(queryKey => + queryKeys.issues.liveRuns(ref), queryKeys.issues.activeRun(ref), + queryKeys.issues.allRecoveryActions(ref)]).map(queryKey => queryClient.invalidateQueries({ queryKey }))); } if (isCurrent && !request.current) { setOptimistic(false); setTakingLonger(false); } @@ -73,7 +76,7 @@ export function useTaskRetry(issue: Issue | undefined) { if (!issue) return; setReconcile(null); mutation.reset(); - request.current ??= { issueId: issue.id, identifier: issue.identifier, idempotencyKey: crypto.randomUUID(), + request.current ??= { issueId: issue.id, identifier: issue.identifier, idempotencyKey: createUuid(), observedRunId: issue.interruption?.observedRunId ?? issue.executionBlocker?.runId ?? null }; setOptimistic(true); setTakingLonger(false); diff --git a/ui/src/lib/queryKeys.ts b/ui/src/lib/queryKeys.ts index f13aae268f5..757f2ae0e8f 100644 --- a/ui/src/lib/queryKeys.ts +++ b/ui/src/lib/queryKeys.ts @@ -310,6 +310,7 @@ export const queryKeys = { issues: { recoveryActions: (id: string, actionId: string | null, runId: string | null) => ["issues", id, "recovery-actions", actionId, runId] as const, + allRecoveryActions: (id: string) => ["issues", id, "recovery-actions"] as const, list: (companyId: string) => ["issues", companyId] as const, onboardingFirstTask: (companyId: string) => ["issues", companyId, "onboarding-first-task"] as const, diff --git a/ui/src/pages/IssueDetail.test.tsx b/ui/src/pages/IssueDetail.test.tsx index acc1346a59e..3167dc1bb31 100644 --- a/ui/src/pages/IssueDetail.test.tsx +++ b/ui/src/pages/IssueDetail.test.tsx @@ -3756,7 +3756,12 @@ describe("IssueDetail", () => { mockIssuesApi.update.mockReset(); }); - it.each(["verified", "unverified", "stale", "denied"] as const)("keeps native export Retry closed with %s preservation access or source", async mode => { + it.each(["verified", "unverified", "unverified-plain-http", "stale", "denied"] as const)("offers exactly one native export recovery control with %s preservation access or source", async mode => { + // Plain-HTTP LAN and tailnet previews have no crypto.randomUUID. + const originalCrypto = globalThis.crypto; + if (mode === "unverified-plain-http") + vi.stubGlobal("crypto", { getRandomValues: originalCrypto.getRandomValues.bind(originalCrypto) }); + try { const action = { id: "file-recovery", companyId: "company-1", sourceIssueId: "issue-1", kind: "active_run_watchdog", cause: "native_workspace_sync_out_retry_exhausted", @@ -3765,9 +3770,11 @@ describe("IssueDetail", () => { stage: "restoring_files", requestIds: [], retryable: true, }, systemRecovery: { state: "exhausted", preservationOperationIds: [] } }, } as unknown as NonNullable; - const issue = createIssue({ status: "blocked", assigneeAgentId: "agent-1", activeRecoveryAction: action, - interruption: { intentId: "held-input", responsibility: "system", state: "failed", nextRetryAt: null, - canRetry: true, userAction: null, sourceRunIds: ["stopped-run"], observedRunId: "stopped-run" }, + const interruption: NonNullable = { intentId: "held-input", responsibility: "system", state: "failed", nextRetryAt: null, + canRetry: true, userAction: null, sourceRunIds: ["stopped-run"], observedRunId: "stopped-run", + retryWait: { reason: "saved_work_recovery_failed", + message: "Saving the agent's files failed after automatic retries. Retry to try again; the agent will not repeat its work." } }; + const issue = createIssue({ status: "blocked", assigneeAgentId: "agent-1", activeRecoveryAction: action, interruption, executionBlocker: { runId: "stopped-run", cause: "native_workspace_sync_out_retry_exhausted", canRetry: false, workspaceRepairRequired: true, workspaceRecovery: { status: "blocked", stage: "restoring_files", canRetry: true } } as Issue["executionBlocker"], }); @@ -3779,18 +3786,35 @@ describe("IssueDetail", () => { }); mockIssuesApi.recoveryActions.mockResolvedValue({ active: action, actions: [action], sourceInspection: { actionId: mode === "stale" ? "other-action" : action.id, runId: "stopped-run", - status: mode === "unverified" ? "blocked" : "verified", scope: "recorded_source_identity", + status: mode.startsWith("unverified") ? "blocked" : "verified", scope: "recorded_source_identity", } }); mockIssuesApi.recoverWorkspace.mockResolvedValue({ actionId: action.id, status: "queued" }); + mockIssuesApi.retryTask.mockResolvedValue({ requestId: "export-retry", accepted: true, stale: false, presentation: interruption }); await act(async () => renderIssueView()); - if (mode !== "verified") { - await waitForAssertion(() => expect(container.textContent).toContain(mode === "denied" - ? "A board member with task access" : "The recorded source is unavailable or has changed")); - expect([...container.querySelectorAll("button")].some(button => ["Recover saved files", "Retry"].includes(button.textContent ?? ""))).toBe(false); + const labels = () => [...container.querySelectorAll("button")].map(button => button.textContent ?? ""); + if (mode === "denied") { + await waitForAssertion(() => expect(container.textContent).toContain("A board member with task access")); + expect(labels().some(label => ["Recover saved files", "Retry"].includes(label))).toBe(false); expect(mockIssuesApi.recoverWorkspace).not.toHaveBeenCalled(); expect(mockIssuesApi.retryTask).not.toHaveBeenCalled(); return; } + if (mode !== "verified") { + // The notice says "Retry to try again"; an unverifiable source must not + // leave it without a Retry. The server re-verifies the source. + await waitForAssertion(() => expect(labels()).toContain("Retry")); + expect(labels()).not.toContain("Recover saved files"); + expect(container.textContent).not.toContain("Refresh the task"); + const retry = [...container.querySelectorAll("button")].find(button => button.textContent === "Retry")!; + await act(async () => retry.click()); + await waitForAssertion(() => expect(mockIssuesApi.retryTask).toHaveBeenCalledExactlyOnceWith(issue.id, { + idempotencyKey: expect.stringMatching(/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/), + observedRunId: "stopped-run", + })); + expect(mockIssuesApi.recoverWorkspace).not.toHaveBeenCalled(); + expect(mockIssuesApi.resolveRecoveryAction).not.toHaveBeenCalled(); + return; + } await waitForAssertion(() => expect([...container.querySelectorAll("button")] .some(button => button.textContent === "Recover saved files")).toBe(true)); expect([...container.querySelectorAll("button")].some(button => button.textContent === "Retry")).toBe(false); @@ -3803,6 +3827,7 @@ describe("IssueDetail", () => { expect(mockIssuesApi.update).not.toHaveBeenCalled(); expect(issue.executionBlocker?.workspaceRepairRequired).toBe(true); expect(container.textContent).not.toContain("Saved files recovered"); + } finally { vi.unstubAllGlobals(); } }); it("keeps the existing Retry path for exhausted legacy workspace restoration", async () => { diff --git a/ui/src/pages/IssueDetail.tsx b/ui/src/pages/IssueDetail.tsx index 6e43bc8ba5b..2c0de49ef87 100644 --- a/ui/src/pages/IssueDetail.tsx +++ b/ui/src/pages/IssueDetail.tsx @@ -3757,6 +3757,9 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS selectedCompanyId, boardAccess, ); + // Retained-file recovery reports whether it offers its own control. The task's + // Retry stays visible whenever it does not, so a retryable hold never dead-ends. + const [retainedRecoveryOwnsRetry, setRetainedRecoveryOwnsRetry] = useState(true); // The break-glass override reconcile is `runtime:manage`-gated server-side, not gated on the // recovery-resolution permission — so hide its affordance behind the matching client check. const canManageBoardRuntime = canBoardManageRuntime( @@ -8181,13 +8184,14 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS className={shellSectionClass} issueId={issue.id} companyId={issue.companyId} showStatus={!issue.interruption && !issue.executionBlocker?.workspaceRecovery} + onRecoveryControlChange={setRetainedRecoveryOwnsRetry} action={issue.activeRecoveryAction ?? null} /> )} {resolvedDetailTab === "chat" && ( )}