fix(workspaces): skip unverifiable recovery sources instead of failing Retry (#20666)

<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote run starts, workspace preparation attaches the
verified saved work of earlier runs of the same task.
> - Preparation failed with `remote_workspace_recovery_receipt_invalid`
when any earlier run had a recovery receipt that it could not verify.
> - A refusal cannot change the saved state, so every Retry failed in
the same way, and the task stayed blocked. Recoveries must fall back
automatically.
> - This pull request skips an unverifiable source. It attaches nothing
unverified and starts from the workspace's newest verified save.
> - The benefit is that Retry and continuations start again, while
strict receipt checks still decide what is attached, and no saved work
is changed.

## Linked Issues or Issue Description

Refs #20631

**What happened?**
On staging, after the deploy that included #20631, a user pressed Retry
on a task whose previous run failed with
`native_controller_recovery_missing_evidence:runner_not_settled`. The
new run failed in setup in under a minute with `setup_failed` /
`remote_workspace_recovery_receipt_invalid` (`executionRecovery:
bootstrap`, `providerWorkStarted: false`). The task stayed blocked.
Another Retry would fail the same way. #20631 accepted one shape of this
refusal (a source that saved only the snapshot its own restore loaded).
This task had a different shape.

**Expected behavior**
A Retry starts. A source whose receipt cannot be verified is not
attached, and its saved state stays intact for workspace recovery.

**Steps to reproduce**
1. Let a remote native run fail before provider work and leave a
recovery receipt or restore hold that does not verify (for example a
receipt on an unsaved reference, a receipt from another epoch, or a
saved reference without a receipt).
2. Press Retry on the task.
3. See the new run fail in setup with
`remote_workspace_recovery_receipt_invalid`.

**Paperclip version or commit**
Staging deploy that includes `e3bc4ca798`. The gap remains on `master`
at `a3b34a9d88`.

## What Changed

- `attachPreservedRemoteWorkspaceSnapshots` no longer throws for an
unverifiable source. Each of the five refusal shapes now skips the
source and logs `remote_workspace_recovery_receipt_invalid` with a
reason: `receipt_without_reference`, `reference_unreadable`,
`receipt_on_unsaved_reference`, `receipt_unverified`, and
`receipt_snapshot_missing` (was
`remote_workspace_recovery_snapshot_missing`).
- `retainRemoteRecoveryAttachments` no longer throws for a retained
recovered copy that it cannot verify. It does not publish that copy
again. The files stay in the workspace.
- The writer check (`remote_workspace_recovery_writer_changed`), the
workspace access check, and snapshot read and digest failures still
refuse.
- `doc/execution-semantics.md` describes the fallback rule.

## Verification

- Red on unfixed code:
`remote-workspace-recovery-receipt-fallback.test.ts` (moved out of
`workspace-recovery-preparation.test.ts` to keep that file under 1,000
lines) "starts a Retry from the newest verified save when a predecessor
has %s" (6 shapes: `receipt_without_reference`,
`unreadable_reference_with_hold`, `receipt_on_prepared_reference`,
`superseded_snapshot_with_hold`, `receipt_from_other_epoch`,
`receipt_for_other_run`). All 6 failed with
`remote_workspace_recovery_receipt_invalid`. The source is a native run
that failed with `runner_not_settled`, and the writer is a Retry
(`retryOfRunId`, `taskRetryRequestId`) that follows the real ancestry
query.
- Green: the same 6 tests pass. They also assert that nothing is
attached, and that the source run row, its workspace operations, and its
snapshot row are unchanged.
- Updated safety tests: the canonical-attachment fault matrix now
expects no attachment for receipt faults, and still expects a refusal
for a missing or corrupt archive and for a wrong company or task. The
retained-history test expects an unverifiable retained copy to be left
out of the published list, while verified copies stay published.
- `npx vitest run
src/services/remote-workspace-recovery-receipt-fallback.test.ts
src/services/workspace-recovery-preparation.test.ts`: 91 passed.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` returns exit 2, because a global doc input changed
(`doc/execution-semantics.md`). Required CI supplies the broad coverage.
- Not verified live on Daytona. I could not read the staging rows for
this task: the browser had no free tab. The tests cover every refusal
shape of this function, so the exact staging shape is one of them.

## Risks

- Previous-release state: the rule reads the same rows as before (run
reference, result hold, workspace operations, recovery actions,
snapshots). Rows that earlier releases saved carry forward. No
migration. A blocked task recovers on its next Retry or wake.
- A Retry can now start without files from an unverifiable source. Those
files are not lost: the source's snapshot, restore hold, receipts and
recovery action stay unchanged, and workspace recovery still owns them.
The skip is logged with a named reason.
- A retained copy that fails verification is dropped from the published
list. Its directory stays in the workspace and is no longer retired
automatically.
- A missing or corrupt snapshot archive still refuses. A permanently
lost archive therefore still blocks attach. This PR does not change
that.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/execution-semantics.md

@@ -3406,7 +3406,23 @@ have no recovery receipt. When its saved reference names the exact snapshot
 that its own succeeded `remote_workspace_restore` loaded, at the same epoch, it
 left no unsaved files. A later continuation into another workspace skips that
 source instead of failing with `remote_workspace_recovery_receipt_invalid`. A
-source that saved any other snapshot still needs the canonical receipt.
+source that saved any other snapshot still needs the canonical receipt to be
+attached.
+
+A predecessor whose recovery receipt cannot be verified never stops a
+continuation or a Retry. Examples are a receipt on a run reference that was
+never saved, a receipt from another epoch or for another run, a saved
+reference without a receipt, a reference that the current release cannot read,
+and a receipt whose snapshot is gone. Preparation then attaches nothing from
+that source. The run starts from its workspace's newest verified save, as a run
+without a predecessor does. Preparation logs the skip with the code
+`remote_workspace_recovery_receipt_invalid` and a reason. The source keeps its
+saved snapshot, restore hold, receipts and recovery action, so workspace
+recovery can still settle it. A recovered copy that is already in the workspace
+but cannot be verified is not published again. Its files stay where they are.
+Before this rule, the refusal could not change the saved state, so every Retry
+failed in the same way. Rows that earlier releases saved follow the same rule
+with no migration. A foreign writer and a failed snapshot read still refuse.
 
 A queued turn for the same task that never started is not a workspace writer for
 automatic recovery. The hold gates its dispatch until the recovery receipt is

server/src/services/remote-workspace-recovery-attachments.ts

@@ -16,6 +16,21 @@ import { assertExecutionWorkspaceRecordAccess } from "./task-workspace-access.js
 import { getWorkspaceSnapshot, loadWorkspaceSnapshotArchive } from "./remote-workspace-snapshot-storage.js";
 import { restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js";
 import { getStorageService, type StorageService } from "../storage/index.js";
+import { logger } from "../middleware/logger.js";
+
+/** A source whose recovery receipt cannot be verified is skipped, never fatal.
+ * The run then starts from its workspace's newest verified save, as a run with
+ * no predecessor does. Nothing unverified is imported, and the source keeps its
+ * saved snapshot, restore hold, receipts and recovery action, so workspace
+ * recovery can still settle it. A refusal here could not change that saved
+ * state, so every Retry failed the same way, including on rows that an earlier
+ * release saved. */
+function skipUnverifiedRecoverySource(input: { companyId: string; issueId?: string | null; runId: string },
+  sourceRunId: string, reason: string) {
+  logger.warn({ companyId: input.companyId, issueId: input.issueId, runId: input.runId, sourceRunId,
+    code: "remote_workspace_recovery_receipt_invalid", reason },
+  "Saved work of a previous run was not attached because its recovery receipt could not be verified");
+}
 
 /** Import verified canonical recovery as a sidecar. The selected workspace and
  * its own files stay intact; this does not grant execution or replay authority. */
@@ -43,23 +58,26 @@ export async function attachPreservedRemoteWorkspaceSnapshots(input: {
   const result: Array<{ operationId: string; path: string; sourceRunId: string }> = [];
   for (const source of sources) {
     if (source.runnerProfileJson?.remoteWorkspace == null) {
-      if (canonicalSources.has(source.id)) throw new Error("remote_workspace_recovery_receipt_invalid");
+      if (canonicalSources.has(source.id)) skipUnverifiedRecoverySource(input, source.id, "receipt_without_reference");
       continue;
     }
     const parsed = remoteWorkspaceRunReferenceSchema.safeParse(source.runnerProfileJson.remoteWorkspace);
     if (!parsed.success) {
-      if (hasRequiredWorkspaceRecovery(source.resultJson)) throw new Error("remote_workspace_recovery_receipt_invalid");
+      if (hasRequiredWorkspaceRecovery(source.resultJson)) skipUnverifiedRecoverySource(input, source.id, "reference_unreadable");
       continue;
     }
     const remote = parsed.data;
-    if (canonicalSources.has(source.id) && (remote.state !== "saved" || !remote.snapshotId))
+    if (canonicalSources.has(source.id) && (remote.state !== "saved" || !remote.snapshotId)) {
-      throw new Error("remote_workspace_recovery_receipt_invalid");
+      skipUnverifiedRecoverySource(input, source.id, "receipt_on_unsaved_reference");
+      continue;
+    }
     if (remote.workspaceId === input.workspaceId || remote.state !== "saved" || !remote.snapshotId) continue;
     const proof = await immutableRemoteWorkspaceRecoveryAttachment(input.db, source);
     if (!proof) {
-      if (hasRequiredWorkspaceRecovery(source.resultJson) && !await savedOnlyItsRestoredSnapshot(input.db, source, remote)) {
+      // A source that saved only the snapshot its own restore loaded left no
-        throw new Error("remote_workspace_recovery_receipt_invalid");
+      // unsaved files; any other unverified source is reported and skipped.
-      }
+      if (hasRequiredWorkspaceRecovery(source.resultJson) && !await savedOnlyItsRestoredSnapshot(input.db, source, remote))
+        skipUnverifiedRecoverySource(input, source.id, "receipt_unverified");
       continue;
     }
     const operationId = proof.operationId;
@@ -70,8 +88,10 @@ export async function attachPreservedRemoteWorkspaceSnapshots(input: {
     input.companyId, remote.workspaceId);
     const saved = await getWorkspaceSnapshot(input.db, { companyId: input.companyId,
       workspaceId: remote.workspaceId, snapshotId: remote.snapshotId });
-    if (!saved || saved.sha256 !== proof.sha256 || saved.generation !== proof.generation)
+    if (!saved || saved.sha256 !== proof.sha256 || saved.generation !== proof.generation) {
-      throw new Error("remote_workspace_recovery_snapshot_missing");
+      skipUnverifiedRecoverySource(input, source.id, "receipt_snapshot_missing");
+      continue;
+    }
     const provenance = JSON.stringify({ schema: "paperclip.remote-workspace-recovery-attachment.v1",
       operationId, sourceRunId: source.id, workspaceId: remote.workspaceId,
       snapshotId: remote.snapshotId, sha256: saved.sha256 });
@@ -253,7 +273,13 @@ export async function retainRemoteRecoveryAttachments(input: {
         provenance = await retainedRecoveryAttachmentProvenance({ db: input.db, companyId: input.workspace.companyId,
           workspace: input.workspace, issueIds: [issueId, input.workspace.sourceIssueId].filter((id): id is string => typeof id === "string"),
           sourceRunId: item.sourceRunId, operationId: item.operationId, verifySelectedSnapshot });
-        if (!provenance) throw new Error("remote_workspace_recovery_receipt_invalid");
+        // An unverifiable record is not published again. Its files stay where
+        // they are in the workspace; only the recovered-copy claim is dropped.
+        if (!provenance) {
+          skipUnverifiedRecoverySource({ companyId: input.workspace.companyId,
+            issueId: input.workspace.sourceIssueId, runId: input.runId }, item.sourceRunId, "retained_copy_unverified");
+          continue;
+        }
       }
       await input.assertOwned?.();
       const location = { target: input.target, remoteCwd: input.remoteRoot, operationId: item.operationId, provenance };

server/src/services/remote-workspace-recovery-receipt-fallback.test.ts

@@ -0,0 +1,127 @@
+import { randomUUID } from "node:crypto";
+import { mkdir, readdir } from "node:fs/promises";
+import path from "node:path";
+import { eq } from "drizzle-orm";
+import { describe, expect, it, vi } from "vitest";
+import { executionWorkspaceSnapshots, executionWorkspaces, heartbeatRuns, issueRecoveryActions, workspaceOperations } from "@paperclipai/db";
+import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
+import { attachPreservedRemoteWorkspaceSnapshots } from "./remote-workspace-recovery-attachments.js";
+import { useWorkspaceRecoveryFixture, db, failedRemotePreparation, root, run } from "./__tests__/workspace-recovery-fixture.js";
+
+const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
+vi.mock("./environment-execution-target.js", async importOriginal => {
+  const actual = await importOriginal<typeof import("./environment-execution-target.js")>();
+  return { ...actual, resolveEnvironmentExecutionTarget: async (input: Parameters<typeof actual.resolveEnvironmentExecutionTarget>[0]) =>
+    (input.leaseId ? targets.get(input.leaseId) : undefined) ?? actual.resolveEnvironmentExecutionTarget(input) };
+});
+
+// A predecessor whose recovery receipt cannot be verified is skipped, never
+// fatal: preparation attaches nothing unverified and leaves its saved state.
+describe("remote workspace recovery receipt fallback", () => {
+  useWorkspaceRecoveryFixture(targets);
+
+  // Nov 10 staging: a continuation into a fresh workspace failed
+  // with remote_workspace_recovery_receipt_invalid. Its predecessor restored the
+  // committed save, failed before provider work, and kept a restore hold with
+  // no recovery receipt. It saved the very snapshot it restored, so it left no
+  // unsaved files to attach. A source that saved anything else is not attached
+  // either: without a receipt it is skipped, and its saved state stays as it was.
+  it.each(["restored_unchanged", "saved_other_snapshot", "restore_not_succeeded", "restored_other_epoch"] as const)(
+    "decides a receipt-less restore hold of a source that %s", async variant => {
+      const f = await failedRemotePreparation();
+      const template = await run(f.runId);
+      const restoredSnapshotId = randomUUID(), sourceId = randomUUID(), leaseId = randomUUID();
+      await db.insert(executionWorkspaceSnapshots).values({ id: restoredSnapshotId, companyId: f.companyId,
+        executionWorkspaceId: f.workspaceId, sourceOperationId: randomUUID(), sourceRunId: randomUUID(),
+        generation: 4, epoch: 3, status: "committed", sha256: "a".repeat(64) });
+      await db.insert(heartbeatRuns).values({ id: sourceId, companyId: f.companyId, agentId: template.agentId,
+        nativeIssueId: f.issueId, issueId: f.issueId, status: "failed", runtimeMode: "native", finishedAt: new Date(),
+        contextSnapshot: { issueId: f.issueId },
+        errorCode: "native_continuation_requires_reconciliation",
+        error: "native_controller_recovery_missing_evidence:retained_sandbox_binding",
+        runnerProfileJson: { remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", epoch: 6, state: "saved",
+          leaseId, providerLeaseId: randomUUID(), remoteCwd: `/home/daytona/paperclip-workspace/.paperclip-workspaces/${f.workspaceId}-6-${leaseId}`,
+          workspaceId: f.workspaceId, snapshotId: variant === "saved_other_snapshot" ? randomUUID() : restoredSnapshotId } },
+        resultJson: { stopReason: "adapter_failed", executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
+          workspaceRestoreFailure: "restore_failed",
+          workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [leaseId] } } });
+      await db.insert(workspaceOperations).values({ companyId: f.companyId, issueId: f.issueId, heartbeatRunId: sourceId,
+        executionWorkspaceId: f.workspaceId, phase: "remote_workspace_restore",
+        status: variant === "restore_not_succeeded" ? "failed" : "succeeded",
+        metadata: { epoch: variant === "restored_other_epoch" ? 5 : 6, snapshotId: restoredSnapshotId } });
+      const [writer] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: template.agentId,
+        issueId: f.issueId, status: "running", responsibleUserId: "local-board", contextSnapshot: { issueId: f.issueId } }).returning();
+      const destination = path.join(root, writer.id); await mkdir(destination);
+      const attach = attachPreservedRemoteWorkspaceSnapshots({ db, companyId: f.companyId, issueId: f.issueId,
+        runId: writer.id, workspaceId: randomUUID(), sourceRunIds: [sourceId], cwd: destination });
+      await expect(attach).resolves.toEqual([]);
+      expect(await readdir(destination)).toEqual([]);
+      expect((await run(sourceId)).resultJson).toMatchObject({ workspaceRestoreRecovery: { leaseIds: [leaseId] } });
+    }, 30_000);
+
+  // Nov 10 staging, after a deploy: Retry of a task whose previous run failed
+  // with native_controller_recovery_missing_evidence:runner_not_settled failed in
+  // setup with remote_workspace_recovery_receipt_invalid, and every later Retry
+  // would fail the same way. A predecessor whose receipt cannot be verified must
+  // not dead-end the task: the Retry starts from the workspace's newest verified
+  // save, attaches nothing unverified, and leaves the source's saved snapshot,
+  // restore hold and receipts exactly as they were.
+  it.each([
+    "receipt_without_reference", "unreadable_reference_with_hold", "receipt_on_prepared_reference",
+    "superseded_snapshot_with_hold", "receipt_from_other_epoch", "receipt_for_other_run",
+  ] as const)("starts a Retry from the newest verified save when a predecessor has %s", async variant => {
+    const f = await failedRemotePreparation();
+    const template = await run(f.runId);
+    const sourceId = randomUUID(), leaseId = randomUUID(), providerLeaseId = randomUUID(), savedSnapshotId = randomUUID();
+    const sourceWorkspaceId = variant === "receipt_on_prepared_reference" ? f.workspaceId : randomUUID();
+    if (sourceWorkspaceId !== f.workspaceId) await db.insert(executionWorkspaces).values({ id: sourceWorkspaceId,
+      companyId: f.companyId, sourceIssueId: f.issueId, name: "Previous run workspace", providerType: "cloud_sandbox",
+      mode: "isolated_workspace", strategyType: "project_primary", cwd: `/workspaces/${sourceWorkspaceId}` });
+    await db.insert(executionWorkspaceSnapshots).values({ id: savedSnapshotId, companyId: f.companyId,
+      executionWorkspaceId: sourceWorkspaceId, sourceOperationId: randomUUID(), sourceRunId: sourceId,
+      generation: 7, epoch: 6, status: variant === "superseded_snapshot_with_hold" ? "superseded" : "committed",
+      sha256: "b".repeat(64) });
+    const reference = { schema: "paperclip.remote-workspace-run.v1", epoch: 6,
+      state: variant === "receipt_on_prepared_reference" ? "prepared" : "saved", leaseId, providerLeaseId,
+      remoteCwd: `/home/daytona/paperclip-workspace/.paperclip-workspaces/${sourceWorkspaceId}-6-${leaseId}`,
+      workspaceId: sourceWorkspaceId, snapshotId: variant === "receipt_on_prepared_reference" ? null : savedSnapshotId,
+      ...(variant === "unreadable_reference_with_hold" ? { savedBy: "previous-release" } : {}) };
+    const hold = { workspaceRestoreFailure: "restore_failed",
+      workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [leaseId] } };
+    await db.insert(heartbeatRuns).values({ id: sourceId, companyId: f.companyId, agentId: template.agentId,
+      nativeIssueId: f.issueId, issueId: f.issueId, status: "failed", runtimeMode: "native",
+      finishedAt: new Date(Date.now() - 60_000), createdAt: new Date(Date.now() - 120_000), contextSnapshot: { issueId: f.issueId },
+      errorCode: "native_continuation_requires_reconciliation",
+      error: "native_controller_recovery_missing_evidence:runner_not_settled",
+      runnerProfileJson: variant === "receipt_without_reference" ? {} : { remoteWorkspace: reference },
+      resultJson: { stopReason: "adapter_failed", ...hold } });
+    const [recoveryAction] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
+      kind: "active_run_watchdog", status: "resolved", ownerType: "board", cause: "native_continuation_requires_reconciliation",
+      fingerprint: `native-workspace:${sourceId}`, nextAction: "Recover saved work",
+      evidence: { runId: sourceId, ...hold, workspaceRecovery: { schema: "paperclip.workspace-recovery.v1",
+        runId: variant === "receipt_for_other_run" ? randomUUID() : sourceId, status: "recovered", stage: "saving_receipts",
+        attempt: 1, requestIds: [] } } }).returning();
+    if (variant !== "unreadable_reference_with_hold" && variant !== "superseded_snapshot_with_hold")
+      await db.insert(workspaceOperations).values({ companyId: f.companyId, issueId: f.issueId, heartbeatRunId: sourceId,
+        executionWorkspaceId: sourceWorkspaceId, phase: "workspace_recovery", status: "succeeded", finishedAt: new Date(),
+        metadata: { recoveryActionId: recoveryAction.id, leaseId, providerLeaseId, snapshotId: savedSnapshotId,
+          generation: 7, sha256: "b".repeat(64), storage: "remote_workspace",
+          epoch: variant === "receipt_from_other_epoch" ? 5 : 6 } });
+    const [writer] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: template.agentId,
+      nativeIssueId: f.issueId, issueId: f.issueId, status: "running", runtimeMode: "native", retryOfRunId: sourceId,
+      responsibleUserId: "local-board", contextSnapshot: { issueId: f.issueId, taskRetryRequestId: randomUUID() } }).returning();
+    const destination = path.join(root, writer.id); await mkdir(destination);
+    const before = { source: await run(sourceId), operations: await db.select().from(workspaceOperations)
+      .where(eq(workspaceOperations.heartbeatRunId, sourceId)), snapshots: await db.select().from(executionWorkspaceSnapshots)
+      .where(eq(executionWorkspaceSnapshots.id, savedSnapshotId)) };
+    const attach = attachPreservedRemoteWorkspaceSnapshots({ db, companyId: f.companyId, issueId: f.issueId,
+      runId: writer.id, workspaceId: f.workspaceId, sourceRunIds: [writer.id], cwd: destination });
+    await expect(attach).resolves.toEqual([]);
+    expect(await readdir(destination)).toEqual([]);
+    // The fallback attaches nothing unverified and never touches the source's saved state.
+    expect(await run(sourceId)).toEqual(before.source);
+    expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, sourceId))).toEqual(before.operations);
+    expect(await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, savedSnapshotId)))
+      .toEqual(before.snapshots);
+  }, 30_000);
+});

server/src/services/workspace-recovery-preparation.test.ts

@@ -46,6 +46,9 @@ vi.mock("./environment-execution-target.js", async importOriginal => {
     (input.leaseId ? targets.get(input.leaseId) : undefined) ?? actual.resolveEnvironmentExecutionTarget(input) };
 });
 
+const publishedSources = (workspace: { metadata?: Record<string, unknown> | null }) =>
+  ((workspace.metadata?.remoteWorkspaceRecoveredFiles ?? []) as Array<{ sourceRunId: string }>).map(item => item.sourceRunId);
+
 describe("automatic retained workspace recovery", () => {
   useWorkspaceRecoveryFixture(targets);
 
@@ -339,9 +342,11 @@ describe("automatic retained workspace recovery", () => {
               await db.update(heartbeatRuns).set({ runnerProfileJson: { legacyWorkspaceRecoveryMigration:
                 { ...marker, workspaceId: fault === "source_marker_workspace" ? randomUUID() : placeholder.id } } })
                 .where(eq(heartbeatRuns.id, related.id));
-              await expect(prepareRemoteWorkspace({ db, storage, workspace: withoutProof, runId: childRun.id,
+              // An unverifiable retained copy is not published again; the run still starts.
-                lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: cold.remoteRoot } }))
+              const unverified = await prepareRemoteWorkspace({ db, storage, workspace: withoutProof, runId: childRun.id,
-                .rejects.toThrow("remote_workspace_recovery_receipt_invalid");
+                lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: cold.remoteRoot } });
+              expect(publishedSources(unverified.workspace)).not.toContain(related.id);
+              expect(publishedSources(unverified.workspace)).toContain(source.id);
             }
             await db.update(workspaceOperations).set({ status: "succeeded", metadata: {
               legacyRecoverySources: [{ runId: related.id, issueId: earlierChild.id }] } }).where(eq(workspaceOperations.id, migration.id));
@@ -349,9 +354,9 @@ describe("automatic retained workspace recovery", () => {
               { ...marker, workspaceId: placeholder.id } } }).where(eq(heartbeatRuns.id, related.id));
             await db.update(workspaceOperations).set({ status: "failed" }).where(eq(workspaceOperations.id,
               (olderPaths.find(item => item.sourceRunId === source.id))!.operationId));
-            await expect(prepareRemoteWorkspace({ db, storage, workspace: withoutProof, runId: childRun.id,
+            const withoutReceipt = await prepareRemoteWorkspace({ db, storage, workspace: withoutProof, runId: childRun.id,
-              lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: cold.remoteRoot } }))
+              lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: cold.remoteRoot } });
-              .rejects.toThrow("remote_workspace_recovery_receipt_invalid");
+            expect(publishedSources(withoutReceipt.workspace)).not.toContain(source.id);
             expect(await readFile(path.join(inherited.remoteRoot, newest!.path, "files", "ticks.log"), "utf8")).toBe(editedLog);
             await db.update(workspaceOperations).set({ status: "succeeded" }).where(eq(workspaceOperations.id,
               (olderPaths.find(item => item.sourceRunId === source.id))!.operationId));
@@ -379,14 +384,14 @@ describe("automatic retained workspace recovery", () => {
               remoteWorkspaceRecoveredFiles: olderPaths.map(({ operationId, sourceRunId, path }) => ({ operationId, sourceRunId, path })) } };
             const authority = readRemoteWorkspaceAuthority(legacySnapshot.metadata)!;
             await db.update(executionWorkspaceSnapshots).set({ status: "uploaded" }).where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!));
-            await expect(prepareRemoteWorkspace({ db, storage, workspace: legacySnapshot, runId: childRun.id,
+            const uncommitted = await prepareRemoteWorkspace({ db, storage, workspace: legacySnapshot, runId: childRun.id,
-              lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: inherited.remoteRoot } }))
+              lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: inherited.remoteRoot } });
-              .rejects.toThrow("remote_workspace_recovery_receipt_invalid");
+            expect(publishedSources(uncommitted.workspace)).not.toContain(related.id);
             await db.update(executionWorkspaceSnapshots).set({ status: "committed" }).where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!));
-            await expect(prepareRemoteWorkspace({ db, storage, workspace: { ...legacySnapshot, metadata: { ...legacySnapshot.metadata,
+            const otherSnapshot = await prepareRemoteWorkspace({ db, storage, workspace: { ...legacySnapshot, metadata: { ...legacySnapshot.metadata,
               remoteWorkspace: { ...authority, snapshotId: sourceReference.snapshotId } } }, runId: childRun.id,
-              lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: inherited.remoteRoot } }))
+              lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: inherited.remoteRoot } });
-              .rejects.toThrow("remote_workspace_recovery_receipt_invalid");
+            expect(publishedSources(otherSnapshot.workspace)).not.toContain(related.id);
             const unreadable = { ...storage, getObject: vi.fn().mockRejectedValue(new Error("fixture: selected snapshot unavailable")) };
             await expect(prepareRemoteWorkspace({ db, storage: unreadable, workspace: legacySnapshot, runId: childRun.id,
               lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: inherited.remoteRoot } }))
@@ -504,7 +509,7 @@ describe("automatic retained workspace recovery", () => {
   }, 30_000);
 
   it.each(["missing_receipt", "missing_reference", "prepared_reference", "missing_snapshot_id", "uncommitted", "foreign_snapshot", "wrong_lease", "missing_object", "corrupt_object", "wrong_company", "wrong_issue"])(
-    "refuses canonical recovery attachments with %s evidence", async fault => {
+    "never attaches canonical recovery attachments with %s evidence", async fault => {
     const f = await failedRemotePreparation();
     await recoverRetainedWorkspace(f.input);
     const source = await run(f.runId), remote = source.runnerProfileJson!.remoteWorkspace as { snapshotId: string };
@@ -532,53 +537,19 @@ describe("automatic retained workspace recovery", () => {
       return { ...object, stream: Readable.from([Buffer.from("corrupt")]) };
     }) : undefined;
     try {
-      await expect(attachPreservedRemoteWorkspaceSnapshots({ db,
+      const attach = attachPreservedRemoteWorkspaceSnapshots({ db,
         companyId: fault === "wrong_company" ? randomUUID() : f.companyId,
         issueId: fault === "wrong_issue" ? randomUUID() : f.issueId,
-        runId: writer.id, workspaceId: randomUUID(), sourceRunIds: [source.id], cwd: destination })).rejects.toThrow();
+        runId: writer.id, workspaceId: randomUUID(), sourceRunIds: [source.id], cwd: destination });
+      // An unverifiable receipt falls back to the newest verified save and
+      // attaches nothing. A failed read and a foreign writer still refuse.
+      if (["missing_object", "corrupt_object", "wrong_company", "wrong_issue"].includes(fault)) await expect(attach).rejects.toThrow();
+      else await expect(attach).resolves.toEqual([]);
       expect(await readdir(destination)).toEqual(fault === "missing_object" || fault === "corrupt_object" ? [".paperclip-recovered"] : []);
       expect(await readFile(path.join(f.remote, "saved.txt"), "utf8")).toBe("finished work\n");
     } finally { failingRead?.mockRestore(); }
   }, 30_000);
 
-  // Nov 10 staging (PAP-3881): a continuation into a fresh workspace failed
-  // with remote_workspace_recovery_receipt_invalid. Its predecessor restored the
-  // committed save, failed before provider work, and kept a restore hold with
-  // no recovery receipt. It saved the very snapshot it restored, so it left no
-  // unsaved files to attach. A source that saved anything else stays strict.
-  it.each(["restored_unchanged", "saved_other_snapshot", "restore_not_succeeded", "restored_other_epoch"] as const)(
-    "decides a receipt-less restore hold of a source that %s", async variant => {
-      const f = await failedRemotePreparation();
-      const template = await run(f.runId);
-      const restoredSnapshotId = randomUUID(), sourceId = randomUUID(), leaseId = randomUUID();
-      await db.insert(executionWorkspaceSnapshots).values({ id: restoredSnapshotId, companyId: f.companyId,
-        executionWorkspaceId: f.workspaceId, sourceOperationId: randomUUID(), sourceRunId: randomUUID(),
-        generation: 4, epoch: 3, status: "committed", sha256: "a".repeat(64) });
-      await db.insert(heartbeatRuns).values({ id: sourceId, companyId: f.companyId, agentId: template.agentId,
-        nativeIssueId: f.issueId, issueId: f.issueId, status: "failed", runtimeMode: "native", finishedAt: new Date(),
-        contextSnapshot: { issueId: f.issueId },
-        errorCode: "native_continuation_requires_reconciliation",
-        error: "native_controller_recovery_missing_evidence:retained_sandbox_binding",
-        runnerProfileJson: { remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", epoch: 6, state: "saved",
-          leaseId, providerLeaseId: randomUUID(), remoteCwd: `/home/daytona/paperclip-workspace/.paperclip-workspaces/${f.workspaceId}-6-${leaseId}`,
-          workspaceId: f.workspaceId, snapshotId: variant === "saved_other_snapshot" ? randomUUID() : restoredSnapshotId } },
-        resultJson: { stopReason: "adapter_failed", executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
-          workspaceRestoreFailure: "restore_failed",
-          workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [leaseId] } } });
-      await db.insert(workspaceOperations).values({ companyId: f.companyId, issueId: f.issueId, heartbeatRunId: sourceId,
-        executionWorkspaceId: f.workspaceId, phase: "remote_workspace_restore",
-        status: variant === "restore_not_succeeded" ? "failed" : "succeeded",
-        metadata: { epoch: variant === "restored_other_epoch" ? 5 : 6, snapshotId: restoredSnapshotId } });
-      const [writer] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: template.agentId,
-        issueId: f.issueId, status: "running", responsibleUserId: "local-board", contextSnapshot: { issueId: f.issueId } }).returning();
-      const destination = path.join(root, writer.id); await mkdir(destination);
-      const attach = attachPreservedRemoteWorkspaceSnapshots({ db, companyId: f.companyId, issueId: f.issueId,
-        runId: writer.id, workspaceId: randomUUID(), sourceRunIds: [sourceId], cwd: destination });
-      if (variant === "restored_unchanged") await expect(attach).resolves.toEqual([]);
-      else await expect(attach).rejects.toThrow("remote_workspace_recovery_receipt_invalid");
-      expect(await readdir(destination)).toEqual([]);
-    }, 30_000);
-
   it.each(["unchanged", "advanced"])("deduplicates immutable canonical sidecars when the source workspace is %s", async state => {
     const f = await failedRemotePreparation(); await recoverRetainedWorkspace(f.input);
     const source = await run(f.runId), destination = path.join(root, randomUUID()); await mkdir(destination);