fix(recovery): continue from the last save when a retained source cannot be verified (#19867)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a native run cannot save its files, a retained-file recovery
hold blocks the task until the worker saves them from the retained
sandbox.
> - The worker verifies the recorded source strictly. This includes the
reusable-lease chain of the sandbox.
> - A reusable sandbox chain can cross from a parent task into a child
task. The check then refuses the source with
`allocation_ownership_ambiguous` on every attempt.
> - Retry repeats the same refusal. The task stays blocked by three
gates: the saved-work wait, the repair flag and the remote writer hold.
The only text left tells a person to inspect the run and its
environment.
> - The source check must stay strict. Recovery must still work without
a person.
> - This pull request adds an automatic fallback: the task continues
from its last committed workspace save, the unverified files are
attached to the task but never applied, and the unverified workspace
epoch is abandoned so its root is never reused.
> - The benefit is that tasks with an unverifiable retained source
continue by themselves, with no lost saved work and no admin action.

## Linked Issues or Issue Description

Refs #19840 (removes the "inspect the run and its environment" text and
keeps Retry; this change removes the need for that Retry).

**What happened?**

A child task ran in a reusable sandbox that it inherited from its parent
task. The run saved a checkpoint, wrote more files, and then failed to
save them. Retained-file recovery refused the source with
`allocation_ownership_ambiguous`, because the lease history of the
sandbox crosses into the parent task. The hold was exhausted after one
attempt. Retry made a new generation that failed the same way. The task
stayed blocked with `saved_work_recovery_failed`,
`workspaceRepairRequired: true`, and the failed run still the dirty
writer of the workspace. A second task had a stale exhausted hold for a
run whose files a sibling recovery action had already saved. That hold
also kept the task from continuing.

**Expected behavior**

Recovery works without a person. When the source cannot be verified, the
task continues from its last saved version. The unverified files are not
applied, but they stay available. A stale hold whose files were already
saved settles as recovered.

**Steps to reproduce**

1. Use a native run with a remote workspace whose reusable sandbox lease
history includes a lease of another task.
2. Let the run save a checkpoint, write more files, and fail its final
save.
3. Let retained-file recovery run, or press Retry.
4. Before this change, the hold is exhausted with
`allocation_ownership_ambiguous` and the task stays blocked.

**Paperclip version or commit**

Observed on deployed builds that include `630411af53`. Holds saved by
the Nov 3–6 releases have this shape.

## What Changed

- New `server/src/services/workspace-recovery-fallback.ts`:
- `continueFromLastSavedWorkspace` runs when a hold is exhausted by a
permanent source refusal: `allocation_ownership_ambiguous`,
`source_binding_invalid`, `source_unavailable` or
`historical_source_missing` with retry class `service_required` or
`use_alternative_strategy`. It applies only to remote workspace sources
in the `prepared` state.
- Quarantine (secrets omitted under the workspace file policy: `.env`,
keys, repository metadata, dependency directories): if the retained root
is readable, the worker copies it with the existing read-only capture
(no write to the sandbox), converts it to a `.tar`, stores it, and
attaches it to the task as `unverified-files-run-<runId>.tar`. A
`workspace_recovery_quarantine` operation records it as `verified:
false, applied: false`. If the root is not readable, the worker records
a closed reason label.
- Stop proof: every allocation of the source run must carry a provider
stop receipt. A missing receipt is obtained through the same stop-only
cleanup as normal recovery (`prepareSandboxStopAndRetain` /
`prepareHistoricalSandboxStopVerification`,
`retryPendingSandboxTeardown`, `settleStopOnlyCleanup`). A sandbox that
another lease holds (`active` / `pending_cleanup`) is never read or
stopped from here; the hold waits.
- One transaction (lock order: task, coordinator, run, recovery action,
workspace) re-checks all facts including the stop receipts, abandons the
epoch, resolves the hold with the named outcome
`retained_source_unverifiable_continued_from_last_saved`, records an
`executionReconciliation` decision with `retainedSourceFallback`, and
denies native replacement. A Blocked task belongs to the hold only while
the hold's own `workspaceFinalizationBlock` receipt still owns that
status version, or, for receipt-less holds saved by older releases,
while the block was set by the run's terminal failure (within its
finalization window) and no status decision followed. Only then is the
task restored to `todo` and a continuation scheduled; a newer block is
kept and no continuation starts. The decision carries machine
`workspaceRepairEvidence`, which the reconciled admission requires for
runs with a restore failure.
- Continuation: with no waiting user input, the hold uses
`continuationDelivery: "pending"`, so the existing reconciled delivery
starts one continuation. Waiting user input continues the task itself
(`not_requested`), so there is never a second continuation.
- Agent note: the decision's `outcomeEvidence` (part of the
continuation's recovery outcomes) says that the run's unsaved changes
could not be verified and were not applied, where the attached copy is,
that the agent should check what to redo, and that completed external
actions must not be repeated.
- `settleSiblingRecoveredHold`: an exhausted hold whose run has a
verified repair receipt from a different recovery action settles as
recovered. It runs for every exhausted hold, including adoptable native
sources. A sibling marked recovered whose receipt no longer verifies
records a permanent deferral, so discovery does not reselect it.
- Deferral: when the fallback cannot settle yet, the hold records
`retainedSourceFallbackCheck` (reason, generation, count). Transient
reasons (live writer, running operation, missing stop, sandbox in use,
pending status effect, lock busy, storage failure) are checked again
after 5 minutes, at most 12 times. Permanent reasons (uncertain external
action, non-remote source) wait for new input, Retry or a new
generation. Storage or database failures after a successful read are
retryable and never recorded as "unreadable".
- New `server/src/services/workspace-recovery-fallback-discovery.ts`:
SQL for pending and due fallbacks. `retainedWorkspaceCandidates` (the
production worker's obligations sweep) and `nextWorkspaceRecoveryAt`
(its scheduler) include them, so the worker reaches old holds without
saved input.
- New `server/src/services/remote-workspace-epoch-abandonment.ts` (kept
out of `remote-workspaces.ts`, which is near 1,000 lines): authority
transition `abandonUnverifiedRemoteWorkspaceEpoch`. It clears the writer
and the dirty flag, keeps `snapshotId`, drops `location`, advances the
epoch, clears `providerRef` and the migration root, and records the
abandoned epoch. The next preparation restores the last save into a new
`<workspace>-<epoch>-<lease>` root.
- `server/src/services/workspace-recovery.ts`: hooks. An exhausted hold
first tries the sibling settlement, then the fallback (under the
existing workspace ownership lock). A fresh permanent refusal in an
attempt runs the fallback in the same pass, under the same ownership.
- `remote-workspace-transport.ts`: export the existing read-only
capture.
- `packages/shared`: optional `retainedSourceFallback` on
`ExecutionReconciliation`.
- `doc/execution-semantics.md`: document the fallback.
- No new polling or worker: the fallback runs inside the existing
retained-workspace recovery pass
(`rules/sleep-aware-background-work.mdc`).

Safety:

- The cross-task lease check (`retiredReusableLeasePredecessors`) is
unchanged and stays strict. Unverified files are never applied.
- No second executor: the source run, its controller and its finalizer
must be stopped. A live writer of the task or workspace, or a running
workspace operation, keeps the hold.
- No repeated external side effects: pending status effects,
non-completed or unknown API receipts, and uncertain tool invocations
keep the existing reconciliation path. The continuation keeps the
existing API receipt fences.
- Never overwrite another run's files: only the exact epoch that the
failed run wrote is abandoned, and the old root is never reused.
- Bounded and idempotent: a settled hold is never revisited. The
quarantine record is reused. A changed generation or revision leaves the
hold unchanged.

Previous-release story (`rules/upgrade-safe-run-state.mdc`): holds saved
by earlier releases carry forward. A hold saved by the Nov 3–6 releases
(`systemRecovery` exhausted with `allocation_ownership_ambiguous` /
`service_required`, board owner, old admin text) takes the fallback on
the next worker pass after deploy (restart sweep, Retry, or a new
message). No migration is needed. Refusals that do not qualify keep
today's behavior. The only new persisted shapes are additive:
`retainedSourceFallback` and `siblingWorkspaceRecovery` on the hold,
`remoteWorkspaceAbandonedEpochs` on the workspace, and a
`workspace_recovery_quarantine` operation. Earlier releases ignore them.

## Verification

- `cd server && npx vitest run
src/services/workspace-recovery-fallback.test.ts` — 15 tests:
- cross-task hold, saved by the previous release and refused fresh:
continues from the checkpoint snapshot, the quarantine `.tar` holds the
unverified file, the epoch is abandoned, `getExecutionBlocker` is
`null`, the task is `todo`, explicit continuation is not blocked by
`saved_work_recovery_failed` or `workspace_repair_required`, one
reconciled continuation is delivered, and the next writer restores into
a new root while the old root keeps its files;
  - unreadable root: continues and records the reason;
- idempotency: a second pass changes nothing and delivers no second
continuation;
  - waiting user input: no second continuation;
- negatives (live writer, pending side effect, uncertain external
action, sandbox held by another lease): the hold stays, the authority is
unchanged, nothing is attached or delivered, the deferral reason and
recheck time are recorded;
- an allocation without a stop receipt is stopped (stop-only) before the
hold settles;
- the production worker (`reconcileRetainedWorkspaces` with
`obligationsOnly`) and `nextWorkspaceRecoveryAt` discover and settle an
old hold with no saved input; a transient deferral is not revisited
until due, then settles;
- an old-release hold without a Blocked receipt keeps its status; a
current hold with the receipt is restored to `todo`;
- stale exhausted hold with a sibling-saved receipt settles as
recovered; one whose sibling receipt no longer verifies is recorded and
no longer discovered;
- a newer Blocked decision is kept and no continuation is delivered,
while the file hold settles;
  - the attached copy omits `.env` and key files.
- RED: with `workspace-recovery.ts` and `system-recovery.ts` reverted to
master, all tests fail.
- Regression suites pass: `workspace-recovery-warm-handoff`,
`workspace-recovery-retry-copyback`,
`workspace-recovery-preparation-gates`, `workspace-recovery-source`,
`task-retry`, `execution-blocker-read` (288 tests), and
`workspace-recovery-admission`, `workspace-recovery-preparation*`,
`remote-workspaces.integration`, `native-failed-workspace-recovery`,
`remote-workspace-transport`, `workspace-restore-repair-route` (683
tests).
- `server` typecheck passes.

## Risks

- A task now continues without the unverified changes of its failed run.
The agent receives a note and the attached copy, so it can redo or
re-apply the work. This replaces an indefinite block.
- Abandoning an epoch on a workspace shared with a parent task also
moves the parent to the last save. This is correct: the unverified
changes are not part of the saved lineage.
- Clearing the migration root means sessions bound to the old root start
fresh. The fallback continuation already forces a fresh session.
- The quarantine capture reads a sandbox that can be shared with another
task. It writes nothing there, and it verifies the copy against
inventories taken before and after.

## Model Used

- Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), with
tool use (shell, file edits, read-only browser for API inspection) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/execution-semantics.md

@@ -2307,6 +2307,41 @@ proof must match exactly; an absent or foreign field cannot certify a folded
 legacy Retry. Ordinary legacy history that already names the exact preservation
 generation retains only its existing passive proof, without status ownership.
 
+Retained-file recovery verifies the recorded source strictly, including the
+reusable-lease chain of its sandbox. When that verification refuses the source
+permanently (`allocation_ownership_ambiguous`, `source_binding_invalid`,
+`source_unavailable` or `historical_source_missing`), retrying repeats the
+refusal. The recovery worker then continues the task from its last committed
+workspace save, with the named outcome
+`retained_source_unverifiable_continued_from_last_saved`. The unverified files
+are never applied. When the retained root can still be read, the worker copies it
+without writing to the sandbox and attaches the copy to the task as unverified
+files of that run. The copy follows the workspace file policy: secrets,
+repository metadata and dependency directories are omitted. Otherwise it records why the root could not be read. In one
+transaction the worker then abandons the unverified remote workspace epoch:
+the writer and dirty flag are cleared, the last save is kept, the location and
+published root are dropped and the epoch advances. The next writer therefore
+restores the last save into a new directory and never reuses the old root. The
+same transaction resolves the hold and denies native replacement. A Blocked
+task belongs to the hold only while the hold's own Blocked receipt still owns
+the status, or, for holds saved before receipts existed, while the block was set
+by the run's terminal failure and no status decision followed. Only then is the
+task restored to `todo`; otherwise the newer block is kept and no continuation is
+started. If no user input is waiting, one continuation is delivered through the
+reconciled execution path; the decision carries the workspace repair evidence
+that a restore failure requires. Its recovery outcome tells the agent which
+changes were not applied and where the attached copy is. Every allocation of
+the source run must carry a provider stop receipt; the worker obtains a missing
+one through the same stop-only cleanup as normal recovery, but never stops a
+sandbox that another lease holds. The fallback does not run while a writer is
+live, a status effect is pending, or an external action has an uncertain
+outcome; those keep their existing reconciliation. A deferred check records its
+reason. Transient reasons are checked again after five minutes, at most twelve
+times; new input, Retry or a new recovery generation reconsiders the hold. The
+recovery worker discovers these holds itself, so holds saved by earlier releases
+take the same path after deploy. An exhausted hold whose run's files a sibling
+recovery action already saved settles as recovered.
+
 An exhausted workspace export records the exact Blocked status version it
 creates, bound to its source run, result, contract and identity. Recovery may
 restore the accepted passive wait only while that receipt still owns the task

packages/shared/src/types/execution-projection.ts

@@ -98,6 +98,22 @@ export interface ExecutionReconciliation {
     sources: Array<{ attachmentId: string; sha256: string; byteSize: number; filename: string | null }>;
     interveningExecutions?: Array<{ runId: string; recoveryActionId: string; evidenceSha256: string }>;
   };
+  /**
+   * The system continued from the last committed workspace save because the
+   * retained source of this run could not be verified. Its unsaved files were
+   * never applied; when readable they are attached to the task, unverified.
+   */
+  retainedSourceFallback?: {
+    schema: "paperclip.retained-source-fallback.v1";
+    outcome: "retained_source_unverifiable_continued_from_last_saved";
+    workspaceId: string;
+    abandonedEpoch: number;
+    epoch: number;
+    snapshotId: string | null;
+    quarantine:
+      | { state: "captured"; attachmentId: string; filename: string; sha256: string; byteSize: number }
+      | { state: "unreadable"; reason: string };
+  };
 }
 /** A task-level view of one interruption. Diagnostics keep the underlying runs
  * and recovery records; ordinary product surfaces render this projection. */

server/src/services/remote-workspace-epoch-abandonment.ts

@@ -0,0 +1,49 @@
+import { and, eq } from "drizzle-orm";
+import { executionWorkspaces, type Db } from "@paperclipai/db";
+import { readRemoteWorkspaceAuthority, type RemoteWorkspaceAuthority } from "@paperclipai/shared";
+import { persistActivity } from "./activity-log.js";
+
+type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
+export const REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA = "paperclip.remote-workspace-abandoned-epoch.v1";
+
+/**
+ * Ends an epoch whose unsaved files cannot be verified, without applying them.
+ * The last committed save stays the workspace's content. The writer, its dirty
+ * flag and its location are cleared and the epoch advances, so the old writer
+ * can no longer publish. The published root and any migration root are
+ * forgotten, so the next preparation restores the saved version into a new
+ * directory and never reuses or overwrites the root that holds the unverified
+ * files. The caller holds the task, run and recovery locks; this locks the
+ * workspace last.
+ */
+export async function abandonUnverifiedRemoteWorkspaceEpoch(tx: Transaction, input: {
+  companyId: string; workspaceId: string; runId: string; epoch: number; snapshotId: string | null;
+  location: NonNullable<RemoteWorkspaceAuthority["location"]>;
+}) {
+  const [workspace] = await tx.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.id, input.workspaceId),
+    eq(executionWorkspaces.companyId, input.companyId))).for("update");
+  const current = workspace ? readRemoteWorkspaceAuthority(workspace.metadata) : null;
+  const location = current?.location;
+  if (!workspace || !current || !location || current.writerRunId !== input.runId || current.epoch !== input.epoch
+    || current.snapshotId !== input.snapshotId
+    || (["environmentId", "leaseId", "providerLeaseId", "remoteCwd"] as const).some(key => location[key] !== input.location[key])) {
+    throw new Error("remote_workspace_authority_changed");
+  }
+  const next: RemoteWorkspaceAuthority = { ...current, epoch: current.epoch + 1, writerRunId: null, dirty: false, location: null };
+  const prior = Array.isArray(workspace.metadata?.remoteWorkspaceAbandonedEpochs)
+    ? workspace.metadata.remoteWorkspaceAbandonedEpochs as unknown[] : [];
+  const abandoned = { schema: REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA, epoch: current.epoch, runId: input.runId,
+    snapshotId: current.snapshotId, location, providerRef: workspace.providerRef,
+    ...(workspace.metadata?.nativeWorkspaceMigration !== undefined ? { nativeWorkspaceMigration: workspace.metadata.nativeWorkspaceMigration } : {}),
+    abandonedAt: new Date().toISOString() };
+  const metadata: Record<string, unknown> = { ...workspace.metadata, remoteWorkspace: next,
+    remoteWorkspaceAbandonedEpochs: [...prior, abandoned].slice(-20) };
+  // The migration receipt names the old root as the next preparation's root.
+  delete metadata.nativeWorkspaceMigration;
+  const [saved] = await tx.update(executionWorkspaces).set({ providerRef: null, cwd: `/workspaces/${workspace.id}`, metadata,
+    updatedAt: new Date() }).where(and(eq(executionWorkspaces.id, workspace.id), eq(executionWorkspaces.companyId, input.companyId))).returning();
+  await persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system", actorId: "remote-workspaces",
+    action: "workspace.remote_epoch_abandoned", entityType: "execution_workspace", entityId: workspace.id, runId: input.runId,
+    details: { abandonedEpoch: current.epoch, epoch: next.epoch, snapshotId: next.snapshotId } });
+  return { workspace: saved!, authority: next };
+}

server/src/services/remote-workspace-transport.ts

@@ -465,7 +465,9 @@ export async function captureRemoteWorkspaceArchive(input: {
   }
 }
 
-async function captureReadOnlyWorkspaceArchive(input: {
+/** Captures a sandbox directory without writing to the sandbox. The copy is
+ * verified against an inventory taken before and after the transfer. */
+export async function captureReadOnlyWorkspaceArchive(input: {
   target: RemoteWorkspaceTarget; remoteCwd: string; operationId: string;
 }) {
   if (!/^[a-f0-9-]{36}$/.test(input.operationId)) throw new Error("remote_workspace_operation_identity_invalid");

server/src/services/system-recovery.ts

@@ -5,6 +5,7 @@ import { writeExecutionControl } from "./execution-control-notifications.js";
 import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
 import { createSystemRecovery, exhaustedSystemRecoveryAction, readSystemRecovery, type SystemRecoveryState } from "./system-recovery-state.js";
 import { workspaceRecoverySourceEligible, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js";
+import { retainedSourceFallbackDueAt, retainedSourceFallbackPendingCondition } from "./workspace-recovery-fallback-discovery.js";
 
 export async function adoptSystemRecovery(db: Db, source: WorkspaceRecoverySourceIdentity) {
   return db.transaction(async tx => {
@@ -113,6 +114,7 @@ export async function loadSystemRecovery(db: Db, source: WorkspaceRecoverySource
 
 export async function nextWorkspaceRecoveryAt(db: Db): Promise<number | null> {
   const rows = await db.select({ at: sql<number>`case
+    when ${retainedSourceFallbackPendingCondition()} then ${retainedSourceFallbackDueAt()}
     when jsonb_typeof(${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}') = 'number'
     then (${issueRecoveryActions.evidence}#>>'{systemRecovery,nextAttemptAt}')::bigint else 0 end` })
     .from(issueRecoveryActions).innerJoin(heartbeatRuns, and(
@@ -123,6 +125,7 @@ export async function nextWorkspaceRecoveryAt(db: Db): Promise<number | null> {
     )).where(and(
       workspaceRecoverySourceEligible(true),
       sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
+        or ${retainedSourceFallbackPendingCondition()}
         or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete'
           and ${issueRecoveryActions.evidence}->>'bootstrapPreparationSettledAt' is not null
           and coalesce(${issueRecoveryActions.evidence}->>'bootstrapPreparationContinuation', 'pending') = 'pending'

server/src/services/workspace-recovery-fallback-discovery.ts

@@ -0,0 +1,48 @@
+import { sql } from "drizzle-orm";
+import { issueRecoveryActions } from "@paperclipai/db";
+
+/** Source refusals that repeat on every retry (see workspace-recovery-fallback). */
+export const UNVERIFIABLE_SOURCE_CODES = ["allocation_ownership_ambiguous", "source_binding_invalid", "source_unavailable", "historical_source_missing"] as const;
+export const PERMANENT_SOURCE_RETRY_CLASSES = ["service_required", "use_alternative_strategy"] as const;
+export const RETAINED_SOURCE_FALLBACK_CHECK_KEY = "retainedSourceFallbackCheck";
+
+const list = (values: readonly string[]) => sql.join(values.map(value => sql`${value}`), sql`, `);
+const check = sql`${issueRecoveryActions.evidence}->${RETAINED_SOURCE_FALLBACK_CHECK_KEY}`;
+
+/**
+ * An exhausted hold that the fallback may still settle: its source is refused
+ * permanently, or a sibling recovery action of the same run saved its files.
+ * The recovery worker discovers it like other unfinished work. A deferred
+ * check of the same generation waits for its recorded time; a bounded number of
+ * deferrals ends with no time, so the worker stops scheduling it (new input,
+ * Retry or a new generation reconsiders it).
+ */
+export function retainedSourceFallbackPendingCondition() {
+  return sql`(${issueRecoveryActions.status} in ('active', 'escalated')
+    and ${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'exhausted'
+    and ${issueRecoveryActions.evidence}->'retainedSourceFallback' is null
+    and ((${issueRecoveryActions.cause} in ('native_workspace_sync_out_unrecoverable', 'native_workspace_sync_out_retry_exhausted')
+        and ${issueRecoveryActions.evidence}#>>'{systemRecovery,failure,code}' in (${list(UNVERIFIABLE_SOURCE_CODES)})
+        and ${issueRecoveryActions.evidence}#>>'{systemRecovery,failure,retryClass}' in (${list(PERMANENT_SOURCE_RETRY_CLASSES)})
+        and exists (select 1 from heartbeat_runs source where source.company_id = ${issueRecoveryActions.companyId}
+          and source.id::text = ${issueRecoveryActions.evidence}->>'runId'
+          and source.runner_profile_json#>>'{remoteWorkspace,state}' = 'prepared'))
+      or exists (select 1 from issue_recovery_actions sibling
+        where sibling.company_id = ${issueRecoveryActions.companyId} and sibling.source_issue_id = ${issueRecoveryActions.sourceIssueId}
+          and sibling.id <> ${issueRecoveryActions.id} and sibling.evidence->>'runId' = ${issueRecoveryActions.evidence}->>'runId'
+          and sibling.evidence#>>'{systemRecovery,state}' = 'complete'
+          and sibling.evidence#>>'{workspaceRecovery,status}' = 'recovered'))
+    and (${check} is null
+      or ${check}->>'generationId' is distinct from ${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}'
+      or jsonb_typeof(${check}->'nextCheckAt') = 'number'))`;
+}
+
+/** When the pending fallback is next due, in epoch milliseconds (0 = now). */
+export function retainedSourceFallbackDueAt() {
+  return sql`case when ${check}->>'generationId' = ${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}'
+    and jsonb_typeof(${check}->'nextCheckAt') = 'number' then (${check}->>'nextCheckAt')::bigint else 0 end`;
+}
+
+export function retainedSourceFallbackDueCondition(now: number) {
+  return sql`(${retainedSourceFallbackPendingCondition()} and ${retainedSourceFallbackDueAt()} <= ${now}::bigint)`;
+}

server/src/services/workspace-recovery-fallback.test.ts

@@ -0,0 +1,402 @@
+import { randomUUID } from "node:crypto";
+import { execFile } from "node:child_process";
+import { cp, mkdtemp, readFile, realpath, rm, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { and, eq, inArray } from "drizzle-orm";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import { agentWakeupRequests, assets, environmentLeases, executionWorkspaces, heartbeatRuns, issueAttachments, issueComments,
+  issueRecoveryActions, issues, nativeRunFinalizations, workspaceOperations } from "@paperclipai/db";
+import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
+import { readRemoteWorkspaceAuthority, type EnvironmentLease } from "@paperclipai/shared";
+import { getStorageService } from "../storage/index.js";
+import { admitExplicitNativeContinuation } from "./explicit-native-continuation.js";
+import { getExecutionBlocker } from "./execution-blocker.js";
+import { deliverReconciledExecutions } from "./execution-recovery-resolution.js";
+import { prepareRemoteWorkspace, reserveRemoteWorkspace, saveRemoteWorkspace } from "./remote-workspaces.js";
+import { readSystemRecovery } from "./system-recovery-state.js";
+import { reconcileRetainedWorkspaces, recoverRetainedWorkspace, retainedWorkspaceCandidates } from "./workspace-recovery.js";
+import { nextWorkspaceRecoveryAt } from "./system-recovery.js";
+import { RETAINED_SOURCE_FALLBACK_OUTCOME, RETAINED_SOURCE_QUARANTINE_PHASE } from "./workspace-recovery-fallback.js";
+import { hasVerifiedWorkspaceRepair, readWorkspaceRecovery } from "./workspace-recovery-receipt.js";
+import { acceptedRemoteExport, action, connectionString, db, run, useWorkspaceRecoveryFixture } from "./__tests__/workspace-recovery-fixture.js";
+
+const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
+vi.mock("./environment-execution-target.js", async importOriginal => {
+  const actual = await importOriginal<typeof import("./environment-execution-target.js")>();
+  return { ...actual, resolveEnvironmentExecutionTarget: async (input: Parameters<typeof actual.resolveEnvironmentExecutionTarget>[0]) =>
+    (input.leaseId ? targets.get(input.leaseId) : undefined) ?? actual.resolveEnvironmentExecutionTarget(input) };
+});
+
+// Recovery text never asks a person to inspect or fix a sandbox.
+const adminText = /sandbox|environment|administrator|board member|owner must|inspect the run/i;
+const workspace = async (id: string) => (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, id)))[0]!;
+
+describe("retained source fallback", () => {
+  useWorkspaceRecoveryFixture(targets);
+  // Remote workspace storage needs external Postgres and a stable key, for
+  // the checkpoint save and the later restore alike.
+  beforeEach(() => {
+    vi.stubEnv("DATABASE_URL", connectionString);
+    vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "63".repeat(32));
+  });
+  afterEach(() => { vi.unstubAllEnvs(); });
+
+  /**
+   * A child task's run saved a checkpoint (the last committed
+   * snapshot), then wrote more files and failed to save them. Its reusable
+   * sandbox lease history crosses into the parent task, so the recorded source
+   * is refused with allocation_ownership_ambiguous on every attempt.
+   */
+  async function unverifiableRetainedSource(saved: "old_release" | "fresh" = "old_release") {
+    const f = await acceptedRemoteExport("retained");
+    const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
+    const checkpoint = await saveRemoteWorkspace({ db, workspace: await workspace(f.workspaceId), runId: f.runId,
+      lease: lease as EnvironmentLease, target: targets.get(f.lease.id)!, assertOwned: async () => {}, boundary: "checkpoint" });
+    // The published root a later preparation would otherwise reuse.
+    await db.update(executionWorkspaces).set({ providerRef: f.remote }).where(eq(executionWorkspaces.id, f.workspaceId));
+    await writeFile(path.join(f.remote, "unverified.txt"), "unsaved work\n");
+    await writeFile(path.join(f.remote, ".env"), "TOKEN=secret\n");
+    await writeFile(path.join(f.remote, "id_rsa"), "private key\n");
+    // The run's terminal failure blocked the task (older releases kept no receipt).
+    const finished = (await run(f.runId)).finishedAt!;
+    await db.update(issues).set({ blockedTransitionAt: new Date(finished.getTime() + 100) }).where(eq(issues.id, f.issueId));
+    await db.update(issueRecoveryActions).set({ createdAt: finished }).where(eq(issueRecoveryActions.id, f.actionId));
+    const [parent] = await db.insert(issues).values({ companyId: f.companyId, title: "Parent task" }).returning();
+    await db.update(issues).set({ parentId: parent.id }).where(eq(issues.id, f.issueId));
+    await db.insert(environmentLeases).values({ companyId: f.companyId, environmentId: lease.environmentId, issueId: parent.id,
+      provider: "daytona", providerLeaseId: f.providerLeaseId, status: "expired", leasePolicy: "reuse_by_environment",
+      cleanupStatus: "success", releasedAt: new Date(), metadata: { remoteCwd: path.dirname(f.remote) } });
+    await db.update(nativeRunFinalizations).set({ failureCode: "native_workspace_sync_out_retry_exhausted" })
+      .where(eq(nativeRunFinalizations.runId, f.runId));
+    const oldMessage = "Automatic recovery of saved work stopped after 1 attempts (last failure: allocation_ownership_ambiguous). "
+      + "Retry recovery to export and merge from the retained sandbox again; no provider turn is submitted. "
+      + "If Retry keeps failing, inspect the run and its environment.";
+    if (saved === "old_release") {
+      // The exact shape saved by the Nov 3-6 releases.
+      const current = readSystemRecovery((await action(f.actionId)).evidence)!;
+      await db.update(issueRecoveryActions).set({ ownerType: "board", nextAction: oldMessage, evidence: { ...(await action(f.actionId)).evidence,
+        systemRecovery: { ...current, revision: current.revision + 5, phase: "verify_execution", state: "exhausted", attempts: 1,
+          attemptId: randomUUID(), firstFailureAt: Date.now() - 60_000, deadlineAt: Date.now() + 60_000, nextAttemptAt: null,
+          failure: { code: "allocation_ownership_ambiguous", retryClass: "service_required", evidenceFingerprint: "a".repeat(64) } },
+        workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: f.runId, status: "blocked", stage: "verifying_stop",
+          attempt: 1, attemptLimit: 6, retryable: true, requestIds: [], message: oldMessage } } })
+        .where(eq(issueRecoveryActions.id, f.actionId));
+    }
+    return { ...f, snapshotId: checkpoint.snapshotId, parentIssueId: parent.id };
+  }
+
+  async function continuationBlock(f: { companyId: string; issueId: string; runId: string }) {
+    const source = await run(f.runId);
+    const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
+      authorType: "user", authorUserId: "operator", body: "Please continue" }).returning();
+    const reasons: string[] = [];
+    await admitExplicitNativeContinuation({ db, companyId: f.companyId, issueId: f.issueId, agentId: source.agentId,
+      actorType: "user", actorId: "operator", reason: "issue_commented", commentId: comment.id, successorRunId: randomUUID(),
+      dryRun: true, onBlocked: reason => reasons.push(reason) });
+    return reasons;
+  }
+
+  it.each(["old_release", "fresh"] as const)("continues a cross-task retained hold (%s) from the last save, quarantined and without reusing its root", async saved => {
+    const f = await unverifiableRetainedSource(saved);
+    expect((await getExecutionBlocker(db, f.companyId, f.issueId))?.workspaceRepairRequired).toBe(true);
+    if (saved === "fresh") {
+      // A current release records which Blocked status version this hold owns.
+      const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
+      const source = await run(f.runId);
+      await db.update(issueRecoveryActions).set({ evidence: { ...(await action(f.actionId)).evidence, workspaceFinalizationBlock: {
+        schema: "paperclip.native-workspace-finalization-block.v1", companyId: f.companyId, issueId: f.issueId, runId: f.runId,
+        agentId: source.agentId, resultId: f.resultId, resultSha256: "result", contractId: source.completionContractId,
+        contractSha256: "contract", identityContextId: null, responsibleUserId: null,
+        statusVersion: Number(task.statusVersion), priorDecisionId: task.lastStatusDecisionId } } })
+        .where(eq(issueRecoveryActions.id, f.actionId));
+    }
+
+    await recoverRetainedWorkspace(f.input);
+
+    const settled = await action(f.actionId);
+    expect(settled).toMatchObject({ status: "resolved", outcome: "restored", ownerType: "system",
+      resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME, returnOwnerAgentId: (await run(f.runId)).agentId });
+    expect(readSystemRecovery(settled.evidence)?.failure?.code).toBe("allocation_ownership_ambiguous");
+    const decision = settled.evidence.executionReconciliation as Record<string, unknown>;
+    expect(decision).toMatchObject({ runId: f.runId, providerStopped: true, actionOutcome: "completed",
+      retainedSourceFallback: { outcome: RETAINED_SOURCE_FALLBACK_OUTCOME, snapshotId: f.snapshotId,
+        quarantine: { state: "captured" } } });
+    expect(settled.evidence.continuationDelivery).toBe("pending");
+    for (const text of [settled.nextAction, readWorkspaceRecovery(settled.evidence)?.message, decision.outcomeEvidence])
+      expect(String(text)).not.toMatch(adminText);
+    expect(String(decision.outcomeEvidence)).toContain(f.snapshotId);
+
+    // The unverified files are attached to the task, never applied.
+    const quarantine = (decision.retainedSourceFallback as { quarantine: { attachmentId: string; sha256: string } }).quarantine;
+    const [attachment] = await db.select({ asset: assets }).from(issueAttachments)
+      .innerJoin(assets, eq(assets.id, issueAttachments.assetId)).where(eq(issueAttachments.id, quarantine.attachmentId));
+    expect(attachment.asset).toMatchObject({ companyId: f.companyId, sha256: quarantine.sha256, contentType: "application/x-tar" });
+    const scratch = await mkdtemp(path.join(os.tmpdir(), "quarantine-check-"));
+    try {
+      const object = await getStorageService().getObject(f.companyId, attachment.asset.objectKey);
+      const chunks: Buffer[] = [];
+      for await (const chunk of object.stream) chunks.push(chunk as Buffer);
+      await writeFile(path.join(scratch, "files.tar"), Buffer.concat(chunks));
+      const listing = await new Promise<string>((resolve, reject) => execFile("tar", ["-tf", path.join(scratch, "files.tar")],
+        (error, stdout) => error ? reject(error) : resolve(stdout)));
+      expect(listing).toContain("unverified.txt");
+      // The workspace file policy applies: secrets are never attached.
+      expect(listing).not.toMatch(/\.env|id_rsa/);
+    } finally { await rm(scratch, { recursive: true, force: true }); }
+
+    // The unverified epoch is abandoned: no writer, not dirty, the last save kept, no location.
+    const abandoned = await workspace(f.workspaceId);
+    const before = (await run(f.runId)).runnerProfileJson!.remoteWorkspace as { epoch: number };
+    expect(readRemoteWorkspaceAuthority(abandoned.metadata)).toMatchObject({ epoch: before.epoch + 1, writerRunId: null,
+      dirty: false, location: null, snapshotId: f.snapshotId });
+    expect(abandoned.providerRef).toBeNull();
+
+    // Every gate is settled: the task hold, the repair flag and the saved-work wait.
+    expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
+    expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("todo");
+    expect(String(decision.workspaceRepairEvidence).length).toBeGreaterThanOrEqual(20);
+    expect(await continuationBlock(f)).not.toEqual(expect.arrayContaining(["saved_work_recovery_failed"]));
+    expect(await continuationBlock(f)).not.toEqual(expect.arrayContaining(["workspace_repair_required"]));
+
+    // One continuation is delivered through the normal reconciled path.
+    const wake = vi.fn(async () => ({ id: randomUUID() }));
+    await deliverReconciledExecutions(db, wake as never);
+    await deliverReconciledExecutions(db, wake as never);
+    const calls = wake.mock.calls.filter(call => (call as unknown[])[0] === settled.returnOwnerAgentId);
+    expect(calls).toHaveLength(1);
+    expect((calls[0] as unknown[])[1]).toMatchObject({ reason: "issue_recovery_action_restored",
+      contextSnapshot: { issueId: f.issueId, previousRunId: f.runId, forceFreshSession: true, source: "execution.reconciled" } });
+
+    // The next writer restores the last save into a new root; the old root is untouched.
+    const source = await run(f.runId);
+    const nextRunId = randomUUID(), nextLeaseId = randomUUID();
+    await db.insert(heartbeatRuns).values({ id: nextRunId, companyId: f.companyId, agentId: source.agentId, issueId: f.issueId,
+      nativeIssueId: f.issueId, runtimeMode: "native", status: "running", contextSnapshot: { issueId: f.issueId } });
+    const storage = { ...getStorageService(), provider: "s3" as const };
+    const reserved = await reserveRemoteWorkspace({ db, storage, companyId: f.companyId, issueId: f.issueId, runId: nextRunId,
+      agentId: source.agentId, environmentId: f.lease.environmentId!, projectId: null, projectWorkspaceId: null,
+      existingWorkspaceId: f.workspaceId, source: { kind: "empty" } });
+    const [nextLease] = await db.insert(environmentLeases).values({ id: nextLeaseId, companyId: f.companyId, issueId: f.issueId,
+      heartbeatRunId: nextRunId, environmentId: f.lease.environmentId, provider: "daytona", providerLeaseId: f.providerLeaseId,
+      status: "active", leasePolicy: "reuse_by_environment" }).returning();
+    const oldRoot = await realpath(f.remote), base = path.dirname(path.dirname(oldRoot));
+    const sandbox = targets.get(f.lease.id)!;
+    // The same sandbox: transfers into it are plain copies.
+    const runner = { ...sandbox.runner!, syncIn: async (operations: Parameters<NonNullable<typeof sandbox.runner>["syncOut"] & {}>[0]) => {
+      for (const operation of operations) for (const file of operation.files)
+        await cp(file.sourcePath, file.targetPath, { recursive: true, dereference: false });
+      return { filesTransferred: 1, bytesTransferred: 0, operations: [] };
+    } };
+    const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: nextRunId,
+      lease: nextLease as EnvironmentLease, target: { ...sandbox, runner, leaseId: nextLeaseId, remoteCwd: base } });
+    expect(prepared.remoteRoot).not.toBe(oldRoot);
+    expect(path.dirname(prepared.remoteRoot)).toBe(path.dirname(oldRoot));
+    expect(await readFile(path.join(prepared.remoteRoot, "saved.txt"), "utf8")).toBe("finished work\n");
+    await expect(readFile(path.join(prepared.remoteRoot, "unverified.txt"))).rejects.toMatchObject({ code: "ENOENT" });
+    expect(await readFile(path.join(f.remote, "unverified.txt"), "utf8")).toBe("unsaved work\n");
+  }, 60_000);
+
+  it("keeps a newer Blocked decision and starts no continuation, while settling the file hold", async () => {
+    const f = await unverifiableRetainedSource();
+    await db.update(issues).set({ blockedTransitionAt: new Date(Date.now() + 120_000) }).where(eq(issues.id, f.issueId));
+    await recoverRetainedWorkspace(f.input);
+    const settled = await action(f.actionId);
+    expect(settled).toMatchObject({ status: "resolved", resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME });
+    expect(settled.evidence.continuationDelivery).toBe("not_requested");
+    expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("blocked");
+    const wake = vi.fn(async () => ({ id: randomUUID() }));
+    await deliverReconciledExecutions(db, wake as never);
+    expect(wake.mock.calls.filter(call => (call as unknown[])[0] === settled.returnOwnerAgentId)).toHaveLength(0);
+  }, 60_000);
+
+  it("still continues when the retained root cannot be read, recording why", async () => {
+    const f = await unverifiableRetainedSource();
+    await rm(f.remote, { recursive: true, force: true });
+    await recoverRetainedWorkspace(f.input);
+    const settled = await action(f.actionId);
+    expect(settled).toMatchObject({ status: "resolved", resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME });
+    const fallback = (settled.evidence.executionReconciliation as { retainedSourceFallback: { quarantine: { state: string; reason: string } } })
+      .retainedSourceFallback;
+    expect(fallback.quarantine.state).toBe("unreadable");
+    expect(fallback.quarantine.reason).toMatch(/^[a-z][a-z0-9_]*$/);
+    expect(String(settled.nextAction)).not.toMatch(adminText);
+    expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)).toMatchObject({ writerRunId: null, dirty: false,
+      location: null, snapshotId: f.snapshotId });
+    expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
+    expect(await db.select().from(issueAttachments).where(eq(issueAttachments.issueId, f.issueId))).toHaveLength(0);
+  }, 60_000);
+
+  it("is idempotent: a second pass changes nothing and delivers no second continuation", async () => {
+    const f = await unverifiableRetainedSource();
+    const agentId = (await run(f.runId)).agentId;
+    await recoverRetainedWorkspace(f.input);
+    const first = await action(f.actionId), firstWorkspace = await workspace(f.workspaceId);
+    await recoverRetainedWorkspace(f.input);
+    expect(await action(f.actionId)).toEqual(first);
+    expect(await workspace(f.workspaceId)).toEqual(firstWorkspace);
+    expect(await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.heartbeatRunId, f.runId),
+      eq(workspaceOperations.phase, RETAINED_SOURCE_QUARANTINE_PHASE)))).toHaveLength(1);
+    expect(await db.select().from(issueAttachments).where(eq(issueAttachments.issueId, f.issueId))).toHaveLength(1);
+    const wake = vi.fn(async () => ({ id: randomUUID() }));
+    await deliverReconciledExecutions(db, wake as never);
+    await deliverReconciledExecutions(db, wake as never);
+    expect(wake.mock.calls.filter(call => (call as unknown[])[0] === agentId)).toHaveLength(1);
+  }, 60_000);
+
+  it("leaves saved user input to continue the task instead of starting a second continuation", async () => {
+    const f = await unverifiableRetainedSource();
+    await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: (await run(f.runId)).agentId, source: "on_demand",
+      reason: "issue_commented", status: "deferred_issue_execution", requestedByActorType: "user", requestedByActorId: "operator",
+      payload: { issueId: f.issueId } });
+    await recoverRetainedWorkspace(f.input);
+    const settled = await action(f.actionId);
+    expect(settled).toMatchObject({ status: "resolved", resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME });
+    expect(settled.evidence.continuationDelivery).toBe("not_requested");
+  }, 60_000);
+
+  it.each(["live_writer", "pending_side_effect", "uncertain_external_action", "sandbox_in_use"] as const)(
+    "keeps the hold and starts no executor with a %s", async fault => {
+      const f = await unverifiableRetainedSource();
+      const source = await run(f.runId);
+      if (fault === "live_writer") await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId,
+        issueId: f.issueId, runtimeMode: "native", status: "running", startedAt: new Date(), contextSnapshot: { issueId: f.issueId } });
+      if (fault === "sandbox_in_use") {
+        // The source allocation has no stop receipt, and another task holds the sandbox.
+        const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
+        const metadata = { ...lease.metadata }; delete metadata.remoteExecutionTermination;
+        await db.update(environmentLeases).set({ metadata }).where(eq(environmentLeases.id, f.lease.id));
+        await db.update(environmentLeases).set({ status: "active", releasedAt: null }).where(and(
+          eq(environmentLeases.issueId, f.parentIssueId), eq(environmentLeases.providerLeaseId, f.providerLeaseId)));
+      }
+      if (fault === "pending_side_effect" || fault === "uncertain_external_action") await db.update(heartbeatRuns).set({ resultJson: { ...source.resultJson, apiToolReceipts: {
+        effect: fault === "pending_side_effect" ? { state: "pending", operationId: "POST /api/x" }
+          : { state: "completed", operationId: "POST /api/x", result: { ok: false, status: 502 } } } } }).where(eq(heartbeatRuns.id, f.runId));
+      const authority = readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata);
+      await recoverRetainedWorkspace(f.input);
+      const held = await action(f.actionId);
+      expect(held.status).toBe("active");
+      expect(held.evidence.executionReconciliation).toBeUndefined();
+      expect(held.evidence.continuationDelivery).toBeUndefined();
+      expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)).toEqual(authority);
+      expect(await db.select().from(issueAttachments).where(eq(issueAttachments.issueId, f.issueId))).toHaveLength(0);
+      const wake = vi.fn(async () => ({ id: randomUUID() }));
+      await deliverReconciledExecutions(db, wake as never);
+      expect(wake.mock.calls.filter(call => (call as unknown[])[0] === source.agentId)).toHaveLength(0);
+      if (fault === "sandbox_in_use") expect(f.runtime.retryPendingSandboxTeardown).not.toHaveBeenCalled();
+      // The deferral is recorded: transient waits are checked again later, a
+      // permanent refusal waits for new input or a new generation.
+      const check = held.evidence.retainedSourceFallbackCheck as { reason: string; nextCheckAt: number | null };
+      expect(check.reason).toBe({ live_writer: "writer_active", pending_side_effect: "external_action_uncertain",
+        uncertain_external_action: "external_action_uncertain", sandbox_in_use: "allocation_in_use" }[fault]);
+      expect(check.nextCheckAt === null).toBe(check.reason === "external_action_uncertain");
+    }, 60_000);
+
+  it("obtains a stop receipt for an allocation that was never stopped before continuing", async () => {
+    const f = await unverifiableRetainedSource();
+    const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
+    const metadata = { ...lease.metadata }; delete metadata.remoteExecutionTermination;
+    await db.update(environmentLeases).set({ metadata }).where(eq(environmentLeases.id, f.lease.id));
+    await recoverRetainedWorkspace(f.input);
+    expect(f.runtime.retryPendingSandboxTeardown).toHaveBeenCalledOnce();
+    expect(await action(f.actionId)).toMatchObject({ status: "resolved", resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME });
+    const [stopped] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
+    expect(stopped.metadata?.remoteExecutionTermination).toMatchObject({ state: "stopped", leaseId: f.lease.id });
+  }, 60_000);
+
+  it("is discovered and settled by the recovery worker without any user action", async () => {
+    const f = await unverifiableRetainedSource();
+    expect(await nextWorkspaceRecoveryAt(db)).toBeLessThanOrEqual(Date.now());
+    const onRecovered = vi.fn(async () => undefined);
+    const work = await reconcileRetainedWorkspaces(db, f.environmentRuntime, onRecovered, true);
+    await Promise.all(work ?? []);
+    expect(await action(f.actionId)).toMatchObject({ status: "resolved", resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME });
+    expect(onRecovered).toHaveBeenCalledOnce();
+    const due = await nextWorkspaceRecoveryAt(db);
+    expect(due === null || due > Date.now() - 1).toBe(true);
+    expect(await retainedWorkspaceCandidates(db, undefined, false, true)).not.toEqual(
+      expect.arrayContaining([expect.objectContaining({ action: expect.objectContaining({ id: f.actionId }) })]));
+  }, 60_000);
+
+  it("waits until a deferred transient check is due before the worker visits the hold again", async () => {
+    const f = await unverifiableRetainedSource();
+    const source = await run(f.runId);
+    const [writer] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, issueId: f.issueId,
+      runtimeMode: "native", status: "running", startedAt: new Date(), contextSnapshot: { issueId: f.issueId } }).returning();
+    const work = await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => undefined, true);
+    await Promise.all(work ?? []);
+    const deferred = await action(f.actionId);
+    expect(deferred.status).toBe("active");
+    const ids = async () => (await retainedWorkspaceCandidates(db, undefined, false, true)).map(row => row.action.id);
+    expect(await ids()).not.toContain(f.actionId);
+    expect(await nextWorkspaceRecoveryAt(db)).toBeGreaterThan(Date.now());
+    // Once the writer ended and the check is due, the worker settles the hold.
+    await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, writer.id));
+    await db.update(issueRecoveryActions).set({ evidence: { ...deferred.evidence, retainedSourceFallbackCheck: {
+      ...(deferred.evidence.retainedSourceFallbackCheck as object), nextCheckAt: Date.now() - 1 } } }).where(eq(issueRecoveryActions.id, f.actionId));
+    expect(await ids()).toContain(f.actionId);
+    await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => undefined, true) ?? []);
+    expect(await action(f.actionId)).toMatchObject({ status: "resolved", resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME });
+  }, 60_000);
+
+  it("settles a stale exhausted hold as recovered when a sibling action saved the run's files", async () => {
+    const f = await acceptedRemoteExport();
+    await recoverRetainedWorkspace(f.input);
+    expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
+    await db.update(issueRecoveryActions).set({ status: "resolved", resolvedAt: new Date() }).where(eq(issueRecoveryActions.id, f.actionId));
+    const staleId = randomUUID(), now = Date.now();
+    await db.insert(issueRecoveryActions).values({ id: staleId, companyId: f.companyId, sourceIssueId: f.issueId,
+      kind: "active_run_watchdog", status: "active", ownerType: "board", cause: "native_workspace_sync_out_retry_exhausted",
+      fingerprint: randomUUID(), nextAction: "Automatic recovery of saved work stopped after 6 attempts.",
+      evidence: { runId: f.runId,
+        systemRecovery: { schema: "paperclip.system-recovery.v1", policyVersion: 1, sourceRunId: f.runId, revision: 54,
+          generationId: randomUUID(), intentId: null, phase: "preserve_files", state: "exhausted", attemptId: randomUUID(),
+          attempts: 6, retriesUsed: 5, firstFailureAt: now - 60_000, nextAttemptAt: null, deadlineAt: now - 1, providerCooldownAt: null,
+          failure: { code: "export_interrupted", retryClass: "retry_transient", evidenceFingerprint: "b".repeat(64) },
+          preservationOperationIds: [], freshAttemptRunId: null },
+        workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: f.runId, status: "blocked", stage: "restoring_files",
+          attempt: 6, attemptLimit: 6, retryable: true, requestIds: [] } } });
+
+    await recoverRetainedWorkspace({ ...f.input, actionId: staleId });
+
+    const stale = await action(staleId);
+    expect(stale).toMatchObject({ status: "resolved", outcome: "restored", ownerType: "system",
+      resolutionNote: "workspace_recovered_by_sibling_action" });
+    expect(readWorkspaceRecovery(stale.evidence)?.status).toBe("recovered");
+    expect(stale.evidence.siblingWorkspaceRecovery).toMatchObject({ recoveryActionId: f.actionId });
+    expect(String(stale.nextAction)).not.toMatch(adminText);
+    const live = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, f.issueId),
+      inArray(issueRecoveryActions.status, ["active", "escalated"])));
+    expect(live).toHaveLength(0);
+  }, 60_000);
+
+  it("stops discovering a stale hold whose sibling receipt no longer verifies", async () => {
+    const f = await acceptedRemoteExport();
+    await recoverRetainedWorkspace(f.input);
+    await db.update(issueRecoveryActions).set({ status: "resolved", resolvedAt: new Date() }).where(eq(issueRecoveryActions.id, f.actionId));
+    // A later writer advanced the workspace: the sibling's receipt no longer verifies.
+    const ws = await workspace(f.workspaceId);
+    await db.update(executionWorkspaces).set({ metadata: { ...ws.metadata, remoteWorkspace: {
+      ...readRemoteWorkspaceAuthority(ws.metadata)!, dirty: true, writerRunId: randomUUID() } } }).where(eq(executionWorkspaces.id, f.workspaceId));
+    expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(false);
+    const staleId = randomUUID(), now = Date.now();
+    await db.insert(issueRecoveryActions).values({ id: staleId, companyId: f.companyId, sourceIssueId: f.issueId,
+      kind: "active_run_watchdog", status: "active", ownerType: "board", cause: "native_workspace_sync_out_retry_exhausted",
+      fingerprint: randomUUID(), nextAction: "Stale",
+      evidence: { runId: f.runId,
+        systemRecovery: { schema: "paperclip.system-recovery.v1", policyVersion: 1, sourceRunId: f.runId, revision: 3,
+          generationId: randomUUID(), intentId: null, phase: "preserve_files", state: "exhausted", attemptId: randomUUID(),
+          attempts: 6, retriesUsed: 5, firstFailureAt: now - 60_000, nextAttemptAt: null, deadlineAt: now - 1, providerCooldownAt: null,
+          failure: { code: "export_interrupted", retryClass: "retry_transient", evidenceFingerprint: "c".repeat(64) },
+          preservationOperationIds: [], freshAttemptRunId: null },
+        workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: f.runId, status: "blocked", stage: "restoring_files",
+          attempt: 6, attemptLimit: 6, retryable: true, requestIds: [] } } });
+    const ids = async () => (await retainedWorkspaceCandidates(db, undefined, false, true)).map(row => row.action.id);
+    expect(await ids()).toContain(staleId);
+    await recoverRetainedWorkspace({ ...f.input, actionId: staleId });
+    const stale = await action(staleId);
+    expect(stale.status).toBe("active");
+    expect(stale.evidence.retainedSourceFallbackCheck).toMatchObject({ reason: "sibling_receipt_unverified", nextCheckAt: null });
+    expect(await ids()).not.toContain(staleId);
+  }, 60_000);
+});

server/src/services/workspace-recovery-fallback.ts

@@ -0,0 +1,500 @@
+import { createHash, randomUUID } from "node:crypto";
+import { createReadStream } from "node:fs";
+import { mkdtemp, readdir, rm } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { and, eq, inArray, ne, or, sql } from "drizzle-orm";
+import {
+  agentWakeupRequests, assets, environmentLeases, environments, executionWorkspaces, executionWorkspaceSnapshots, heartbeatRuns,
+  issueAttachments, issueRecoveryActions, issues, nativeRunFinalizations, statusDecisionEffects, toolInvocations, workspaceOperations, type Db,
+} from "@paperclipai/db";
+import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type Environment, type EnvironmentLease, type ExecutionReconciliation } from "@paperclipai/shared";
+import { createTarballFromDirectory } from "@paperclipai/adapter-utils/sandbox-managed-runtime";
+import { logger } from "../middleware/logger.js";
+import { assertWorkspaceFilePathAllowed } from "./workspace-file-resources.js";
+import { getSecretProvider } from "../secrets/provider-registry.js";
+import { getStorageService } from "../storage/index.js";
+import { adapterExecutionControls } from "./adapter-execution-control.js";
+import { persistActivity, publishActivity } from "./activity-log.js";
+import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
+import type { EnvironmentRuntimeService } from "./environment-runtime.js";
+import { resolveEnvironmentExecutionTarget } from "./environment-execution-target.js";
+import { notifyRoutineExecutionStatus } from "./routine-execution-status-notifications.js";
+import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
+import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
+import { abandonUnverifiedRemoteWorkspaceEpoch } from "./remote-workspace-epoch-abandonment.js";
+import { hasStopOnlyCleanup, prepareHistoricalSandboxStopVerification, prepareSandboxStopAndRetain, restoreSandboxStopAfterFailedRestart, settleStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
+import { nativeWorkspaceFinalizationBlockSchema } from "./native-runtime/preserved-native-result-finalization.js";
+import { withNativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
+import { PERMANENT_SOURCE_RETRY_CLASSES, RETAINED_SOURCE_FALLBACK_CHECK_KEY, UNVERIFIABLE_SOURCE_CODES } from "./workspace-recovery-fallback-discovery.js";
+import { captureReadOnlyWorkspaceArchive, sandboxProviderFailureLabel } from "./remote-workspace-transport.js";
+import { restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js";
+import { readSystemRecovery, type SystemRecoveryState } from "./system-recovery-state.js";
+import { readWorkspaceRecovery, verifiedWorkspaceRepairReceipt } from "./workspace-recovery-receipt.js";
+
+/**
+ * Automatic fallback for a retained-file hold whose recorded source can never
+ * be verified (for example a reusable sandbox chain that crosses tasks). The
+ * strict source check stays in force: unverified files are never applied. The
+ * task continues from its last committed workspace save instead, and the
+ * unverified files are attached to the task when they can still be read.
+ */
+export const RETAINED_SOURCE_FALLBACK_OUTCOME = "retained_source_unverifiable_continued_from_last_saved" as const;
+export const RETAINED_SOURCE_FALLBACK_SCHEMA = "paperclip.retained-source-fallback.v1" as const;
+export const SIBLING_WORKSPACE_RECOVERY_SCHEMA = "paperclip.sibling-workspace-recovery.v1" as const;
+export const RETAINED_SOURCE_QUARANTINE_PHASE = "workspace_recovery_quarantine";
+
+const exportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
+const terminal = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
+// A check deferred for a transient reason runs again after this delay, a bounded number of times.
+const FALLBACK_RECHECK_MS = 5 * 60_000;
+const FALLBACK_MAX_CHECKS = 12;
+const outOfScopeReasons = ["not_unverifiable_source_hold", "not_remote_workspace_source"];
+const transientReasons = ["writer_active", "workspace_operation_running", "execution_not_stopped", "side_effect_pending",
+  "allocation_in_use", "ownership_busy", "fallback_failed", "source_changed"];
+const object = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
+
+type Source = { companyId: string; issueId: string; runId: string; actionId: string };
+type Ownership = { token: string; assertHeld(): Promise<void> };
+type Quarantine = NonNullable<ExecutionReconciliation["retainedSourceFallback"]>["quarantine"];
+export type RetainedSourceFallbackResult = { kind: "continued" | "settled" } | { kind: "ineligible"; reason: string };
+
+export function isUnverifiableSourceFailure(failure: SystemRecoveryState["failure"] | null | undefined) {
+  return Boolean(failure && (UNVERIFIABLE_SOURCE_CODES as readonly string[]).includes(failure.code)
+    && (PERMANENT_SOURCE_RETRY_CLASSES as readonly string[]).includes(failure.retryClass));
+}
+
+/** Text the agent receives with its continuation. It names no person who must act. */
+export function retainedSourceFallbackNote(input: { runId: string; snapshotId: string | null; quarantine: Quarantine }) {
+  const from = input.snapshotId ? `the last saved version of the workspace (snapshot ${input.snapshotId})` : "the workspace's original source";
+  const files = input.quarantine.state === "captured"
+    ? `They are attached to this task as ${input.quarantine.filename} (attachment ${input.quarantine.attachmentId}), without secrets, repository metadata or dependency directories. Inspect them and re-apply anything still needed deliberately.`
+    : "They could not be read, so they are not available.";
+  return `Paperclip could not verify the unsaved file changes of run ${input.runId}, so they were not applied. `
+    + `This task continues from ${from}. ${files} Check which work from that run needs redoing. `
+    + "Its completed external actions remain recorded and must not be repeated.";
+}
+
+async function uncertainExternalAction(db: Db, run: typeof heartbeatRuns.$inferSelect) {
+  for (const receipt of Object.values(object(run.resultJson?.apiToolReceipts)).map(object)) {
+    let result = object(receipt.result);
+    if (receipt.encryptedResult) {
+      try {
+        result = object(JSON.parse(await getSecretProvider("local_encrypted").resolveVersion({ material: object(receipt.encryptedResult), externalRef: null })));
+      } catch { return true; }
+    }
+    if (receipt.state !== "completed" || result.outcome === "unknown" || result.error === "api_outcome_unknown") return true;
+    // A definite client rejection had no effect. Anything else that is not ok is unknown.
+    const definiteRejection = result.ok === false && typeof result.status === "number" && result.status >= 400 && result.status < 500 && result.status !== 408;
+    if (result.ok !== true && !definiteRejection) return true;
+  }
+  // Semantic control-plane receipts are written in the same transaction as
+  // their effect, so they are never uncertain. Upstream tool calls can be.
+  const [invocation] = await db.select({ id: toolInvocations.id }).from(toolInvocations).where(and(
+    eq(toolInvocations.companyId, run.companyId), eq(toolInvocations.runId, run.id),
+    sql`${toolInvocations.riskLevel} is distinct from 'read'`,
+    inArray(toolInvocations.status, ["executing", "failed", "timed_out"]),
+    sql`${toolInvocations.startedAt} is not null`)).limit(1);
+  return Boolean(invocation);
+}
+
+/** Every fact the fallback depends on, read in the existing lock order:
+ * task, coordinator, run, recovery action, then workspace. */
+async function inspectFallback(db: Db, input: Source, lock: boolean, requireStopped = lock) {
+  const one = async <T>(query: PromiseLike<T[]> & { for(mode: "update"): PromiseLike<T[]> }): Promise<T | null> =>
+    (await (lock ? query.for("update") : query))[0] ?? null;
+  const refuse = (reason: string) => ({ kind: "ineligible" as const, reason });
+  const issue = await one(db.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).limit(1));
+  const coordinator = await one(db.select().from(nativeRunFinalizations).where(and(
+    eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId))).limit(1));
+  const run = await one(db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))).limit(1));
+  const action = await one(db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId),
+    eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId))).limit(1));
+  if (!issue || !run || !action) return refuse("source_changed");
+  if (object(action.evidence.retainedSourceFallback).schema === RETAINED_SOURCE_FALLBACK_SCHEMA) return { kind: "settled" as const };
+  const system = readSystemRecovery(action.evidence);
+  if (!exportCauses.includes(action.cause) || !["active", "escalated"].includes(action.status) || action.evidence.runId !== run.id
+    || action.evidence.executionReconciliation || system?.state !== "exhausted" || system.sourceRunId !== run.id
+    || !isUnverifiableSourceFailure(system.failure)) return refuse("not_unverifiable_source_hold");
+  if (run.runtimeMode !== "native" || !terminal.includes(run.status) || !run.finishedAt
+    || (run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) !== issue.id) return refuse("source_run_not_terminal");
+  // No second executor: the source run, its controller and its finalizer have stopped.
+  if (adapterExecutionControls.has(run.id) || (run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date())
+    || coordinator?.leaseOwner || coordinator?.failureDetail?.successorRunId) return refuse("execution_not_stopped");
+  const parsed = remoteWorkspaceRunReferenceSchema.safeParse(run.runnerProfileJson?.remoteWorkspace);
+  if (!parsed.success || parsed.data.state !== "prepared") return refuse("not_remote_workspace_source");
+  const reference = parsed.data;
+  const workspace = await one(db.select().from(executionWorkspaces).where(and(
+    eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, reference.workspaceId))).limit(1));
+  let authority;
+  try { authority = readRemoteWorkspaceAuthority(workspace?.metadata); } catch { return refuse("workspace_authority_invalid"); }
+  // Only the exact epoch this run wrote is abandoned; a newer writer is never touched.
+  if (!workspace || !authority?.location || authority.writerRunId !== run.id || authority.epoch !== reference.epoch
+    || authority.snapshotId !== reference.snapshotId || authority.location.leaseId !== reference.leaseId
+    || authority.location.providerLeaseId !== reference.providerLeaseId || authority.location.remoteCwd !== reference.remoteCwd) {
+    return refuse("workspace_authority_changed");
+  }
+  if (reference.snapshotId) {
+    const [snapshot] = await db.select({ id: executionWorkspaceSnapshots.id }).from(executionWorkspaceSnapshots).where(and(
+      eq(executionWorkspaceSnapshots.companyId, input.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id),
+      eq(executionWorkspaceSnapshots.id, reference.snapshotId), eq(executionWorkspaceSnapshots.status, "committed"))).limit(1);
+    if (!snapshot) return refuse("last_save_missing");
+  }
+  // A live writer keeps its files. A queued turn of this task that never
+  // started cannot write: this hold gates its dispatch.
+  const [writer] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
+    eq(heartbeatRuns.companyId, input.companyId), ne(heartbeatRuns.id, run.id), inArray(heartbeatRuns.status, ["queued", "running"]),
+    sql`not coalesce(${heartbeatRuns.status} = 'queued' and ${heartbeatRuns.startedAt} is null
+      and ${heartbeatRuns.processPid} is null and ${heartbeatRuns.processGroupId} is null, false)`,
+    or(eq(heartbeatRuns.nativeIssueId, issue.id), eq(heartbeatRuns.issueId, issue.id),
+      sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`,
+      sql`${heartbeatRuns.contextSnapshot}->>'executionWorkspaceId' = ${workspace.id}`))).limit(1);
+  if (writer) return refuse("writer_active");
+  // No second executor: every allocation of the source run has a provider stop receipt.
+  if (requireStopped) {
+    const leases = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
+      eq(environmentLeases.heartbeatRunId, run.id)));
+    if (!leases.some(lease => lease.id === reference.leaseId)
+      || leases.some(lease => !hasRemoteTerminationReceipt(lease) && !hasRecoveredCleanupStopReceipt(lease))) return refuse("execution_not_stopped");
+  }
+  const [operation] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
+    eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id),
+    eq(workspaceOperations.status, "running"))).limit(1);
+  if (operation) return refuse("workspace_operation_running");
+  // Pending and uncertain effects keep the existing reconciliation path.
+  const [effect] = coordinator?.decisionId ? await db.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and(
+    eq(statusDecisionEffects.companyId, input.companyId), eq(statusDecisionEffects.issueId, issue.id),
+    eq(statusDecisionEffects.decisionId, coordinator.decisionId), inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]))).limit(1) : [];
+  if (effect) return refuse("side_effect_pending");
+  if (await uncertainExternalAction(db, run)) return refuse("external_action_uncertain");
+  return { kind: "eligible" as const, issue, coordinator, run, action, system, reference, workspace, authority };
+}
+
+/** The attachment follows the workspace file policy: secrets (such as `.env`
+ * and keys), repository metadata and dependency or build directories are omitted. */
+async function omitDeniedWorkspaceFiles(root: string, relative = "") {
+  for (const entry of await readdir(path.join(root, relative), { withFileTypes: true })) {
+    const child = relative ? `${relative}/${entry.name}` : entry.name;
+    let allowed = true;
+    try { assertWorkspaceFilePathAllowed(child); } catch { allowed = false; }
+    if (!allowed) await rm(path.join(root, child), { recursive: true, force: true });
+    else if (entry.isDirectory() && !entry.isSymbolicLink()) await omitDeniedWorkspaceFiles(root, child);
+  }
+}
+
+function unreadable(reason: string): Quarantine {
+  return { state: "unreadable", reason: /^[a-z][a-z0-9_]{0,47}$/.test(reason) ? reason : "operation_failed" };
+}
+
+/** Reads the retained root without writing to the sandbox and attaches it to
+ * the task as an unverified archive. Never applied, never a recovery receipt.
+ * A root that cannot be read is recorded as such. Storage and database
+ * failures after a successful read throw, so the fallback retries later. */
+async function quarantineRetainedRoot(db: Db, input: Source & { environmentRuntime: EnvironmentRuntimeService; ownership: Ownership },
+  source: Extract<Awaited<ReturnType<typeof inspectFallback>>, { kind: "eligible" }>): Promise<Quarantine> {
+  const [existing] = await db.select({ metadata: workspaceOperations.metadata }).from(workspaceOperations).where(and(
+    eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.heartbeatRunId, input.runId),
+    eq(workspaceOperations.phase, RETAINED_SOURCE_QUARANTINE_PHASE), eq(workspaceOperations.status, "succeeded"),
+    sql`${workspaceOperations.metadata}->>'recoveryActionId' = ${input.actionId}`)).limit(1);
+  const saved = object(existing?.metadata?.quarantine);
+  if (saved.state === "captured") return saved as Quarantine;
+  const { reference } = source;
+  const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
+    eq(environmentLeases.id, reference.leaseId), eq(environmentLeases.heartbeatRunId, input.runId)));
+  if (!lease?.environmentId || lease.providerLeaseId !== reference.providerLeaseId) return unreadable("allocation_missing");
+  if (hasRemoteTerminationReceipt(lease) && object(lease.metadata?.remoteExecutionTermination).state === "destroyed")
+    return unreadable("sandbox_destroyed");
+  const [environment] = await db.select().from(environments).where(eq(environments.id, lease.environmentId));
+  if (!environment) return unreadable("environment_missing");
+  const scratch = await mkdtemp(path.join(os.tmpdir(), "paperclip-quarantine-"));
+  try {
+    const tarball = path.join(scratch, "unverified.tar");
+    try {
+      const target = await resolveEnvironmentExecutionTarget({ db, companyId: input.companyId, adapterType: "paperclip_runner",
+        environment, leaseId: lease.id, leaseMetadata: lease.metadata, lease: lease as EnvironmentLease, environmentRuntime: input.environmentRuntime });
+      if (target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner) return unreadable("target_unavailable");
+      await input.ownership.assertHeld();
+      const captured = await captureReadOnlyWorkspaceArchive({ target, remoteCwd: reference.remoteCwd, operationId: randomUUID() });
+      const files = path.join(scratch, "files");
+      try {
+        await restoreRemoteWorkspaceArchive({ archivePath: captured.archivePath, targetDir: files, expectedManifestDigest: captured.manifestDigest });
+      } finally { await captured.cleanup(); }
+      await omitDeniedWorkspaceFiles(files);
+      await createTarballFromDirectory({ localDir: files, archivePath: tarball });
+    } catch (error) {
+      const message = error instanceof Error ? error.message : "";
+      const reason = /^remote_workspace_readonly_capture_failed: .*reason=([a-z_]+)/.exec(message)?.[1];
+      logger.warn({ err: error, runId: input.runId }, "Unverified retained files could not be read; continuing from the last save");
+      return unreadable(reason ?? sandboxProviderFailureLabel(error).label);
+    }
+    const hash = createHash("sha256");
+    let byteSize = 0;
+    for await (const chunk of createReadStream(tarball)) { hash.update(chunk as Buffer); byteSize += (chunk as Buffer).length; }
+    const sha256 = hash.digest("hex");
+    const filename = `unverified-files-run-${input.runId}.tar`;
+    await input.ownership.assertHeld();
+    const stored = await getStorageService().putFile({ companyId: input.companyId, namespace: "workspace-recovery-quarantine",
+      objectKey: `${input.companyId}/workspace-recovery-quarantine/${input.runId}/${sha256}.tar`, originalFilename: filename,
+      contentType: "application/x-tar", body: createReadStream(tarball), byteSize, sha256 });
+    await input.ownership.assertHeld();
+    return await db.transaction(async tx => {
+      const [asset] = await tx.insert(assets).values({ companyId: input.companyId, provider: stored.provider, objectKey: stored.objectKey,
+        contentType: "application/x-tar", byteSize, sha256, originalFilename: filename }).returning();
+      const [attachment] = await tx.insert(issueAttachments).values({ companyId: input.companyId, issueId: input.issueId,
+        assetId: asset!.id }).returning();
+      const quarantine: Quarantine = { state: "captured", attachmentId: attachment!.id, filename, sha256, byteSize };
+      await tx.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId, heartbeatRunId: input.runId,
+        executionWorkspaceId: source.workspace.id, phase: RETAINED_SOURCE_QUARANTINE_PHASE, status: "succeeded", finishedAt: new Date(),
+        metadata: { schema: RETAINED_SOURCE_FALLBACK_SCHEMA, recoveryActionId: input.actionId, verified: false, applied: false,
+          leaseId: lease.id, providerLeaseId: lease.providerLeaseId, remoteCwd: reference.remoteCwd, epoch: reference.epoch, quarantine } });
+      return quarantine;
+    });
+  } finally { await rm(scratch, { recursive: true, force: true }).catch(() => undefined); }
+}
+
+async function sandboxHeldByAnotherLease(db: Db, lease: Pick<typeof environmentLeases.$inferSelect, "id" | "provider" | "providerLeaseId">) {
+  if (!lease.provider || !lease.providerLeaseId) return false;
+  const [other] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
+    ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider), eq(environmentLeases.providerLeaseId, lease.providerLeaseId),
+    inArray(environmentLeases.status, ["active", "pending_cleanup"]))).limit(1);
+  return Boolean(other);
+}
+
+/** Obtains the provider stop receipt for the source allocation through the
+ * same stop-only cleanup as normal recovery. A sandbox that another lease holds
+ * is never stopped from here. */
+async function confirmSourceAllocationStopped(db: Db, input: Source & { environmentRuntime: EnvironmentRuntimeService; ownership: Ownership },
+  leaseId: string): Promise<"stopped" | "in_use" | "unconfirmed"> {
+  const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
+    eq(environmentLeases.id, leaseId), eq(environmentLeases.heartbeatRunId, input.runId)));
+  if (!lease?.environmentId || !lease.provider || !lease.providerLeaseId) return "unconfirmed";
+  if (hasRemoteTerminationReceipt(lease) || hasRecoveredCleanupStopReceipt(lease)) return "stopped";
+  if (await sandboxHeldByAnotherLease(db, lease)) return "in_use";
+  const [environment] = await db.select().from(environments).where(eq(environments.id, lease.environmentId));
+  if (!environment) return "unconfirmed";
+  await input.ownership.assertHeld();
+  const pending = await db.transaction(async tx => {
+    const [fresh] = await tx.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
+      eq(environmentLeases.id, lease.id))).for("update");
+    if (!fresh || fresh.metadata?.pendingCleanupInFlight === true) return null;
+    const prepared = hasStopOnlyCleanup(fresh) ? fresh : fresh.status === "released"
+      ? await prepareHistoricalSandboxStopVerification(tx as unknown as Db, fresh)
+      : await prepareSandboxStopAndRetain(tx as unknown as Db, fresh);
+    if (!prepared) return null;
+    const [claimed] = await tx.update(environmentLeases).set({ metadata: sql`${environmentLeases.metadata} ||
+      ${JSON.stringify({ pendingCleanupAttemptId: input.ownership.token, pendingCleanupInFlight: true,
+        pendingCleanupLeaseExpiresAtMs: Date.now() + 15 * 60_000 })}::jsonb` }).where(eq(environmentLeases.id, lease.id)).returning();
+    return claimed ?? null;
+  });
+  if (!pending) return "unconfirmed";
+  const attemptId = input.ownership.token;
+  const settled = await input.environmentRuntime.retryPendingSandboxTeardown({ environment: environment as unknown as Environment,
+    lease: pending as EnvironmentLease })
+    .then(receipt => settleStopOnlyCleanup(db, pending, { attemptId, receipt }),
+      async () => await restoreSandboxStopAfterFailedRestart(db, pending, { attemptId }) ?? await settleStopOnlyCleanup(db, pending, { attemptId }));
+  return settled && hasRemoteTerminationReceipt(settled) ? "stopped" : "unconfirmed";
+}
+
+/**
+ * Settles a retained-file hold whose source can never be verified. In one
+ * transaction it abandons the unverified workspace epoch, resolves the hold
+ * with a named outcome, and hands the task to its normal continuation from the
+ * last committed save. Waiting user input continues the task itself; otherwise
+ * the recovery delivery starts one continuation. Bounded and idempotent: a
+ * settled hold is never revisited, and a lost race leaves the hold unchanged.
+ */
+export async function continueFromLastSavedWorkspace(input: Source & {
+  db: Db; environmentRuntime: EnvironmentRuntimeService; ownership: Ownership;
+}): Promise<RetainedSourceFallbackResult> {
+  const { db } = input;
+  const initial = await inspectFallback(db, input, false);
+  if (initial.kind !== "eligible") return initial;
+  // An unstopped source in a sandbox another lease holds waits; nothing is read or stopped.
+  const [sourceLease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
+    eq(environmentLeases.id, initial.reference.leaseId), eq(environmentLeases.heartbeatRunId, input.runId)));
+  if (sourceLease && !hasRemoteTerminationReceipt(sourceLease) && !hasRecoveredCleanupStopReceipt(sourceLease)
+    && await sandboxHeldByAnotherLease(db, sourceLease)) return { kind: "ineligible", reason: "allocation_in_use" };
+  // Read before stopping: a stopped sandbox cannot be read without resuming it.
+  const quarantine = await quarantineRetainedRoot(db, input, initial);
+  const stop = await confirmSourceAllocationStopped(db, input, initial.reference.leaseId);
+  if (stop !== "stopped") return { kind: "ineligible", reason: stop === "in_use" ? "allocation_in_use" : "execution_not_stopped" };
+  await input.ownership.assertHeld();
+  const outcome = await db.transaction(async tx => {
+    const current = await inspectFallback(tx as unknown as Db, input, true);
+    if (current.kind !== "eligible") return { result: current, publication: null };
+    if (current.system.generationId !== initial.system.generationId || current.system.revision !== initial.system.revision)
+      return { result: { kind: "ineligible" as const, reason: "source_changed" }, publication: null };
+    const { issue, run, action, reference } = current;
+    const abandoned = await abandonUnverifiedRemoteWorkspaceEpoch(tx, { companyId: input.companyId, workspaceId: reference.workspaceId,
+      runId: run.id, epoch: reference.epoch, snapshotId: reference.snapshotId, location: current.authority.location! });
+    const [waitingInput] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
+      eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
+      eq(agentWakeupRequests.requestedByActorType, "user"), sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`)).limit(1);
+    const fallback: NonNullable<ExecutionReconciliation["retainedSourceFallback"]> = { schema: RETAINED_SOURCE_FALLBACK_SCHEMA,
+      outcome: RETAINED_SOURCE_FALLBACK_OUTCOME, workspaceId: reference.workspaceId, abandonedEpoch: reference.epoch,
+      epoch: abandoned.authority.epoch, snapshotId: reference.snapshotId, quarantine };
+    const decision: ExecutionReconciliation = { runId: run.id, providerStopped: true, actionOutcome: "completed",
+      outcomeEvidence: retainedSourceFallbackNote({ runId: run.id, snapshotId: reference.snapshotId, quarantine }),
+      // The workspace repair that a restore failure requires before continuing:
+      // the unverified epoch is abandoned and the next writer restores the last
+      // committed save into a new root.
+      workspaceRepairEvidence: `Paperclip abandoned unverified workspace epoch ${reference.epoch} of run ${run.id}; `
+        + `the continuation restores ${reference.snapshotId ? `committed snapshot ${reference.snapshotId}` : "the workspace source"} into a new root.`,
+      retainedSourceFallback: fallback };
+    const now = new Date();
+    // A Blocked status belongs to this hold only while its own receipt (or, for
+    // holds saved before receipts existed, a block set by this run's terminal
+    // failure that nothing changed since) still names it. Otherwise a newer
+    // decision owns the block: it is kept and no continuation is started.
+    const block = nativeWorkspaceFinalizationBlockSchema.safeParse(action.evidence.workspaceFinalizationBlock);
+    const blockedAt = issue.blockedTransitionAt?.getTime();
+    const ownsBlock = issue.status === "blocked" && (block.success
+      ? block.data.issueId === issue.id && block.data.runId === run.id
+        && block.data.statusVersion === Number(issue.statusVersion) && block.data.priorDecisionId === issue.lastStatusDecisionId
+      : action.evidence.workspaceFinalizationBlock === undefined && issue.lastStatusDecisionId === null && blockedAt !== undefined
+        && blockedAt >= run.finishedAt!.getTime() - 1_000 && blockedAt <= action.createdAt.getTime() + 60_000);
+    const dispatchable = issue.status !== "blocked" || ownsBlock;
+    const userText = quarantine.state === "captured"
+      ? "Paperclip could not verify the previous run's unsaved files, so the task continues from the last saved version. Those files are attached to the task, unapplied."
+      : "Paperclip could not verify the previous run's unsaved files, so the task continues from the last saved version. Those files could not be read.";
+    const evidence = { ...action.evidence, automaticRecovery: undefined, [RETAINED_SOURCE_FALLBACK_CHECK_KEY]: undefined,
+      retainedSourceFallback: { ...fallback, failureCode: current.system.failure?.code ?? null, recordedAt: now.toISOString() },
+      workspaceRecovery: { ...(readWorkspaceRecovery(action.evidence) ?? {}), message: userText, retryable: false },
+      executionReconciliation: { ...decision, actorId: "workspace-recovery", recordedAt: now.toISOString() },
+      // Saved user input continues the task through normal admission.
+      continuationDelivery: waitingInput || !dispatchable ? "not_requested" : "pending" };
+    await tx.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: now, updatedAt: now,
+      ownerType: "system", ownerAgentId: null, ownerUserId: null, returnOwnerAgentId: action.returnOwnerAgentId ?? run.agentId,
+      resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME, nextAction: userText, wakePolicy: null, monitorPolicy: null, evidence,
+    }).where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, action.id)));
+    // The abandoned source must never be replayed or replaced automatically.
+    await tx.update(nativeRunFinalizations).set({ failureDetail: sql`coalesce(${nativeRunFinalizations.failureDetail}, '{}'::jsonb)
+      || ${JSON.stringify({ replacementDenied: RETAINED_SOURCE_FALLBACK_OUTCOME })}::jsonb` }).where(and(
+      eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, run.id)));
+    if (ownsBlock) {
+      await tx.update(issues).set({ status: "todo", unblockDescriptor: null, blockedTransitionAt: null,
+        blockedOwnerNotifiedAt: null, updatedAt: now }).where(eq(issues.id, issue.id));
+    }
+    await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionContinuation);
+    await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
+    await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
+    await notifyStatusCardIssueChange(tx, input.companyId);
+    await notifyRoutineExecutionStatus(tx, issue.id, input.companyId);
+    const publication = await persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system",
+      actorId: "workspace-recovery", action: "issue.workspace_recovery_continued_from_last_saved", entityType: "issue",
+      entityId: issue.id, runId: run.id, details: { recoveryActionId: action.id, failureCode: current.system.failure?.code ?? null,
+        ...fallback } });
+    return { result: { kind: "continued" as const }, publication };
+  });
+  if (outcome.publication) {
+    try { publishActivity(outcome.publication.publication); } catch { logger.warn({ runId: input.runId }, "Recovery activity delivery deferred"); }
+  }
+  return outcome.result;
+}
+
+/** Records why the fallback could not settle this generation yet. A transient
+ * reason is checked again after a delay, a bounded number of times; a
+ * permanent one waits for new input, Retry or a new recovery generation. */
+export async function recordRetainedSourceFallbackDeferral(db: Db, input: Source, reason: string) {
+  const [row] = await db.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
+    eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId),
+    eq(issueRecoveryActions.id, input.actionId)));
+  const system = readSystemRecovery(row?.evidence);
+  // A hold outside the fallback's scope keeps its evidence unchanged.
+  if (!row || !system || row.evidence.retainedSourceFallback || outOfScopeReasons.includes(reason)) return;
+  const prior = object(row.evidence[RETAINED_SOURCE_FALLBACK_CHECK_KEY]);
+  const checks = prior.generationId === system.generationId && typeof prior.checks === "number" ? prior.checks + 1 : 1;
+  const transient = transientReasons.includes(reason) && checks < FALLBACK_MAX_CHECKS;
+  const marker = { generationId: system.generationId, reason, checks, checkedAt: Date.now(),
+    nextCheckAt: transient ? Date.now() + FALLBACK_RECHECK_MS : null };
+  await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} ||
+    jsonb_build_object(${RETAINED_SOURCE_FALLBACK_CHECK_KEY}::text, ${JSON.stringify(marker)}::jsonb)` }).where(and(
+    eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, input.actionId),
+    sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${system.generationId}`,
+    sql`${issueRecoveryActions.evidence}->'retainedSourceFallback' is null`));
+}
+
+/**
+ * The recovery worker's entry for an exhausted hold: settle it when a sibling
+ * saved its files, or continue from the last save when its source is refused
+ * permanently. Runs under the workspace ownership lock, like any physical
+ * recovery attempt. Returns whether the hold is settled.
+ */
+export async function settleExhaustedRetainedHold(input: Source & { db: Db; environmentRuntime: EnvironmentRuntimeService },
+  state: SystemRecoveryState, options: { fallback: boolean } = { fallback: true }) {
+  const { db } = input;
+  const sibling = await settleSiblingRecoveredHold(db, input);
+  if (sibling === "settled") return true;
+  if (!options.fallback || !isUnverifiableSourceFailure(state.failure)) {
+    // A sibling marked recovered without a receipt that still verifies cannot
+    // settle this hold. Record it so discovery does not select it again.
+    if (sibling === "unverified") await recordRetainedSourceFallbackDeferral(db, input, "sibling_receipt_unverified");
+    return false;
+  }
+  let reason: string;
+  try {
+    const owned = await withNativeWorkspaceFinalizationOwnership({ db, companyId: input.companyId, runId: input.runId,
+      retainedRepairActionId: input.actionId }, ownership => continueFromLastSavedWorkspace({ ...input, ownership }));
+    if (owned.acquired && owned.value.kind !== "ineligible") return true;
+    reason = !owned.acquired ? "ownership_busy" : owned.value.kind === "ineligible" ? owned.value.reason : "fallback_failed";
+  } catch (error) {
+    logger.warn({ err: error, runId: input.runId }, "Continuing from the last save was deferred; the hold remains");
+    reason = "fallback_failed";
+  }
+  logger.info({ runId: input.runId, reason }, "Retained source fallback deferred");
+  await recordRetainedSourceFallbackDeferral(db, input, reason);
+  return false;
+}
+
+/**
+ * An exhausted hold for a run whose files another recovery action already
+ * saved (a verified receipt bound to that sibling) has nothing left to do.
+ * Settle it as recovered instead of handing a spent budget to a person.
+ */
+export async function settleSiblingRecoveredHold(db: Db, input: Source) {
+  const settled = await db.transaction(async tx => {
+    const [issue] = await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, input.companyId),
+      eq(issues.id, input.issueId))).for("update");
+    const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId),
+      eq(heartbeatRuns.id, input.runId))).for("update");
+    const [action] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId),
+      eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId))).for("update");
+    if (!issue || !run || !action || action.evidence.runId !== run.id) return null;
+    if (object(action.evidence.siblingWorkspaceRecovery).schema === SIBLING_WORKSPACE_RECOVERY_SCHEMA) return "already" as const;
+    const system = readSystemRecovery(action.evidence);
+    if (!["active", "escalated"].includes(action.status) || system?.state !== "exhausted" || system.sourceRunId !== run.id) return null;
+    const receiptId = await verifiedWorkspaceRepairReceipt(tx as unknown as Db, run);
+    const [receipt] = receiptId ? await tx.select({ metadata: workspaceOperations.metadata }).from(workspaceOperations).where(and(
+      eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.id, receiptId))) : [];
+    const siblingActionId = receipt?.metadata?.recoveryActionId;
+    if (!receiptId || typeof siblingActionId !== "string" || siblingActionId === action.id) {
+      const [recoveredSibling] = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
+        eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId),
+        ne(issueRecoveryActions.id, action.id), sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
+        sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete'`,
+        sql`${issueRecoveryActions.evidence}#>>'{workspaceRecovery,status}' = 'recovered'`)).limit(1);
+      return recoveredSibling ? "unverified" as const : null;
+    }
+    const now = new Date();
+    const prior = readWorkspaceRecovery(action.evidence);
+    await tx.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: now, updatedAt: now,
+      ownerType: "system", ownerAgentId: null, ownerUserId: null, nextAction: "Saved work recovered.",
+      resolutionNote: "workspace_recovered_by_sibling_action", wakePolicy: null, monitorPolicy: null,
+      evidence: { ...action.evidence,
+        workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: run.id, status: "recovered", stage: "saving_receipts",
+          attempt: prior?.attempt ?? system.attempts, attemptLimit: prior?.attemptLimit ?? 6, requestIds: prior?.requestIds ?? [], retryable: false },
+        siblingWorkspaceRecovery: { schema: SIBLING_WORKSPACE_RECOVERY_SCHEMA, receiptId, recoveryActionId: siblingActionId,
+          recordedAt: now.toISOString() } },
+    }).where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, action.id)));
+    await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionContinuation);
+    return persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system", actorId: "workspace-recovery",
+      action: "issue.workspace_recovered", entityType: "issue", entityId: issue.id, runId: run.id,
+      details: { recoveryActionId: action.id, recoveredByRecoveryActionId: siblingActionId, receiptId,
+        summary: "Saved work recovered by another recovery of the same run." } });
+  });
+  if (settled === "unverified") return "unverified" as const;
+  if (settled && settled !== "already") {
+    try { publishActivity(settled.publication); } catch { logger.warn({ runId: input.runId }, "Recovery activity delivery deferred"); }
+  }
+  return settled ? "settled" as const : "none" as const;
+}

server/src/services/workspace-recovery.ts

@@ -35,6 +35,8 @@ import { preserveWorkspaceSnapshot } from "./workspace-recovery-snapshot.js";
 import { adoptHistoricalWorkspaceRecoveries } from "./workspace-recovery-backfill.js";
 import { canAdoptNativeSourceRetention, retainNativeRecoverySource, settleRecoveredNativeSourceRetention } from "./native-runtime/native-source-retention.js";
 import { saveRemoteWorkspace } from "./remote-workspaces.js";
+import { continueFromLastSavedWorkspace, isUnverifiableSourceFailure, recordRetainedSourceFallbackDeferral, settleExhaustedRetainedHold } from "./workspace-recovery-fallback.js";
+import { retainedSourceFallbackDueCondition } from "./workspace-recovery-fallback-discovery.js";
 
 const exportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
 const terminal = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"];
@@ -160,6 +162,12 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
   const { db } = input;
   await adoptSystemRecovery(db, input);
   const initial = await loadSystemRecovery(db, input);
+  // A hold whose files a sibling saved, or whose source is refused permanently
+  // (including holds saved by older releases), settles without a person.
+  // An adoptable native source keeps its retention path; only the sibling
+  // settlement applies to it.
+  if (initial?.state === "exhausted"
+    && await settleExhaustedRetainedHold(input, initial, { fallback: !canAdoptNativeSourceRetention(initial) })) return;
   if (initial?.state === "exhausted" && !canAdoptNativeSourceRetention(initial)) {
     // A budget spent before this hold existed still names its receipt, so the
     // board confirmation in its handoff can release it (RC-50 wave databases).
@@ -430,10 +438,20 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
       const generic = !(error instanceof SystemRecoveryFailure) && failure.retryClass === "retry_transient"
         && (failure.code === "provider_unavailable" || failure.code === "export_interrupted");
       const cause = generic ? sandboxProviderFailureLabel(error) : undefined;
-      await updateSystemRecovery(db, input, current!, failSystemRecovery(current!, failure, {
+      const failed = failSystemRecovery(current!, failure, {
         ...(typeof retryAfterAt === "number" && Number.isFinite(retryAfterAt) ? { retryAfterAt } : {}),
         ...(cause ? { cause } : {}),
-      }));
+      });
+      const recorded = await updateSystemRecovery(db, input, current!, failed);
+      // A permanent refusal of the recorded source repeats on every retry.
+      // Continue from the last save in this same pass, under this ownership.
+      if (recorded && failed.state === "exhausted" && isUnverifiableSourceFailure(failed.failure)) {
+        const fallback = await continueFromLastSavedWorkspace({ ...input, ownership }).catch(fallbackError => {
+          logger.warn({ err: fallbackError, runId: input.runId }, "Continuing from the last save was deferred; the hold remains");
+          return { kind: "ineligible" as const, reason: "fallback_failed" };
+        });
+        if (fallback.kind === "ineligible") await recordRetainedSourceFallbackDeferral(db, input, fallback.reason);
+      }
       if (failure.retryClass === "recheck_evidence" && ["agent_stop_unconfirmed", "copyback_stop_unconfirmed"].includes(failure.code))
         await recheckStoppedWorkspaceRecovery(db, input, false);
       logger.warn({ err: error, runId: input.runId, failureCode: failure.code, failureCause: cause?.label, phase: current!.phase }, "Workspace preservation deferred; source retained");
@@ -556,7 +574,8 @@ export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedO
           and ${issueRecoveryActions.evidence}->>'bootstrapPreparationSettledAt' is not null
           and coalesce(${issueRecoveryActions.evidence}->>'bootstrapPreparationContinuation', 'pending') = 'pending'))` : undefined,
       obligationsOnly ? sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')
-        or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete' and (${cleanup} or ${interruptedPreparation})) or ${waitingInput})` : undefined,
+        or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete' and (${cleanup} or ${interruptedPreparation})) or ${waitingInput}
+        or ${retainedSourceFallbackDueCondition(Date.now())})` : undefined,
       sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence', 'complete', 'exhausted')
         and (${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' = 'null'::jsonb
           or ${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' <= to_jsonb(${Date.now()}::bigint)))`,