fix(recovery): treat a retired prepared remote epoch as nothing to recover (#20491)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A restart must never leave a task stuck: an interrupted run must
resume on its own or have a working Retry
> - When a graceful restart interrupts a remote-workspace run that is
still preparing, the run keeps a `prepared` reference to its workspace
epoch, and a preserve-without-replay hold blocks the task
> - The unsaved-files reconciler then retires that epoch with
`abandonUnverifiedRemoteWorkspaceEpoch` and writes a
`paperclip.remote-workspace-abandoned-epoch.v1` receipt. The workspace
content is the last committed save again
> - Recovery never read that receipt. The held-preparation reconciler
waited for a repair receipt that could never exist, and restore recovery
refused the source as `source_binding_invalid`. The task stayed blocked,
and a UI Retry returned 202 and started nothing
> - This pull request adds one strict shared rule: an abandoned-epoch
receipt for the run's exact prepared epoch and allocation means there is
nothing to recover. Every recovery path now uses it
> - The benefit is that the task resumes automatically from the last
committed save after the restart, whichever recovery path owns the hold

## Linked Issues or Issue Description

Refs #20072 (replay a preparation a restart interrupted), #20299
(storage outage waits).

**What existing behavior does this improve?**
Recovery of a remote-workspace run that a graceful restart interrupted
during preparation, after its unsaved epoch was retired.

**Subsystem affected**
Server: recovery (held-preparation reconciler, workspace restore
recovery, the workspace repair check).

**Current behavior**
These are Daytona R1 restart-chaos waves (Paperclip-sized workspace,
`--storage s3`, K2 graceful restart at +110 s, during the ~730 MiB first
save):
- `w-20261110T052533-s8805` and `w-20261110T062703-s8805` (ST2 slow
store), held-preparation path: both runs ended `interrupted` in
`executionStage: preparing`. The reference was `{ state: "prepared",
epoch: 1 }`, and the workspace had moved to `{ epoch: 2, location: null,
dirty: false, writerRunId: null }` with an abandoned-epoch receipt for
epoch 1. The `legacy_interrupted_preparation_hold` reconciler never
settled the hold (`I2_settled` violations), because
`isInterruptedNativePreparation` requires `hasVerifiedWorkspaceRepair`,
and that check accepts only a `saved` reference. Grader:
`R3:retry_no_new_run`.
- `w-20261110T091639-s11936` (ST1 outage), restore-recovery path: the
run also had `workspaceRestoreFailure: "restore_failed"` and
`workspaceRestoreRecovery`. Restore recovery owns that hold.
`inspectWorkspaceRecoverySource` threw `source_binding_invalid` /
`workspace_reference_mismatch` with retry class `service_required`, and
the hold says Retry "succeeds only after Paperclip is updated". Grader:
`R3:retry_no_new_run`.

**Proposed behavior**
`isAbandonedPreparedRemoteWorkspace(db, run)` is true only when all of
these hold:
- The run has a `prepared` remote reference and no native workspace sync
reference.
- The workspace authority is at a later epoch, and the run is not its
writer.
- The workspace has an abandoned-epoch receipt for the same run, the
same epoch, and the same lease, provider lease and remote cwd.

That rule is used in three places:
- `hasVerifiedWorkspaceRepair` (`workspace-recovery-receipt.ts`): the
retired epoch counts as repaired. This covers held-preparation replay,
the restore-failure check in `settleInterruptedNativeBootstrap`, the
execution blocker, and explicit Retry admission.
`verifiedWorkspaceRepairReceipt`, which returns a receipt id, is
unchanged.
- `inspectWorkspaceRecoverySource` (`workspace-recovery-source.ts`): the
retired source is reported as `source_unavailable` /
`use_alternative_strategy`, not as a binding failure that needs a
service update.
- `heldInterruptedPreparationWhere`: a restore-recovery hold is listed
for the held-preparation reconciler only when the matching receipt
exists (SQL pre-filter). The settle then re-checks the full rule in JS
before it acts.

**Reason and benefit**
The retired epoch held only preparation files: the hold proof requires
that no provider work started. Its unsaved files were already given up
by the reconciler that wrote the receipt. Waiting for a repair, or for a
service update, cannot finish.

## What Changed

- New `server/src/services/remote-workspace-abandoned-preparation.ts`:
`isAbandonedPreparedRemoteWorkspace` (the strict rule) and
`abandonedPreparedRemoteWorkspaceExists` (its SQL pre-filter).
- `workspace-recovery-receipt.ts`: `hasVerifiedWorkspaceRepair` also
accepts the rule.
- `workspace-recovery-source.ts`: a retired prepared source is
`source_unavailable` / `use_alternative_strategy`.
- `interrupted-preparation-hold-settlement.ts`: restore-recovery holds
are listed only with a matching receipt.
- `interrupted-preparation-hold-settle.test.ts`: held-preparation replay
(before and after the first save committed); a restore-recovery hold in
the exact shape saved by the previous build (`restore_failed`,
`workspaceRestoreRecovery`, exhausted `source_binding_invalid`) that now
settles and replays; inspection reporting `source_unavailable`; and the
rule refusing a workspace still at the run's epoch, a run still the
writer, a native sync reference, another run's receipt, or another
allocation's receipt.

Not changed: `run-completion.ts` and the graceful-drain path in
`recovery.ts`.

## Verification

- `cd server && npx vitest run
src/__tests__/interrupted-preparation-hold-settle.test.ts
src/services/task-retry.test.ts`: 101 pass.
- Red before: with the three recovery files at master, 4 tests fail
(held-preparation replay ×2, the restore-recovery replay, and the
inspection result). Green after.
- Neighbouring recovery suites: see the comment below.
- `cd server && npx tsc --noEmit -p tsconfig.json`: pass.
- Chaos rerun on a build with this change: see the comment below.

## Risks

- Previous-release story (upgrade-safe run state): no persisted shape
changes. Holds, run references and receipts written by the previous
build are read as they are. The restore-recovery test uses the exact
shape the previous build saved. A task already stuck in that state
settles on the next reconciler pass after deploy.
- The rule needs an exact receipt (run, epoch, lease, provider lease,
cwd) and a later workspace epoch, so a real recovery is never skipped.
Any mismatch keeps today's behavior.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/interrupted-preparation-hold-settlement.ts

@@ -6,6 +6,7 @@ import { logger } from "../middleware/logger.js";
 import type { OwnerProof } from "./owner-proof.js";
 import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
 import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js";
+import { abandonedPreparedRemoteWorkspaceExists } from "./remote-workspace-abandoned-preparation.js";
 
 /**
  * `legacy_interrupted_preparation_hold`: a run a graceful shutdown interrupted
@@ -19,9 +20,11 @@ import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js
  *
  * Only holds the system recorded itself are listed: an operator decision (an
  * owner-gone attestation, an explicit user continuation, an assigned owner) is
- * never reopened, and a workspace-restore hold (restore failure or recovery
- * evidence) belongs to restore recovery, not to this kind. A run with a lease still cleaning up is not listed yet; the
- * next pass lists it once the lease settles.
+ * never reopened. A workspace-restore hold (restore failure or recovery
+ * evidence) belongs to restore recovery, unless the workspace already retired
+ * the run's exact prepared epoch (`isAbandonedPreparedRemoteWorkspace`): then
+ * nothing is left to restore and this kind settles it. A run with a lease still
+ * cleaning up is not listed yet; the next pass lists it once the lease settles.
  */
 export function heldInterruptedPreparationWhere() {
   return and(inArray(heartbeatRuns.status, ["interrupted", "failed"]), eq(heartbeatRuns.runtimeMode, "legacy"),
@@ -32,8 +35,9 @@ export function heldInterruptedPreparationWhere() {
       and hold.status = 'resolved' and hold.outcome = 'blocked' and hold.owner_user_id is null
       and hold.evidence->'automaticRecovery'->>'policy' = 'preserve_without_replay_v1'
       and hold.evidence->'automaticRecovery'->>'replay' = 'blocked'
-      and not (hold.evidence ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation',
-        'workspaceRestoreFailure', 'workspaceRestoreRecovery']))`,
+      and not (hold.evidence ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation'])
+      and (not (hold.evidence ?| array['workspaceRestoreFailure', 'workspaceRestoreRecovery'])
+        or ${abandonedPreparedRemoteWorkspaceExists()}))`,
     sql`not exists (select 1 from environment_leases lease where lease.company_id = ${heartbeatRuns.companyId}
       and lease.heartbeat_run_id = ${heartbeatRuns.id}
       and (lease.released_at is null or lease.status = 'pending_cleanup' or lease.cleanup_status = 'failed'))`);

server/src/services/remote-workspace-abandoned-preparation.ts

@@ -0,0 +1,37 @@
+import { and, eq, sql } from "drizzle-orm";
+import { executionWorkspaces, heartbeatRuns, type Db } from "@paperclipai/db";
+import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared";
+import { REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA } from "./remote-workspace-epoch-abandonment.js";
+
+type Run = Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "runnerProfileJson">;
+
+export async function isAbandonedPreparedRemoteWorkspace(db: Db, run: Run): Promise<boolean> {
+  if (run.runnerProfileJson?.nativeWorkspaceSync != null) return false;
+  const parsed = remoteWorkspaceRunReferenceSchema.safeParse(run.runnerProfileJson?.remoteWorkspace);
+  if (!parsed.success || parsed.data.state !== "prepared") return false;
+  const reference = parsed.data;
+  const [workspace] = await db.select({ metadata: executionWorkspaces.metadata }).from(executionWorkspaces).where(and(
+    eq(executionWorkspaces.companyId, run.companyId), eq(executionWorkspaces.id, reference.workspaceId)));
+  const authority = readRemoteWorkspaceAuthority(workspace?.metadata);
+  if (!authority || authority.epoch <= reference.epoch || authority.writerRunId === run.id) return false;
+  const receipts = Array.isArray(workspace?.metadata?.remoteWorkspaceAbandonedEpochs)
+    ? workspace!.metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>> : [];
+  return receipts.some(receipt => {
+    const location = receipt?.location as Record<string, unknown> | undefined;
+    return receipt?.schema === REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA && receipt.runId === run.id
+      && receipt.epoch === reference.epoch
+      && location?.leaseId === reference.leaseId && location?.providerLeaseId === reference.providerLeaseId
+      && location?.remoteCwd === reference.remoteCwd;
+  });
+}
+
+export function abandonedPreparedRemoteWorkspaceExists() {
+  const reference = sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'`;
+  return sql`(${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync' is null and ${reference}->>'state' = 'prepared'
+    and exists (select 1 from execution_workspaces workspace
+      cross join lateral jsonb_array_elements(case when jsonb_typeof(workspace.metadata->'remoteWorkspaceAbandonedEpochs') = 'array'
+        then workspace.metadata->'remoteWorkspaceAbandonedEpochs' else '[]'::jsonb end) receipt
+      where workspace.company_id = ${heartbeatRuns.companyId} and workspace.id::text = ${reference}->>'workspaceId'
+        and receipt->>'schema' = ${REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA} and receipt->>'runId' = ${heartbeatRuns.id}::text
+        and receipt->'epoch' = ${reference}->'epoch'))`;
+}

server/src/services/workspace-recovery-receipt.ts

@@ -7,6 +7,7 @@ import { readMigratedLegacyWorkspaceSource } from "./legacy-workspace-recovery-m
 import { readSnapshot } from "./workspace-recovery-snapshot.js";
 import { exhaustedSystemRecoveryAction, readSystemRecovery } from "./system-recovery-state.js";
 import { verifiedWorkspaceSnapshotReuse } from "./remote-workspace-snapshot-reuse-receipt.js";
+import { isAbandonedPreparedRemoteWorkspace } from "./remote-workspace-abandoned-preparation.js";
 
 export const workspaceRecoverySchema = z.object({
   schema: z.literal("paperclip.workspace-recovery.v1"),
@@ -176,7 +177,7 @@ export async function verifiedWorkspaceRepairReceipt(db: Db, run: Pick<typeof he
 }
 
 export async function hasVerifiedWorkspaceRepair(...args: Parameters<typeof verifiedWorkspaceRepairReceipt>) {
-  return (await verifiedWorkspaceRepairReceipt(...args)) !== null;
+  return (await verifiedWorkspaceRepairReceipt(...args)) !== null || await isAbandonedPreparedRemoteWorkspace(...args);
 }
 
 /** Read-only historical bytes have no execution authority. Their immutable

server/src/services/workspace-recovery-source.ts

@@ -7,6 +7,7 @@ import { readNativeWorkspaceSyncReference } from "./native-runtime/native-worksp
 import { readSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js";
 import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
 import { readStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
+import { isAbandonedPreparedRemoteWorkspace } from "./remote-workspace-abandoned-preparation.js";
 import { LEGACY_WORKSPACE_RECOVERY_SCHEMA, SETTLED_WORKSPACE_RESTORE_HOLD_KEY } from "./workspace-restore-recovery-state.js";
 import { ENVIRONMENT_LEASE_POLICIES, EXECUTION_RECONCILIATION_CAUSES, readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type EnvironmentLeasePolicy,
   type WorkspaceSourceVerificationFailure } from "@paperclipai/shared";
@@ -109,6 +110,7 @@ export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRec
     || remoteAuthority.location.remoteCwd !== remoteReference.remoteCwd
     || (remoteReference.state === "prepared" ? !remoteAuthority.dirty || remoteAuthority.writerRunId !== run.id
       : remoteAuthority.dirty || remoteAuthority.writerRunId !== null || remoteAuthority.snapshotId !== remoteReference.snapshotId))) {
+    if (await isAbandonedPreparedRemoteWorkspace(db, run)) throw new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy");
     throw invalid("workspace_reference_mismatch");
   }
   if (remoteReference?.state === "saved") {