From 23fa37f5bbc57e936879fca4ccece7a76c4a4481 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 10 Nov 2026 05:17:46 -0600 Subject: [PATCH] fix(recovery): treat a retired prepared remote epoch as nothing to recover (#20491) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A restart must never leave a task stuck: an interrupted run must resume on its own or have a working Retry > - When a graceful restart interrupts a remote-workspace run that is still preparing, the run keeps a `prepared` reference to its workspace epoch, and a preserve-without-replay hold blocks the task > - The unsaved-files reconciler then retires that epoch with `abandonUnverifiedRemoteWorkspaceEpoch` and writes a `paperclip.remote-workspace-abandoned-epoch.v1` receipt. The workspace content is the last committed save again > - Recovery never read that receipt. The held-preparation reconciler waited for a repair receipt that could never exist, and restore recovery refused the source as `source_binding_invalid`. The task stayed blocked, and a UI Retry returned 202 and started nothing > - This pull request adds one strict shared rule: an abandoned-epoch receipt for the run's exact prepared epoch and allocation means there is nothing to recover. Every recovery path now uses it > - The benefit is that the task resumes automatically from the last committed save after the restart, whichever recovery path owns the hold ## Linked Issues or Issue Description Refs #20072 (replay a preparation a restart interrupted), #20299 (storage outage waits). **What existing behavior does this improve?** Recovery of a remote-workspace run that a graceful restart interrupted during preparation, after its unsaved epoch was retired. **Subsystem affected** Server: recovery (held-preparation reconciler, workspace restore recovery, the workspace repair check). **Current behavior** These are Daytona R1 restart-chaos waves (Paperclip-sized workspace, `--storage s3`, K2 graceful restart at +110 s, during the ~730 MiB first save): - `w-20261110T052533-s8805` and `w-20261110T062703-s8805` (ST2 slow store), held-preparation path: both runs ended `interrupted` in `executionStage: preparing`. The reference was `{ state: "prepared", epoch: 1 }`, and the workspace had moved to `{ epoch: 2, location: null, dirty: false, writerRunId: null }` with an abandoned-epoch receipt for epoch 1. The `legacy_interrupted_preparation_hold` reconciler never settled the hold (`I2_settled` violations), because `isInterruptedNativePreparation` requires `hasVerifiedWorkspaceRepair`, and that check accepts only a `saved` reference. Grader: `R3:retry_no_new_run`. - `w-20261110T091639-s11936` (ST1 outage), restore-recovery path: the run also had `workspaceRestoreFailure: "restore_failed"` and `workspaceRestoreRecovery`. Restore recovery owns that hold. `inspectWorkspaceRecoverySource` threw `source_binding_invalid` / `workspace_reference_mismatch` with retry class `service_required`, and the hold says Retry "succeeds only after Paperclip is updated". Grader: `R3:retry_no_new_run`. **Proposed behavior** `isAbandonedPreparedRemoteWorkspace(db, run)` is true only when all of these hold: - The run has a `prepared` remote reference and no native workspace sync reference. - The workspace authority is at a later epoch, and the run is not its writer. - The workspace has an abandoned-epoch receipt for the same run, the same epoch, and the same lease, provider lease and remote cwd. That rule is used in three places: - `hasVerifiedWorkspaceRepair` (`workspace-recovery-receipt.ts`): the retired epoch counts as repaired. This covers held-preparation replay, the restore-failure check in `settleInterruptedNativeBootstrap`, the execution blocker, and explicit Retry admission. `verifiedWorkspaceRepairReceipt`, which returns a receipt id, is unchanged. - `inspectWorkspaceRecoverySource` (`workspace-recovery-source.ts`): the retired source is reported as `source_unavailable` / `use_alternative_strategy`, not as a binding failure that needs a service update. - `heldInterruptedPreparationWhere`: a restore-recovery hold is listed for the held-preparation reconciler only when the matching receipt exists (SQL pre-filter). The settle then re-checks the full rule in JS before it acts. **Reason and benefit** The retired epoch held only preparation files: the hold proof requires that no provider work started. Its unsaved files were already given up by the reconciler that wrote the receipt. Waiting for a repair, or for a service update, cannot finish. ## What Changed - New `server/src/services/remote-workspace-abandoned-preparation.ts`: `isAbandonedPreparedRemoteWorkspace` (the strict rule) and `abandonedPreparedRemoteWorkspaceExists` (its SQL pre-filter). - `workspace-recovery-receipt.ts`: `hasVerifiedWorkspaceRepair` also accepts the rule. - `workspace-recovery-source.ts`: a retired prepared source is `source_unavailable` / `use_alternative_strategy`. - `interrupted-preparation-hold-settlement.ts`: restore-recovery holds are listed only with a matching receipt. - `interrupted-preparation-hold-settle.test.ts`: held-preparation replay (before and after the first save committed); a restore-recovery hold in the exact shape saved by the previous build (`restore_failed`, `workspaceRestoreRecovery`, exhausted `source_binding_invalid`) that now settles and replays; inspection reporting `source_unavailable`; and the rule refusing a workspace still at the run's epoch, a run still the writer, a native sync reference, another run's receipt, or another allocation's receipt. Not changed: `run-completion.ts` and the graceful-drain path in `recovery.ts`. ## Verification - `cd server && npx vitest run src/__tests__/interrupted-preparation-hold-settle.test.ts src/services/task-retry.test.ts`: 101 pass. - Red before: with the three recovery files at master, 4 tests fail (held-preparation replay ×2, the restore-recovery replay, and the inspection result). Green after. - Neighbouring recovery suites: see the comment below. - `cd server && npx tsc --noEmit -p tsconfig.json`: pass. - Chaos rerun on a build with this change: see the comment below. ## Risks - Previous-release story (upgrade-safe run state): no persisted shape changes. Holds, run references and receipts written by the previous build are read as they are. The restore-recovery test uses the exact shape the previous build saved. A task already stuck in that state settles on the next reconciler pass after deploy. - The rule needs an exact receipt (run, epoch, lease, provider lease, cwd) and a later workspace epoch, so a real recovery is never skipped. Any mismatch keeps today's behavior. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ...nterrupted-preparation-hold-settle.test.ts | 88 +++++++++++++++++++ ...interrupted-preparation-hold-settlement.ts | 14 +-- .../remote-workspace-abandoned-preparation.ts | 37 ++++++++ .../services/workspace-recovery-receipt.ts | 3 +- .../src/services/workspace-recovery-source.ts | 2 + 5 files changed, 138 insertions(+), 6 deletions(-) create mode 100644 server/src/services/remote-workspace-abandoned-preparation.ts diff --git a/server/src/__tests__/interrupted-preparation-hold-settle.test.ts b/server/src/__tests__/interrupted-preparation-hold-settle.test.ts index 4f4373898db..31e563a2dca 100644 --- a/server/src/__tests__/interrupted-preparation-hold-settle.test.ts +++ b/server/src/__tests__/interrupted-preparation-hold-settle.test.ts @@ -11,6 +11,7 @@ import { createDb, environmentLeases, environments, + executionWorkspaces, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, @@ -20,6 +21,8 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro import { currentProcessSpaceHash } from "../services/controller-process-identity.js"; import { heartbeatService } from "../services/heartbeat.js"; import { settleHeldInterruptedPreparation } from "../services/interrupted-preparation-hold-settlement.js"; +import { isAbandonedPreparedRemoteWorkspace } from "../services/remote-workspace-abandoned-preparation.js"; +import { inspectWorkspaceRecoverySource } from "../services/workspace-recovery-source.js"; import { legacyControllerBootId } from "../services/legacy-controller-lease.js"; import { createOwnershipReportState, runOwnershipReconciliationPass } from "../services/recovery/ownership-invariant-report.js"; @@ -67,6 +70,7 @@ suite("held interrupted preparations on the shared ownership path", () => { await db.delete(activityLog); await db.delete(issueRecoveryActions); await db.delete(environmentLeases); + await db.delete(executionWorkspaces); await db.delete(heartbeatRunEvents); await db.delete(heartbeatRuns); await db.delete(issues); @@ -171,6 +175,90 @@ suite("held interrupted preparations on the shared ownership path", () => { expect(await hold(f.holdId)).toEqual(before); }); + async function preparedRemoteWorkspace(runId: string, issueId: string, options: { receipt?: Record; + savedSnapshotId?: string | null; authority?: Record; profile?: Record } = {}) { + const workspaceId = randomUUID(), leaseId = randomUUID(), providerLeaseId = `sandbox-${leaseId}`; + const remoteCwd = `/home/daytona/paperclip-workspace/.paperclip-workspaces/${workspaceId}-1-${leaseId}`; + const location = { environmentId, leaseId, providerLeaseId, remoteCwd }; + const source = { kind: "git", repoUrl: "https://github.com/paperclipai/paperclip.git", ref: "a".repeat(40), + branchMode: "create", branchName: "IPH-1-change" }; + const authority = { schema: "paperclip.remote-workspace.v1", epoch: 2, source, location: null, dirty: false, writerRunId: null, + snapshotId: options.savedSnapshotId ?? null, ...options.authority }; + const receipts = [{ schema: "paperclip.remote-workspace-abandoned-epoch.v1", epoch: 1, runId, + snapshotId: options.savedSnapshotId ?? null, location, providerRef: remoteCwd, abandonedAt: new Date().toISOString(), ...options.receipt }]; + await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId, mode: "isolated_workspace", + strategyType: "git_worktree", name: "Remote workspace", status: "active", providerType: "git_worktree", + metadata: { remoteWorkspace: authority, remoteWorkspaceAbandonedEpochs: receipts } }); + await db.update(heartbeatRuns).set({ runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" }, + remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", state: "prepared", workspaceId, epoch: 1, leaseId, + providerLeaseId, remoteCwd, snapshotId: null }, ...options.profile } }).where(eq(heartbeatRuns.id, runId)); + return { leaseId }; + } + + const runRow = async (runId: string) => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]!; + + it("replays a restore-recovery hold saved by an older build once the workspace retired the run's epoch", async () => { + heartbeatService(db); + const f = await interruptedPreparation(); + const { leaseId } = await preparedRemoteWorkspace(f.runId, f.issueId); + const restore = { workspaceRestoreFailure: "restore_failed", + workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [leaseId] } }; + const run = await runRow(f.runId); + await db.update(heartbeatRuns).set({ resultJson: { ...run.resultJson, ...restore } }).where(eq(heartbeatRuns.id, f.runId)); + const saved = await hold(f.holdId); + await db.update(issueRecoveryActions).set({ evidence: { ...saved.evidence, ...restore, + systemRecovery: { schema: "paperclip.system-recovery.v1", policyVersion: 1, state: "exhausted", phase: "verify_execution", + sourceRunId: f.runId, attempts: 1, retriesUsed: 0, revision: 5, nextAttemptAt: null, + failure: { code: "source_binding_invalid", retryClass: "service_required", sourceVerification: { reason: "workspace_reference_mismatch" } } } } }) + .where(eq(issueRecoveryActions.id, f.holdId)); + expect(await pass()).toMatchObject({ inProgress: 1, settled: 1 }); + expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "in_progress" }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.runId))).toHaveLength(1); + }); + + it("reports a retired prepared epoch to workspace recovery as an unavailable source, not a binding failure", async () => { + const f = await interruptedPreparation(); + await preparedRemoteWorkspace(f.runId, f.issueId); + await expect(inspectWorkspaceRecoverySource(db, { companyId, issueId: f.issueId, runId: f.runId, actionId: f.holdId })) + .rejects.toMatchObject({ code: "source_unavailable", retryClass: "use_alternative_strategy" }); + }); + + const proofCases: Array<[string, { authority?: Record; profile?: Record; writer?: boolean }, boolean]> = [ + ["the workspace is still at the run's epoch", { authority: { epoch: 1 } }, false], + ["the run is still the workspace writer", { writer: true }, false], + ["the run also has a native workspace sync reference", { profile: { nativeWorkspaceSync: { state: "prepared" } } }, false], + ["the receipt matches the run's exact epoch and allocation", {}, true], + ]; + it.each(proofCases)("treats the prepared workspace as retired only when proven: %s", async (_label, options, expected) => { + const f = await interruptedPreparation(); + const authority = options.writer ? { writerRunId: f.runId } : options.authority; + await preparedRemoteWorkspace(f.runId, f.issueId, { profile: options.profile, authority }); + expect(await isAbandonedPreparedRemoteWorkspace(db, await runRow(f.runId))).toBe(expected); + }); + + it.each([ + ["before its first save finished", null], + ["after its first save committed", randomUUID()], + ] as const)("replays a remote preparation whose unsaved epoch the workspace already retired %s", async (_label, savedSnapshotId) => { + heartbeatService(db); + const f = await interruptedPreparation(); + await preparedRemoteWorkspace(f.runId, f.issueId, { savedSnapshotId }); + expect(await pass()).toMatchObject({ inProgress: 1, settled: 1 }); + expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "in_progress" }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.runId))).toHaveLength(1); + }); + + it.each([ + ["the retired epoch belongs to another run", { receipt: { runId: randomUUID() } }], + ["the retired epoch was another allocation", { receipt: { location: { leaseId: randomUUID() } } }], + ] as const)("keeps a remote preparation hold when %s", async (_label, options) => { + const f = await interruptedPreparation(); + await preparedRemoteWorkspace(f.runId, f.issueId, options); + const before = await hold(f.holdId); + expect((await pass())?.settled ?? 0).toBe(0); + expect(await hold(f.holdId)).toEqual(before); + }); + it("settles exactly once when two processes judge the same hold", async () => { heartbeatService(db); const f = await interruptedPreparation(); diff --git a/server/src/services/interrupted-preparation-hold-settlement.ts b/server/src/services/interrupted-preparation-hold-settlement.ts index 99de8f7ccc6..c6647d03ed8 100644 --- a/server/src/services/interrupted-preparation-hold-settlement.ts +++ b/server/src/services/interrupted-preparation-hold-settlement.ts @@ -6,6 +6,7 @@ import { logger } from "../middleware/logger.js"; import type { OwnerProof } from "./owner-proof.js"; import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js"; import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js"; +import { abandonedPreparedRemoteWorkspaceExists } from "./remote-workspace-abandoned-preparation.js"; /** * `legacy_interrupted_preparation_hold`: a run a graceful shutdown interrupted @@ -19,9 +20,11 @@ import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js * * Only holds the system recorded itself are listed: an operator decision (an * owner-gone attestation, an explicit user continuation, an assigned owner) is - * never reopened, and a workspace-restore hold (restore failure or recovery - * evidence) belongs to restore recovery, not to this kind. A run with a lease still cleaning up is not listed yet; the - * next pass lists it once the lease settles. + * never reopened. A workspace-restore hold (restore failure or recovery + * evidence) belongs to restore recovery, unless the workspace already retired + * the run's exact prepared epoch (`isAbandonedPreparedRemoteWorkspace`): then + * nothing is left to restore and this kind settles it. A run with a lease still + * cleaning up is not listed yet; the next pass lists it once the lease settles. */ export function heldInterruptedPreparationWhere() { return and(inArray(heartbeatRuns.status, ["interrupted", "failed"]), eq(heartbeatRuns.runtimeMode, "legacy"), @@ -32,8 +35,9 @@ export function heldInterruptedPreparationWhere() { and hold.status = 'resolved' and hold.outcome = 'blocked' and hold.owner_user_id is null and hold.evidence->'automaticRecovery'->>'policy' = 'preserve_without_replay_v1' and hold.evidence->'automaticRecovery'->>'replay' = 'blocked' - and not (hold.evidence ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation', - 'workspaceRestoreFailure', 'workspaceRestoreRecovery']))`, + and not (hold.evidence ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation']) + and (not (hold.evidence ?| array['workspaceRestoreFailure', 'workspaceRestoreRecovery']) + or ${abandonedPreparedRemoteWorkspaceExists()}))`, sql`not exists (select 1 from environment_leases lease where lease.company_id = ${heartbeatRuns.companyId} and lease.heartbeat_run_id = ${heartbeatRuns.id} and (lease.released_at is null or lease.status = 'pending_cleanup' or lease.cleanup_status = 'failed'))`); diff --git a/server/src/services/remote-workspace-abandoned-preparation.ts b/server/src/services/remote-workspace-abandoned-preparation.ts new file mode 100644 index 00000000000..dfc94a92cd2 --- /dev/null +++ b/server/src/services/remote-workspace-abandoned-preparation.ts @@ -0,0 +1,37 @@ +import { and, eq, sql } from "drizzle-orm"; +import { executionWorkspaces, heartbeatRuns, type Db } from "@paperclipai/db"; +import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared"; +import { REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA } from "./remote-workspace-epoch-abandonment.js"; + +type Run = Pick; + +export async function isAbandonedPreparedRemoteWorkspace(db: Db, run: Run): Promise { + if (run.runnerProfileJson?.nativeWorkspaceSync != null) return false; + const parsed = remoteWorkspaceRunReferenceSchema.safeParse(run.runnerProfileJson?.remoteWorkspace); + if (!parsed.success || parsed.data.state !== "prepared") return false; + const reference = parsed.data; + const [workspace] = await db.select({ metadata: executionWorkspaces.metadata }).from(executionWorkspaces).where(and( + eq(executionWorkspaces.companyId, run.companyId), eq(executionWorkspaces.id, reference.workspaceId))); + const authority = readRemoteWorkspaceAuthority(workspace?.metadata); + if (!authority || authority.epoch <= reference.epoch || authority.writerRunId === run.id) return false; + const receipts = Array.isArray(workspace?.metadata?.remoteWorkspaceAbandonedEpochs) + ? workspace!.metadata!.remoteWorkspaceAbandonedEpochs as Array> : []; + return receipts.some(receipt => { + const location = receipt?.location as Record | undefined; + return receipt?.schema === REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA && receipt.runId === run.id + && receipt.epoch === reference.epoch + && location?.leaseId === reference.leaseId && location?.providerLeaseId === reference.providerLeaseId + && location?.remoteCwd === reference.remoteCwd; + }); +} + +export function abandonedPreparedRemoteWorkspaceExists() { + const reference = sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'`; + return sql`(${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync' is null and ${reference}->>'state' = 'prepared' + and exists (select 1 from execution_workspaces workspace + cross join lateral jsonb_array_elements(case when jsonb_typeof(workspace.metadata->'remoteWorkspaceAbandonedEpochs') = 'array' + then workspace.metadata->'remoteWorkspaceAbandonedEpochs' else '[]'::jsonb end) receipt + where workspace.company_id = ${heartbeatRuns.companyId} and workspace.id::text = ${reference}->>'workspaceId' + and receipt->>'schema' = ${REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA} and receipt->>'runId' = ${heartbeatRuns.id}::text + and receipt->'epoch' = ${reference}->'epoch'))`; +} diff --git a/server/src/services/workspace-recovery-receipt.ts b/server/src/services/workspace-recovery-receipt.ts index a79334c7dc7..12dbd51bc50 100644 --- a/server/src/services/workspace-recovery-receipt.ts +++ b/server/src/services/workspace-recovery-receipt.ts @@ -7,6 +7,7 @@ import { readMigratedLegacyWorkspaceSource } from "./legacy-workspace-recovery-m import { readSnapshot } from "./workspace-recovery-snapshot.js"; import { exhaustedSystemRecoveryAction, readSystemRecovery } from "./system-recovery-state.js"; import { verifiedWorkspaceSnapshotReuse } from "./remote-workspace-snapshot-reuse-receipt.js"; +import { isAbandonedPreparedRemoteWorkspace } from "./remote-workspace-abandoned-preparation.js"; export const workspaceRecoverySchema = z.object({ schema: z.literal("paperclip.workspace-recovery.v1"), @@ -176,7 +177,7 @@ export async function verifiedWorkspaceRepairReceipt(db: Db, run: Pick) { - return (await verifiedWorkspaceRepairReceipt(...args)) !== null; + return (await verifiedWorkspaceRepairReceipt(...args)) !== null || await isAbandonedPreparedRemoteWorkspace(...args); } /** Read-only historical bytes have no execution authority. Their immutable diff --git a/server/src/services/workspace-recovery-source.ts b/server/src/services/workspace-recovery-source.ts index 04b4f04cc0e..f35a69fb599 100644 --- a/server/src/services/workspace-recovery-source.ts +++ b/server/src/services/workspace-recovery-source.ts @@ -7,6 +7,7 @@ import { readNativeWorkspaceSyncReference } from "./native-runtime/native-worksp import { readSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js"; import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js"; import { readStopOnlyCleanup } from "./sandbox-stop-and-retain.js"; +import { isAbandonedPreparedRemoteWorkspace } from "./remote-workspace-abandoned-preparation.js"; import { LEGACY_WORKSPACE_RECOVERY_SCHEMA, SETTLED_WORKSPACE_RESTORE_HOLD_KEY } from "./workspace-restore-recovery-state.js"; import { ENVIRONMENT_LEASE_POLICIES, EXECUTION_RECONCILIATION_CAUSES, readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type EnvironmentLeasePolicy, type WorkspaceSourceVerificationFailure } from "@paperclipai/shared"; @@ -109,6 +110,7 @@ export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRec || remoteAuthority.location.remoteCwd !== remoteReference.remoteCwd || (remoteReference.state === "prepared" ? !remoteAuthority.dirty || remoteAuthority.writerRunId !== run.id : remoteAuthority.dirty || remoteAuthority.writerRunId !== null || remoteAuthority.snapshotId !== remoteReference.snapshotId))) { + if (await isAbandonedPreparedRemoteWorkspace(db, run)) throw new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy"); throw invalid("workspace_reference_mismatch"); } if (remoteReference?.state === "saved") {