fix(recovery): treat a retired prepared remote epoch as nothing to recover (#20491)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A restart must never leave a task stuck: an interrupted run must
resume on its own or have a working Retry
> - When a graceful restart interrupts a remote-workspace run that is
still preparing, the run keeps a `prepared` reference to its workspace
epoch, and a preserve-without-replay hold blocks the task
> - The unsaved-files reconciler then retires that epoch with
`abandonUnverifiedRemoteWorkspaceEpoch` and writes a
`paperclip.remote-workspace-abandoned-epoch.v1` receipt. The workspace
content is the last committed save again
> - Recovery never read that receipt. The held-preparation reconciler
waited for a repair receipt that could never exist, and restore recovery
refused the source as `source_binding_invalid`. The task stayed blocked,
and a UI Retry returned 202 and started nothing
> - This pull request adds one strict shared rule: an abandoned-epoch
receipt for the run's exact prepared epoch and allocation means there is
nothing to recover. Every recovery path now uses it
> - The benefit is that the task resumes automatically from the last
committed save after the restart, whichever recovery path owns the hold

## Linked Issues or Issue Description

Refs #20072 (replay a preparation a restart interrupted), #20299
(storage outage waits).

**What existing behavior does this improve?**
Recovery of a remote-workspace run that a graceful restart interrupted
during preparation, after its unsaved epoch was retired.

**Subsystem affected**
Server: recovery (held-preparation reconciler, workspace restore
recovery, the workspace repair check).

**Current behavior**
These are Daytona R1 restart-chaos waves (Paperclip-sized workspace,
`--storage s3`, K2 graceful restart at +110 s, during the ~730 MiB first
save):
- `w-20261110T052533-s8805` and `w-20261110T062703-s8805` (ST2 slow
store), held-preparation path: both runs ended `interrupted` in
`executionStage: preparing`. The reference was `{ state: "prepared",
epoch: 1 }`, and the workspace had moved to `{ epoch: 2, location: null,
dirty: false, writerRunId: null }` with an abandoned-epoch receipt for
epoch 1. The `legacy_interrupted_preparation_hold` reconciler never
settled the hold (`I2_settled` violations), because
`isInterruptedNativePreparation` requires `hasVerifiedWorkspaceRepair`,
and that check accepts only a `saved` reference. Grader:
`R3:retry_no_new_run`.
- `w-20261110T091639-s11936` (ST1 outage), restore-recovery path: the
run also had `workspaceRestoreFailure: "restore_failed"` and
`workspaceRestoreRecovery`. Restore recovery owns that hold.
`inspectWorkspaceRecoverySource` threw `source_binding_invalid` /
`workspace_reference_mismatch` with retry class `service_required`, and
the hold says Retry "succeeds only after Paperclip is updated". Grader:
`R3:retry_no_new_run`.

**Proposed behavior**
`isAbandonedPreparedRemoteWorkspace(db, run)` is true only when all of
these hold:
- The run has a `prepared` remote reference and no native workspace sync
reference.
- The workspace authority is at a later epoch, and the run is not its
writer.
- The workspace has an abandoned-epoch receipt for the same run, the
same epoch, and the same lease, provider lease and remote cwd.

That rule is used in three places:
- `hasVerifiedWorkspaceRepair` (`workspace-recovery-receipt.ts`): the
retired epoch counts as repaired. This covers held-preparation replay,
the restore-failure check in `settleInterruptedNativeBootstrap`, the
execution blocker, and explicit Retry admission.
`verifiedWorkspaceRepairReceipt`, which returns a receipt id, is
unchanged.
- `inspectWorkspaceRecoverySource` (`workspace-recovery-source.ts`): the
retired source is reported as `source_unavailable` /
`use_alternative_strategy`, not as a binding failure that needs a
service update.
- `heldInterruptedPreparationWhere`: a restore-recovery hold is listed
for the held-preparation reconciler only when the matching receipt
exists (SQL pre-filter). The settle then re-checks the full rule in JS
before it acts.

**Reason and benefit**
The retired epoch held only preparation files: the hold proof requires
that no provider work started. Its unsaved files were already given up
by the reconciler that wrote the receipt. Waiting for a repair, or for a
service update, cannot finish.

## What Changed

- New `server/src/services/remote-workspace-abandoned-preparation.ts`:
`isAbandonedPreparedRemoteWorkspace` (the strict rule) and
`abandonedPreparedRemoteWorkspaceExists` (its SQL pre-filter).
- `workspace-recovery-receipt.ts`: `hasVerifiedWorkspaceRepair` also
accepts the rule.
- `workspace-recovery-source.ts`: a retired prepared source is
`source_unavailable` / `use_alternative_strategy`.
- `interrupted-preparation-hold-settlement.ts`: restore-recovery holds
are listed only with a matching receipt.
- `interrupted-preparation-hold-settle.test.ts`: held-preparation replay
(before and after the first save committed); a restore-recovery hold in
the exact shape saved by the previous build (`restore_failed`,
`workspaceRestoreRecovery`, exhausted `source_binding_invalid`) that now
settles and replays; inspection reporting `source_unavailable`; and the
rule refusing a workspace still at the run's epoch, a run still the
writer, a native sync reference, another run's receipt, or another
allocation's receipt.

Not changed: `run-completion.ts` and the graceful-drain path in
`recovery.ts`.

## Verification

- `cd server && npx vitest run
src/__tests__/interrupted-preparation-hold-settle.test.ts
src/services/task-retry.test.ts`: 101 pass.
- Red before: with the three recovery files at master, 4 tests fail
(held-preparation replay ×2, the restore-recovery replay, and the
inspection result). Green after.
- Neighbouring recovery suites: see the comment below.
- `cd server && npx tsc --noEmit -p tsconfig.json`: pass.
- Chaos rerun on a build with this change: see the comment below.

## Risks

- Previous-release story (upgrade-safe run state): no persisted shape
changes. Holds, run references and receipts written by the previous
build are read as they are. The restore-recovery test uses the exact
shape the previous build saved. A task already stuck in that state
settles on the next reconciler pass after deploy.
- The rule needs an exact receipt (run, epoch, lease, provider lease,
cwd) and a later workspace epoch, so a real recovery is never skipped.
Any mismatch keeps today's behavior.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-10 05:17:46 -06:00
1 parent c2423e30e8
commit 23fa37f5bb
5 files changed
+138 -6

No files matched your search

@@ -11,6 +11,7 @@ import {
createDb, createDb,
environmentLeases, environmentLeases,
environments, environments,
executionWorkspaces,
heartbeatRunEvents, heartbeatRunEvents,
heartbeatRuns, heartbeatRuns,
issueRecoveryActions, issueRecoveryActions,
@@ -20,6 +21,8 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro
import { currentProcessSpaceHash } from "../services/controller-process-identity.js"; import { currentProcessSpaceHash } from "../services/controller-process-identity.js";
import { heartbeatService } from "../services/heartbeat.js"; import { heartbeatService } from "../services/heartbeat.js";
import { settleHeldInterruptedPreparation } from "../services/interrupted-preparation-hold-settlement.js"; import { settleHeldInterruptedPreparation } from "../services/interrupted-preparation-hold-settlement.js";
import { isAbandonedPreparedRemoteWorkspace } from "../services/remote-workspace-abandoned-preparation.js";
import { inspectWorkspaceRecoverySource } from "../services/workspace-recovery-source.js";
import { legacyControllerBootId } from "../services/legacy-controller-lease.js"; import { legacyControllerBootId } from "../services/legacy-controller-lease.js";
import { createOwnershipReportState, runOwnershipReconciliationPass } from "../services/recovery/ownership-invariant-report.js"; import { createOwnershipReportState, runOwnershipReconciliationPass } from "../services/recovery/ownership-invariant-report.js";
@@ -67,6 +70,7 @@ suite("held interrupted preparations on the shared ownership path", () => {
await db.delete(activityLog); await db.delete(activityLog);
await db.delete(issueRecoveryActions); await db.delete(issueRecoveryActions);
await db.delete(environmentLeases); await db.delete(environmentLeases);
await db.delete(executionWorkspaces);
await db.delete(heartbeatRunEvents); await db.delete(heartbeatRunEvents);
await db.delete(heartbeatRuns); await db.delete(heartbeatRuns);
await db.delete(issues); await db.delete(issues);
@@ -171,6 +175,90 @@ suite("held interrupted preparations on the shared ownership path", () => {
expect(await hold(f.holdId)).toEqual(before); expect(await hold(f.holdId)).toEqual(before);
}); });
async function preparedRemoteWorkspace(runId: string, issueId: string, options: { receipt?: Record<string, unknown>;
savedSnapshotId?: string | null; authority?: Record<string, unknown>; profile?: Record<string, unknown> } = {}) {
const workspaceId = randomUUID(), leaseId = randomUUID(), providerLeaseId = `sandbox-${leaseId}`;
const remoteCwd = `/home/daytona/paperclip-workspace/.paperclip-workspaces/${workspaceId}-1-${leaseId}`;
const location = { environmentId, leaseId, providerLeaseId, remoteCwd };
const source = { kind: "git", repoUrl: "https://github.com/paperclipai/paperclip.git", ref: "a".repeat(40),
branchMode: "create", branchName: "IPH-1-change" };
const authority = { schema: "paperclip.remote-workspace.v1", epoch: 2, source, location: null, dirty: false, writerRunId: null,
snapshotId: options.savedSnapshotId ?? null, ...options.authority };
const receipts = [{ schema: "paperclip.remote-workspace-abandoned-epoch.v1", epoch: 1, runId,
snapshotId: options.savedSnapshotId ?? null, location, providerRef: remoteCwd, abandonedAt: new Date().toISOString(), ...options.receipt }];
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId, mode: "isolated_workspace",
strategyType: "git_worktree", name: "Remote workspace", status: "active", providerType: "git_worktree",
metadata: { remoteWorkspace: authority, remoteWorkspaceAbandonedEpochs: receipts } });
await db.update(heartbeatRuns).set({ runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" },
remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", state: "prepared", workspaceId, epoch: 1, leaseId,
providerLeaseId, remoteCwd, snapshotId: null }, ...options.profile } }).where(eq(heartbeatRuns.id, runId));
return { leaseId };
}
const runRow = async (runId: string) => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]!;
it("replays a restore-recovery hold saved by an older build once the workspace retired the run's epoch", async () => {
heartbeatService(db);
const f = await interruptedPreparation();
const { leaseId } = await preparedRemoteWorkspace(f.runId, f.issueId);
const restore = { workspaceRestoreFailure: "restore_failed",
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [leaseId] } };
const run = await runRow(f.runId);
await db.update(heartbeatRuns).set({ resultJson: { ...run.resultJson, ...restore } }).where(eq(heartbeatRuns.id, f.runId));
const saved = await hold(f.holdId);
await db.update(issueRecoveryActions).set({ evidence: { ...saved.evidence, ...restore,
systemRecovery: { schema: "paperclip.system-recovery.v1", policyVersion: 1, state: "exhausted", phase: "verify_execution",
sourceRunId: f.runId, attempts: 1, retriesUsed: 0, revision: 5, nextAttemptAt: null,
failure: { code: "source_binding_invalid", retryClass: "service_required", sourceVerification: { reason: "workspace_reference_mismatch" } } } } })
.where(eq(issueRecoveryActions.id, f.holdId));
expect(await pass()).toMatchObject({ inProgress: 1, settled: 1 });
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "in_progress" });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.runId))).toHaveLength(1);
});
it("reports a retired prepared epoch to workspace recovery as an unavailable source, not a binding failure", async () => {
const f = await interruptedPreparation();
await preparedRemoteWorkspace(f.runId, f.issueId);
await expect(inspectWorkspaceRecoverySource(db, { companyId, issueId: f.issueId, runId: f.runId, actionId: f.holdId }))
.rejects.toMatchObject({ code: "source_unavailable", retryClass: "use_alternative_strategy" });
});
const proofCases: Array<[string, { authority?: Record<string, unknown>; profile?: Record<string, unknown>; writer?: boolean }, boolean]> = [
["the workspace is still at the run's epoch", { authority: { epoch: 1 } }, false],
["the run is still the workspace writer", { writer: true }, false],
["the run also has a native workspace sync reference", { profile: { nativeWorkspaceSync: { state: "prepared" } } }, false],
["the receipt matches the run's exact epoch and allocation", {}, true],
];
it.each(proofCases)("treats the prepared workspace as retired only when proven: %s", async (_label, options, expected) => {
const f = await interruptedPreparation();
const authority = options.writer ? { writerRunId: f.runId } : options.authority;
await preparedRemoteWorkspace(f.runId, f.issueId, { profile: options.profile, authority });
expect(await isAbandonedPreparedRemoteWorkspace(db, await runRow(f.runId))).toBe(expected);
});
it.each([
["before its first save finished", null],
["after its first save committed", randomUUID()],
] as const)("replays a remote preparation whose unsaved epoch the workspace already retired %s", async (_label, savedSnapshotId) => {
heartbeatService(db);
const f = await interruptedPreparation();
await preparedRemoteWorkspace(f.runId, f.issueId, { savedSnapshotId });
expect(await pass()).toMatchObject({ inProgress: 1, settled: 1 });
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "in_progress" });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.runId))).toHaveLength(1);
});
it.each([
["the retired epoch belongs to another run", { receipt: { runId: randomUUID() } }],
["the retired epoch was another allocation", { receipt: { location: { leaseId: randomUUID() } } }],
] as const)("keeps a remote preparation hold when %s", async (_label, options) => {
const f = await interruptedPreparation();
await preparedRemoteWorkspace(f.runId, f.issueId, options);
const before = await hold(f.holdId);
expect((await pass())?.settled ?? 0).toBe(0);
expect(await hold(f.holdId)).toEqual(before);
});
it("settles exactly once when two processes judge the same hold", async () => { it("settles exactly once when two processes judge the same hold", async () => {
heartbeatService(db); heartbeatService(db);
const f = await interruptedPreparation(); const f = await interruptedPreparation();
@@ -6,6 +6,7 @@ import { logger } from "../middleware/logger.js";
import type { OwnerProof } from "./owner-proof.js"; import type { OwnerProof } from "./owner-proof.js";
import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js"; import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js"; import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js";
import { abandonedPreparedRemoteWorkspaceExists } from "./remote-workspace-abandoned-preparation.js";
/** /**
* `legacy_interrupted_preparation_hold`: a run a graceful shutdown interrupted * `legacy_interrupted_preparation_hold`: a run a graceful shutdown interrupted
@@ -19,9 +20,11 @@ import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js
* *
* Only holds the system recorded itself are listed: an operator decision (an * Only holds the system recorded itself are listed: an operator decision (an
* owner-gone attestation, an explicit user continuation, an assigned owner) is * owner-gone attestation, an explicit user continuation, an assigned owner) is
* never reopened, and a workspace-restore hold (restore failure or recovery * never reopened. A workspace-restore hold (restore failure or recovery
* evidence) belongs to restore recovery, not to this kind. A run with a lease still cleaning up is not listed yet; the * evidence) belongs to restore recovery, unless the workspace already retired
* next pass lists it once the lease settles. * the run's exact prepared epoch (`isAbandonedPreparedRemoteWorkspace`): then
* nothing is left to restore and this kind settles it. A run with a lease still
* cleaning up is not listed yet; the next pass lists it once the lease settles.
*/ */
export function heldInterruptedPreparationWhere() { export function heldInterruptedPreparationWhere() {
return and(inArray(heartbeatRuns.status, ["interrupted", "failed"]), eq(heartbeatRuns.runtimeMode, "legacy"), return and(inArray(heartbeatRuns.status, ["interrupted", "failed"]), eq(heartbeatRuns.runtimeMode, "legacy"),
@@ -32,8 +35,9 @@ export function heldInterruptedPreparationWhere() {
and hold.status = 'resolved' and hold.outcome = 'blocked' and hold.owner_user_id is null and hold.status = 'resolved' and hold.outcome = 'blocked' and hold.owner_user_id is null
and hold.evidence->'automaticRecovery'->>'policy' = 'preserve_without_replay_v1' and hold.evidence->'automaticRecovery'->>'policy' = 'preserve_without_replay_v1'
and hold.evidence->'automaticRecovery'->>'replay' = 'blocked' and hold.evidence->'automaticRecovery'->>'replay' = 'blocked'
and not (hold.evidence ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation', and not (hold.evidence ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation'])
'workspaceRestoreFailure', 'workspaceRestoreRecovery']))`, and (not (hold.evidence ?| array['workspaceRestoreFailure', 'workspaceRestoreRecovery'])
or ${abandonedPreparedRemoteWorkspaceExists()}))`,
sql`not exists (select 1 from environment_leases lease where lease.company_id = ${heartbeatRuns.companyId} sql`not exists (select 1 from environment_leases lease where lease.company_id = ${heartbeatRuns.companyId}
and lease.heartbeat_run_id = ${heartbeatRuns.id} and lease.heartbeat_run_id = ${heartbeatRuns.id}
and (lease.released_at is null or lease.status = 'pending_cleanup' or lease.cleanup_status = 'failed'))`); and (lease.released_at is null or lease.status = 'pending_cleanup' or lease.cleanup_status = 'failed'))`);
@@ -0,0 +1,37 @@
import { and, eq, sql } from "drizzle-orm";
import { executionWorkspaces, heartbeatRuns, type Db } from "@paperclipai/db";
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared";
import { REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA } from "./remote-workspace-epoch-abandonment.js";
type Run = Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "runnerProfileJson">;
export async function isAbandonedPreparedRemoteWorkspace(db: Db, run: Run): Promise<boolean> {
if (run.runnerProfileJson?.nativeWorkspaceSync != null) return false;
const parsed = remoteWorkspaceRunReferenceSchema.safeParse(run.runnerProfileJson?.remoteWorkspace);
if (!parsed.success || parsed.data.state !== "prepared") return false;
const reference = parsed.data;
const [workspace] = await db.select({ metadata: executionWorkspaces.metadata }).from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, run.companyId), eq(executionWorkspaces.id, reference.workspaceId)));
const authority = readRemoteWorkspaceAuthority(workspace?.metadata);
if (!authority || authority.epoch <= reference.epoch || authority.writerRunId === run.id) return false;
const receipts = Array.isArray(workspace?.metadata?.remoteWorkspaceAbandonedEpochs)
? workspace!.metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>> : [];
return receipts.some(receipt => {
const location = receipt?.location as Record<string, unknown> | undefined;
return receipt?.schema === REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA && receipt.runId === run.id
&& receipt.epoch === reference.epoch
&& location?.leaseId === reference.leaseId && location?.providerLeaseId === reference.providerLeaseId
&& location?.remoteCwd === reference.remoteCwd;
});
}
export function abandonedPreparedRemoteWorkspaceExists() {
const reference = sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'`;
return sql`(${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync' is null and ${reference}->>'state' = 'prepared'
and exists (select 1 from execution_workspaces workspace
cross join lateral jsonb_array_elements(case when jsonb_typeof(workspace.metadata->'remoteWorkspaceAbandonedEpochs') = 'array'
then workspace.metadata->'remoteWorkspaceAbandonedEpochs' else '[]'::jsonb end) receipt
where workspace.company_id = ${heartbeatRuns.companyId} and workspace.id::text = ${reference}->>'workspaceId'
and receipt->>'schema' = ${REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA} and receipt->>'runId' = ${heartbeatRuns.id}::text
and receipt->'epoch' = ${reference}->'epoch'))`;
}
@@ -7,6 +7,7 @@ import { readMigratedLegacyWorkspaceSource } from "./legacy-workspace-recovery-m
import { readSnapshot } from "./workspace-recovery-snapshot.js"; import { readSnapshot } from "./workspace-recovery-snapshot.js";
import { exhaustedSystemRecoveryAction, readSystemRecovery } from "./system-recovery-state.js"; import { exhaustedSystemRecoveryAction, readSystemRecovery } from "./system-recovery-state.js";
import { verifiedWorkspaceSnapshotReuse } from "./remote-workspace-snapshot-reuse-receipt.js"; import { verifiedWorkspaceSnapshotReuse } from "./remote-workspace-snapshot-reuse-receipt.js";
import { isAbandonedPreparedRemoteWorkspace } from "./remote-workspace-abandoned-preparation.js";
export const workspaceRecoverySchema = z.object({ export const workspaceRecoverySchema = z.object({
schema: z.literal("paperclip.workspace-recovery.v1"), schema: z.literal("paperclip.workspace-recovery.v1"),
@@ -176,7 +177,7 @@ export async function verifiedWorkspaceRepairReceipt(db: Db, run: Pick<typeof he
} }
export async function hasVerifiedWorkspaceRepair(...args: Parameters<typeof verifiedWorkspaceRepairReceipt>) { export async function hasVerifiedWorkspaceRepair(...args: Parameters<typeof verifiedWorkspaceRepairReceipt>) {
return (await verifiedWorkspaceRepairReceipt(...args)) !== null; return (await verifiedWorkspaceRepairReceipt(...args)) !== null || await isAbandonedPreparedRemoteWorkspace(...args);
} }
/** Read-only historical bytes have no execution authority. Their immutable /** Read-only historical bytes have no execution authority. Their immutable
@@ -7,6 +7,7 @@ import { readNativeWorkspaceSyncReference } from "./native-runtime/native-worksp
import { readSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js"; import { readSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js";
import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js"; import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
import { readStopOnlyCleanup } from "./sandbox-stop-and-retain.js"; import { readStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
import { isAbandonedPreparedRemoteWorkspace } from "./remote-workspace-abandoned-preparation.js";
import { LEGACY_WORKSPACE_RECOVERY_SCHEMA, SETTLED_WORKSPACE_RESTORE_HOLD_KEY } from "./workspace-restore-recovery-state.js"; import { LEGACY_WORKSPACE_RECOVERY_SCHEMA, SETTLED_WORKSPACE_RESTORE_HOLD_KEY } from "./workspace-restore-recovery-state.js";
import { ENVIRONMENT_LEASE_POLICIES, EXECUTION_RECONCILIATION_CAUSES, readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type EnvironmentLeasePolicy, import { ENVIRONMENT_LEASE_POLICIES, EXECUTION_RECONCILIATION_CAUSES, readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type EnvironmentLeasePolicy,
type WorkspaceSourceVerificationFailure } from "@paperclipai/shared"; type WorkspaceSourceVerificationFailure } from "@paperclipai/shared";
@@ -109,6 +110,7 @@ export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRec
|| remoteAuthority.location.remoteCwd !== remoteReference.remoteCwd || remoteAuthority.location.remoteCwd !== remoteReference.remoteCwd
|| (remoteReference.state === "prepared" ? !remoteAuthority.dirty || remoteAuthority.writerRunId !== run.id || (remoteReference.state === "prepared" ? !remoteAuthority.dirty || remoteAuthority.writerRunId !== run.id
: remoteAuthority.dirty || remoteAuthority.writerRunId !== null || remoteAuthority.snapshotId !== remoteReference.snapshotId))) { : remoteAuthority.dirty || remoteAuthority.writerRunId !== null || remoteAuthority.snapshotId !== remoteReference.snapshotId))) {
if (await isAbandonedPreparedRemoteWorkspace(db, run)) throw new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy");
throw invalid("workspace_reference_mismatch"); throw invalid("workspace_reference_mismatch");
} }
if (remoteReference?.state === "saved") { if (remoteReference?.state === "saved") {