mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
fix(recovery): treat a retired prepared remote epoch as nothing to recover (#20491)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A restart must never leave a task stuck: an interrupted run must resume on its own or have a working Retry > - When a graceful restart interrupts a remote-workspace run that is still preparing, the run keeps a `prepared` reference to its workspace epoch, and a preserve-without-replay hold blocks the task > - The unsaved-files reconciler then retires that epoch with `abandonUnverifiedRemoteWorkspaceEpoch` and writes a `paperclip.remote-workspace-abandoned-epoch.v1` receipt. The workspace content is the last committed save again > - Recovery never read that receipt. The held-preparation reconciler waited for a repair receipt that could never exist, and restore recovery refused the source as `source_binding_invalid`. The task stayed blocked, and a UI Retry returned 202 and started nothing > - This pull request adds one strict shared rule: an abandoned-epoch receipt for the run's exact prepared epoch and allocation means there is nothing to recover. Every recovery path now uses it > - The benefit is that the task resumes automatically from the last committed save after the restart, whichever recovery path owns the hold ## Linked Issues or Issue Description Refs #20072 (replay a preparation a restart interrupted), #20299 (storage outage waits). **What existing behavior does this improve?** Recovery of a remote-workspace run that a graceful restart interrupted during preparation, after its unsaved epoch was retired. **Subsystem affected** Server: recovery (held-preparation reconciler, workspace restore recovery, the workspace repair check). **Current behavior** These are Daytona R1 restart-chaos waves (Paperclip-sized workspace, `--storage s3`, K2 graceful restart at +110 s, during the ~730 MiB first save): - `w-20261110T052533-s8805` and `w-20261110T062703-s8805` (ST2 slow store), held-preparation path: both runs ended `interrupted` in `executionStage: preparing`. The reference was `{ state: "prepared", epoch: 1 }`, and the workspace had moved to `{ epoch: 2, location: null, dirty: false, writerRunId: null }` with an abandoned-epoch receipt for epoch 1. The `legacy_interrupted_preparation_hold` reconciler never settled the hold (`I2_settled` violations), because `isInterruptedNativePreparation` requires `hasVerifiedWorkspaceRepair`, and that check accepts only a `saved` reference. Grader: `R3:retry_no_new_run`. - `w-20261110T091639-s11936` (ST1 outage), restore-recovery path: the run also had `workspaceRestoreFailure: "restore_failed"` and `workspaceRestoreRecovery`. Restore recovery owns that hold. `inspectWorkspaceRecoverySource` threw `source_binding_invalid` / `workspace_reference_mismatch` with retry class `service_required`, and the hold says Retry "succeeds only after Paperclip is updated". Grader: `R3:retry_no_new_run`. **Proposed behavior** `isAbandonedPreparedRemoteWorkspace(db, run)` is true only when all of these hold: - The run has a `prepared` remote reference and no native workspace sync reference. - The workspace authority is at a later epoch, and the run is not its writer. - The workspace has an abandoned-epoch receipt for the same run, the same epoch, and the same lease, provider lease and remote cwd. That rule is used in three places: - `hasVerifiedWorkspaceRepair` (`workspace-recovery-receipt.ts`): the retired epoch counts as repaired. This covers held-preparation replay, the restore-failure check in `settleInterruptedNativeBootstrap`, the execution blocker, and explicit Retry admission. `verifiedWorkspaceRepairReceipt`, which returns a receipt id, is unchanged. - `inspectWorkspaceRecoverySource` (`workspace-recovery-source.ts`): the retired source is reported as `source_unavailable` / `use_alternative_strategy`, not as a binding failure that needs a service update. - `heldInterruptedPreparationWhere`: a restore-recovery hold is listed for the held-preparation reconciler only when the matching receipt exists (SQL pre-filter). The settle then re-checks the full rule in JS before it acts. **Reason and benefit** The retired epoch held only preparation files: the hold proof requires that no provider work started. Its unsaved files were already given up by the reconciler that wrote the receipt. Waiting for a repair, or for a service update, cannot finish. ## What Changed - New `server/src/services/remote-workspace-abandoned-preparation.ts`: `isAbandonedPreparedRemoteWorkspace` (the strict rule) and `abandonedPreparedRemoteWorkspaceExists` (its SQL pre-filter). - `workspace-recovery-receipt.ts`: `hasVerifiedWorkspaceRepair` also accepts the rule. - `workspace-recovery-source.ts`: a retired prepared source is `source_unavailable` / `use_alternative_strategy`. - `interrupted-preparation-hold-settlement.ts`: restore-recovery holds are listed only with a matching receipt. - `interrupted-preparation-hold-settle.test.ts`: held-preparation replay (before and after the first save committed); a restore-recovery hold in the exact shape saved by the previous build (`restore_failed`, `workspaceRestoreRecovery`, exhausted `source_binding_invalid`) that now settles and replays; inspection reporting `source_unavailable`; and the rule refusing a workspace still at the run's epoch, a run still the writer, a native sync reference, another run's receipt, or another allocation's receipt. Not changed: `run-completion.ts` and the graceful-drain path in `recovery.ts`. ## Verification - `cd server && npx vitest run src/__tests__/interrupted-preparation-hold-settle.test.ts src/services/task-retry.test.ts`: 101 pass. - Red before: with the three recovery files at master, 4 tests fail (held-preparation replay ×2, the restore-recovery replay, and the inspection result). Green after. - Neighbouring recovery suites: see the comment below. - `cd server && npx tsc --noEmit -p tsconfig.json`: pass. - Chaos rerun on a build with this change: see the comment below. ## Risks - Previous-release story (upgrade-safe run state): no persisted shape changes. Holds, run references and receipts written by the previous build are read as they are. The restore-recovery test uses the exact shape the previous build saved. A task already stuck in that state settles on the next reconciler pass after deploy. - The rule needs an exact receipt (run, epoch, lease, provider lease, cwd) and a later workspace epoch, so a real recovery is never skipped. Any mismatch keeps today's behavior. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c2423e30e8
commit
23fa37f5bb
5 files changed
+138
-6
No files matched your search
@@ -11,6 +11,7 @@ import {
|
||||
createDb,
|
||||
environmentLeases,
|
||||
environments,
|
||||
executionWorkspaces,
|
||||
heartbeatRunEvents,
|
||||
heartbeatRuns,
|
||||
issueRecoveryActions,
|
||||
@@ -20,6 +21,8 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro
|
||||
import { currentProcessSpaceHash } from "../services/controller-process-identity.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { settleHeldInterruptedPreparation } from "../services/interrupted-preparation-hold-settlement.js";
|
||||
import { isAbandonedPreparedRemoteWorkspace } from "../services/remote-workspace-abandoned-preparation.js";
|
||||
import { inspectWorkspaceRecoverySource } from "../services/workspace-recovery-source.js";
|
||||
import { legacyControllerBootId } from "../services/legacy-controller-lease.js";
|
||||
import { createOwnershipReportState, runOwnershipReconciliationPass } from "../services/recovery/ownership-invariant-report.js";
|
||||
|
||||
@@ -67,6 +70,7 @@ suite("held interrupted preparations on the shared ownership path", () => {
|
||||
await db.delete(activityLog);
|
||||
await db.delete(issueRecoveryActions);
|
||||
await db.delete(environmentLeases);
|
||||
await db.delete(executionWorkspaces);
|
||||
await db.delete(heartbeatRunEvents);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(issues);
|
||||
@@ -171,6 +175,90 @@ suite("held interrupted preparations on the shared ownership path", () => {
|
||||
expect(await hold(f.holdId)).toEqual(before);
|
||||
});
|
||||
|
||||
async function preparedRemoteWorkspace(runId: string, issueId: string, options: { receipt?: Record<string, unknown>;
|
||||
savedSnapshotId?: string | null; authority?: Record<string, unknown>; profile?: Record<string, unknown> } = {}) {
|
||||
const workspaceId = randomUUID(), leaseId = randomUUID(), providerLeaseId = `sandbox-${leaseId}`;
|
||||
const remoteCwd = `/home/daytona/paperclip-workspace/.paperclip-workspaces/${workspaceId}-1-${leaseId}`;
|
||||
const location = { environmentId, leaseId, providerLeaseId, remoteCwd };
|
||||
const source = { kind: "git", repoUrl: "https://github.com/paperclipai/paperclip.git", ref: "a".repeat(40),
|
||||
branchMode: "create", branchName: "IPH-1-change" };
|
||||
const authority = { schema: "paperclip.remote-workspace.v1", epoch: 2, source, location: null, dirty: false, writerRunId: null,
|
||||
snapshotId: options.savedSnapshotId ?? null, ...options.authority };
|
||||
const receipts = [{ schema: "paperclip.remote-workspace-abandoned-epoch.v1", epoch: 1, runId,
|
||||
snapshotId: options.savedSnapshotId ?? null, location, providerRef: remoteCwd, abandonedAt: new Date().toISOString(), ...options.receipt }];
|
||||
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId, mode: "isolated_workspace",
|
||||
strategyType: "git_worktree", name: "Remote workspace", status: "active", providerType: "git_worktree",
|
||||
metadata: { remoteWorkspace: authority, remoteWorkspaceAbandonedEpochs: receipts } });
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" },
|
||||
remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", state: "prepared", workspaceId, epoch: 1, leaseId,
|
||||
providerLeaseId, remoteCwd, snapshotId: null }, ...options.profile } }).where(eq(heartbeatRuns.id, runId));
|
||||
return { leaseId };
|
||||
}
|
||||
|
||||
const runRow = async (runId: string) => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]!;
|
||||
|
||||
it("replays a restore-recovery hold saved by an older build once the workspace retired the run's epoch", async () => {
|
||||
heartbeatService(db);
|
||||
const f = await interruptedPreparation();
|
||||
const { leaseId } = await preparedRemoteWorkspace(f.runId, f.issueId);
|
||||
const restore = { workspaceRestoreFailure: "restore_failed",
|
||||
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [leaseId] } };
|
||||
const run = await runRow(f.runId);
|
||||
await db.update(heartbeatRuns).set({ resultJson: { ...run.resultJson, ...restore } }).where(eq(heartbeatRuns.id, f.runId));
|
||||
const saved = await hold(f.holdId);
|
||||
await db.update(issueRecoveryActions).set({ evidence: { ...saved.evidence, ...restore,
|
||||
systemRecovery: { schema: "paperclip.system-recovery.v1", policyVersion: 1, state: "exhausted", phase: "verify_execution",
|
||||
sourceRunId: f.runId, attempts: 1, retriesUsed: 0, revision: 5, nextAttemptAt: null,
|
||||
failure: { code: "source_binding_invalid", retryClass: "service_required", sourceVerification: { reason: "workspace_reference_mismatch" } } } } })
|
||||
.where(eq(issueRecoveryActions.id, f.holdId));
|
||||
expect(await pass()).toMatchObject({ inProgress: 1, settled: 1 });
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "in_progress" });
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.runId))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("reports a retired prepared epoch to workspace recovery as an unavailable source, not a binding failure", async () => {
|
||||
const f = await interruptedPreparation();
|
||||
await preparedRemoteWorkspace(f.runId, f.issueId);
|
||||
await expect(inspectWorkspaceRecoverySource(db, { companyId, issueId: f.issueId, runId: f.runId, actionId: f.holdId }))
|
||||
.rejects.toMatchObject({ code: "source_unavailable", retryClass: "use_alternative_strategy" });
|
||||
});
|
||||
|
||||
const proofCases: Array<[string, { authority?: Record<string, unknown>; profile?: Record<string, unknown>; writer?: boolean }, boolean]> = [
|
||||
["the workspace is still at the run's epoch", { authority: { epoch: 1 } }, false],
|
||||
["the run is still the workspace writer", { writer: true }, false],
|
||||
["the run also has a native workspace sync reference", { profile: { nativeWorkspaceSync: { state: "prepared" } } }, false],
|
||||
["the receipt matches the run's exact epoch and allocation", {}, true],
|
||||
];
|
||||
it.each(proofCases)("treats the prepared workspace as retired only when proven: %s", async (_label, options, expected) => {
|
||||
const f = await interruptedPreparation();
|
||||
const authority = options.writer ? { writerRunId: f.runId } : options.authority;
|
||||
await preparedRemoteWorkspace(f.runId, f.issueId, { profile: options.profile, authority });
|
||||
expect(await isAbandonedPreparedRemoteWorkspace(db, await runRow(f.runId))).toBe(expected);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["before its first save finished", null],
|
||||
["after its first save committed", randomUUID()],
|
||||
] as const)("replays a remote preparation whose unsaved epoch the workspace already retired %s", async (_label, savedSnapshotId) => {
|
||||
heartbeatService(db);
|
||||
const f = await interruptedPreparation();
|
||||
await preparedRemoteWorkspace(f.runId, f.issueId, { savedSnapshotId });
|
||||
expect(await pass()).toMatchObject({ inProgress: 1, settled: 1 });
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "in_progress" });
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.runId))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["the retired epoch belongs to another run", { receipt: { runId: randomUUID() } }],
|
||||
["the retired epoch was another allocation", { receipt: { location: { leaseId: randomUUID() } } }],
|
||||
] as const)("keeps a remote preparation hold when %s", async (_label, options) => {
|
||||
const f = await interruptedPreparation();
|
||||
await preparedRemoteWorkspace(f.runId, f.issueId, options);
|
||||
const before = await hold(f.holdId);
|
||||
expect((await pass())?.settled ?? 0).toBe(0);
|
||||
expect(await hold(f.holdId)).toEqual(before);
|
||||
});
|
||||
|
||||
it("settles exactly once when two processes judge the same hold", async () => {
|
||||
heartbeatService(db);
|
||||
const f = await interruptedPreparation();
|
||||
|
||||
@@ -6,6 +6,7 @@ import { logger } from "../middleware/logger.js";
|
||||
import type { OwnerProof } from "./owner-proof.js";
|
||||
import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
|
||||
import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js";
|
||||
import { abandonedPreparedRemoteWorkspaceExists } from "./remote-workspace-abandoned-preparation.js";
|
||||
|
||||
/**
|
||||
* `legacy_interrupted_preparation_hold`: a run a graceful shutdown interrupted
|
||||
@@ -19,9 +20,11 @@ import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js
|
||||
*
|
||||
* Only holds the system recorded itself are listed: an operator decision (an
|
||||
* owner-gone attestation, an explicit user continuation, an assigned owner) is
|
||||
* never reopened, and a workspace-restore hold (restore failure or recovery
|
||||
* evidence) belongs to restore recovery, not to this kind. A run with a lease still cleaning up is not listed yet; the
|
||||
* next pass lists it once the lease settles.
|
||||
* never reopened. A workspace-restore hold (restore failure or recovery
|
||||
* evidence) belongs to restore recovery, unless the workspace already retired
|
||||
* the run's exact prepared epoch (`isAbandonedPreparedRemoteWorkspace`): then
|
||||
* nothing is left to restore and this kind settles it. A run with a lease still
|
||||
* cleaning up is not listed yet; the next pass lists it once the lease settles.
|
||||
*/
|
||||
export function heldInterruptedPreparationWhere() {
|
||||
return and(inArray(heartbeatRuns.status, ["interrupted", "failed"]), eq(heartbeatRuns.runtimeMode, "legacy"),
|
||||
@@ -32,8 +35,9 @@ export function heldInterruptedPreparationWhere() {
|
||||
and hold.status = 'resolved' and hold.outcome = 'blocked' and hold.owner_user_id is null
|
||||
and hold.evidence->'automaticRecovery'->>'policy' = 'preserve_without_replay_v1'
|
||||
and hold.evidence->'automaticRecovery'->>'replay' = 'blocked'
|
||||
and not (hold.evidence ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation',
|
||||
'workspaceRestoreFailure', 'workspaceRestoreRecovery']))`,
|
||||
and not (hold.evidence ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation'])
|
||||
and (not (hold.evidence ?| array['workspaceRestoreFailure', 'workspaceRestoreRecovery'])
|
||||
or ${abandonedPreparedRemoteWorkspaceExists()}))`,
|
||||
sql`not exists (select 1 from environment_leases lease where lease.company_id = ${heartbeatRuns.companyId}
|
||||
and lease.heartbeat_run_id = ${heartbeatRuns.id}
|
||||
and (lease.released_at is null or lease.status = 'pending_cleanup' or lease.cleanup_status = 'failed'))`);
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { executionWorkspaces, heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared";
|
||||
import { REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA } from "./remote-workspace-epoch-abandonment.js";
|
||||
|
||||
type Run = Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "runnerProfileJson">;
|
||||
|
||||
export async function isAbandonedPreparedRemoteWorkspace(db: Db, run: Run): Promise<boolean> {
|
||||
if (run.runnerProfileJson?.nativeWorkspaceSync != null) return false;
|
||||
const parsed = remoteWorkspaceRunReferenceSchema.safeParse(run.runnerProfileJson?.remoteWorkspace);
|
||||
if (!parsed.success || parsed.data.state !== "prepared") return false;
|
||||
const reference = parsed.data;
|
||||
const [workspace] = await db.select({ metadata: executionWorkspaces.metadata }).from(executionWorkspaces).where(and(
|
||||
eq(executionWorkspaces.companyId, run.companyId), eq(executionWorkspaces.id, reference.workspaceId)));
|
||||
const authority = readRemoteWorkspaceAuthority(workspace?.metadata);
|
||||
if (!authority || authority.epoch <= reference.epoch || authority.writerRunId === run.id) return false;
|
||||
const receipts = Array.isArray(workspace?.metadata?.remoteWorkspaceAbandonedEpochs)
|
||||
? workspace!.metadata!.remoteWorkspaceAbandonedEpochs as Array<Record<string, unknown>> : [];
|
||||
return receipts.some(receipt => {
|
||||
const location = receipt?.location as Record<string, unknown> | undefined;
|
||||
return receipt?.schema === REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA && receipt.runId === run.id
|
||||
&& receipt.epoch === reference.epoch
|
||||
&& location?.leaseId === reference.leaseId && location?.providerLeaseId === reference.providerLeaseId
|
||||
&& location?.remoteCwd === reference.remoteCwd;
|
||||
});
|
||||
}
|
||||
|
||||
export function abandonedPreparedRemoteWorkspaceExists() {
|
||||
const reference = sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'`;
|
||||
return sql`(${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync' is null and ${reference}->>'state' = 'prepared'
|
||||
and exists (select 1 from execution_workspaces workspace
|
||||
cross join lateral jsonb_array_elements(case when jsonb_typeof(workspace.metadata->'remoteWorkspaceAbandonedEpochs') = 'array'
|
||||
then workspace.metadata->'remoteWorkspaceAbandonedEpochs' else '[]'::jsonb end) receipt
|
||||
where workspace.company_id = ${heartbeatRuns.companyId} and workspace.id::text = ${reference}->>'workspaceId'
|
||||
and receipt->>'schema' = ${REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA} and receipt->>'runId' = ${heartbeatRuns.id}::text
|
||||
and receipt->'epoch' = ${reference}->'epoch'))`;
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import { readMigratedLegacyWorkspaceSource } from "./legacy-workspace-recovery-m
|
||||
import { readSnapshot } from "./workspace-recovery-snapshot.js";
|
||||
import { exhaustedSystemRecoveryAction, readSystemRecovery } from "./system-recovery-state.js";
|
||||
import { verifiedWorkspaceSnapshotReuse } from "./remote-workspace-snapshot-reuse-receipt.js";
|
||||
import { isAbandonedPreparedRemoteWorkspace } from "./remote-workspace-abandoned-preparation.js";
|
||||
|
||||
export const workspaceRecoverySchema = z.object({
|
||||
schema: z.literal("paperclip.workspace-recovery.v1"),
|
||||
@@ -176,7 +177,7 @@ export async function verifiedWorkspaceRepairReceipt(db: Db, run: Pick<typeof he
|
||||
}
|
||||
|
||||
export async function hasVerifiedWorkspaceRepair(...args: Parameters<typeof verifiedWorkspaceRepairReceipt>) {
|
||||
return (await verifiedWorkspaceRepairReceipt(...args)) !== null;
|
||||
return (await verifiedWorkspaceRepairReceipt(...args)) !== null || await isAbandonedPreparedRemoteWorkspace(...args);
|
||||
}
|
||||
|
||||
/** Read-only historical bytes have no execution authority. Their immutable
|
||||
|
||||
@@ -7,6 +7,7 @@ import { readNativeWorkspaceSyncReference } from "./native-runtime/native-worksp
|
||||
import { readSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js";
|
||||
import type { NativeWorkspaceFinalizationOwnership } from "./native-runtime/native-workspace-finalization-ownership.js";
|
||||
import { readStopOnlyCleanup } from "./sandbox-stop-and-retain.js";
|
||||
import { isAbandonedPreparedRemoteWorkspace } from "./remote-workspace-abandoned-preparation.js";
|
||||
import { LEGACY_WORKSPACE_RECOVERY_SCHEMA, SETTLED_WORKSPACE_RESTORE_HOLD_KEY } from "./workspace-restore-recovery-state.js";
|
||||
import { ENVIRONMENT_LEASE_POLICIES, EXECUTION_RECONCILIATION_CAUSES, readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type EnvironmentLeasePolicy,
|
||||
type WorkspaceSourceVerificationFailure } from "@paperclipai/shared";
|
||||
@@ -109,6 +110,7 @@ export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRec
|
||||
|| remoteAuthority.location.remoteCwd !== remoteReference.remoteCwd
|
||||
|| (remoteReference.state === "prepared" ? !remoteAuthority.dirty || remoteAuthority.writerRunId !== run.id
|
||||
: remoteAuthority.dirty || remoteAuthority.writerRunId !== null || remoteAuthority.snapshotId !== remoteReference.snapshotId))) {
|
||||
if (await isAbandonedPreparedRemoteWorkspace(db, run)) throw new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy");
|
||||
throw invalid("workspace_reference_mismatch");
|
||||
}
|
||||
if (remoteReference?.state === "saved") {
|
||||
|
||||
Reference in new issue
Block a user