fix(recovery): end a remote workspace epoch whose writer died during preparation (#20346)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task with a remote workspace prepares that workspace before the
run selects its runtime, and preparation marks the workspace epoch dirty
with the run as its writer
> - If the server is killed during preparation, the run never selected a
runtime and never started provider work, but the dirty epoch remains
> - Failed-workspace preservation, the path that resolves a dead
writer's dirty epoch, applies to native runs only, so nothing owned this
epoch
> - The dirty epoch kept the dead run's sandbox lease as a pending
export and held the workspace against the task's retry, so the task
never recovered
> - This pull request lets the ownership reconciler end such an epoch at
the last committed save
> - The benefit is that a restart during workspace preparation no longer
wedges the task

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave on Daytona (a remote workspace that clones a
real repository), the controller was killed while each run's first
workspace save was in flight. That save is now settled by #20326. Each
run ended `process_lost` with a bounded retry, and both runs were still
in `execution_stage: preparing` with `runtime_mode_resolved_at: null`.
The workspace authority stayed `dirty: true`, `snapshotId: null`, with
the dead run as writer. The dead run's environment lease was proven
owner-gone but was refused by its settle as `workspace_export_pending`,
and the reconciler reported `I2_settled` violations. The retries stayed
`queued` until the test harness deleted the environment 21 minutes
later. Then they failed `remote_workspace_execution_required`, because
the environment was gone.

**Expected behavior**
When a writer dies before runtime selection, its unsaved files are only
the restored source and preparation output, because no agent turn ran.
Recovery must end that epoch at the last committed save, so the lease
settles and the retry restores the workspace into a new directory. A
writer that reached runtime selection must keep its existing recovery.

**Steps to reproduce**
1. Run a task whose agent uses Paperclip Runner with a remote sandbox
workspace.
2. Kill the server while the run is still preparing the workspace (for
example during its first save).
3. After the restart, the run ends `process_lost` and its retry is
queued, but the workspace stays dirty and owned by the dead run, the
lease stays active, and the retry never starts.

**Paperclip version or commit**
`master` at 251c6d5c69 (after #20326).

Refs #20326 (settles the save that was in flight when the controller
died).

## What Changed

- New `settlePreparationOnlyRemoteWorkspaceEpoch`
(`recovery/remote-workspace-unsaved-files-settlement.ts`), wired as the
settle of the existing `remote_workspace_unsaved_files` ownership kind.
Once the writer run is proven gone, it ends the epoch with the existing
`abandonUnverifiedRemoteWorkspaceEpoch` only when all of these are true:
  - the run is terminal;
  - its runtime was never resolved (`runtime_mode_resolved_at` is null);
  - it ended in `execution_stage: preparing`;
  - it has no native finalization;
- durable records show no work began: no `adapter.invoke` or
`adapter.progress` events, no `turn.*`, `provider.*` or
`native.process_*` events, and no recorded process id, process group,
native session or runner instance on the run;
  - the workspace authority is still dirty, with this run as writer;
  - no save for that epoch is still `running` or `pending`.
The epoch then advances with no writer and no location, and the
abandoned epoch is recorded in `remoteWorkspaceAbandonedEpochs` with an
activity entry.
- After that, the lease no longer matches a prepared source, so the
existing environment lease settle releases it and the workspace writer
hold is free for the retry.
- The ownership inventory row for this kind names the new settle.

## Verification

- Red before the fix, green after: `ends a dirty remote epoch whose
writer died during preparation, and keeps every other writer's files`
(`ownership-reconciliation-report.test.ts`). On master it settles 0
epochs. With the fix it ends only the preparation-only epoch, and keeps
the dirty epoch of a writer whose runtime was resolved, a writer that
reached execution, a live writer, and a writer with a save still in
flight.
- Red before the fix, green after, end to end: `lets a retry prepare the
workspace again after its first writer was killed during preparation`
(`remote-workspace-save-concurrency.test.ts`). It runs the real reserve
and prepare. The first writer's initial save is cut off, which leaves a
first epoch with no committed save (`snapshotId: null`). The run is
reaped as `process_lost` while still preparing, with its lease left
active. Two reconciliation passes then end the epoch and release the
lease. The retry reserves the workspace and prepares it again from the
project base in a new directory, then commits its own initial save. On
master, the epoch stays dirty.
- The safety cases also keep a preparation-stage writer with an
`adapter.invoke` event, and one with a recorded process id.
- Red before the fix, green after: `releases the lease and frees the
workspace for the retry once a preparation-only epoch ends`. Before the
passes, the lease counts as a prepared source and the workspace writer
hold is `held` for a retry. After two reconciliation passes, the lease
is no longer active, the prepared source is gone, and the writer hold is
`free`. On master the prepared source remains.
- The reconciler, retained-source fallback, remote workspace lifecycle,
save-concurrency, warm-handoff recovery, and retained orphan sandbox
suites pass (139 tests, 1 environment-gated file skipped). After
rebasing onto #20326, the reconciler, save-concurrency, retained-source
fallback, and controller instance lease suites pass (91 tests).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.
- Evidence: the runs in the wave's kept database all had
`runtime_mode_resolved_at: null` and `execution_stage: preparing`. Their
`runtime_mode: legacy` is only the column default, not a resolved
runtime.

## Risks

- This ends unsaved files without saving them. That is safe only because
no provider turn can have started before runtime selection, and the
guard requires the writer never to have resolved a runtime. A writer
that did resolve one keeps its existing preservation path.
- The abandoned root stays in the sandbox and is not reused. The next
preparation restores the last committed save, or the source, into a new
directory.
- No schema or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, local database inspection).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/plans/2026-11-07-ownership-inventory.md

@@ -351,7 +351,7 @@ Scope: workspace operations (#19807 intents), runtime leases, runtime services,
 | Execution workspace runtime lease | `execution_workspace_runtime_leases` (row existence) | workspace-runtime-leases.ts:194-209 (create), :235-247 (renew), :255-273 (reclaim). Called only from routes/execution-workspaces.ts:567 | row present | `owner_key` `issue:`/`run:`/`agent:` (+ FK columns). No process identity | `release` :291-313. In production it is called only with `force` on archive (routes/execution-workspaces.ts:1270). Otherwise the lease is reclaimed lazily at the next claim (:250-283) | `evaluateStaleOwner` :85-119: issue hidden or terminal, run terminal, or **TTL/age** 30 min (:39, :90). An agent-scoped owner has TTL only (:116-118) | ownership | 1, 2 | A, C |
 | Workspace runtime service (local process) | `workspace_runtime_services.status` | workspace-runtime.ts (insert line unverified) | `provisioning`, `starting`, `running` | `startedByRunId`, `ownerAgentId`. The pid lives in a local service registry file, not on the row | Startup `reconcilePersistedRuntimeServicesOnStartup` workspace-runtime.ts:8636, which adopts or stops each service (index.ts:1101, not awaited) | Registry record match (workspace-runtime.ts:8781-8797) plus `isPidAlive` local-service-supervisor.ts:218, :245-250. Same host only | ownership | 1, 2 | A, C |
 | Remote workspace writer lane | `execution_workspaces.metadata.remoteWorkspace.writerRunId` (+ `epoch`) | remote-workspaces.ts:230 (epoch+1), :243 (new) | non-null writerRunId | writer run id + epoch | Final save sets it to null: remote-workspaces.ts:780-781. Migration: remote-workspace-migration.ts:516 | `readRemoteWorkspaceWriterHold` remote-workspace-writer-hold.ts:25-79: run status (:42), lease termination/stop receipts (:53-56), recovery-save progress (:63-76). The waiter gives up after **TTL/age** 60 min of writer inactivity (heartbeat/workspace-preparation.ts:1050-1051, run-dispatch/domain/wake-context.ts:36) | ownership | 1, 2, 3, 7 | A, E |
-| Remote workspace unsaved-files flag | `execution_workspaces.metadata.remoteWorkspace.dirty` | remote-workspaces.ts:780-781 (`dirty: boundary !== "final"`) | `dirty=true` | writerRunId (same object) | Final save :781. Failed-run recovery native-runtime/native-failed-workspace-recovery.ts:52-60 (pins/saves) | Writer-hold rule (above). Lease termination receipt | workspace-preservation | 1, 2 | C |
+| Remote workspace unsaved-files flag | `execution_workspaces.metadata.remoteWorkspace.dirty` | remote-workspaces.ts:780-781 (`dirty: boundary !== "final"`) | `dirty=true` | writerRunId (same object) | Final save :781. Failed-run recovery native-runtime/native-failed-workspace-recovery.ts:52-60 (pins/saves). A writer that died during preparation (terminal, no resolved runtime, no native finalization, no save in flight) is ended at the last committed save by the ownership reconciler (`settlePreparationOnlyRemoteWorkspaceEpoch`) | Writer-hold rule (above). Lease termination receipt | workspace-preservation | 1, 2 | C |
 | Remote snapshot ledger | `execution_workspace_snapshots.status` | remote-workspace-snapshot-storage.ts:73-83 (`pending`, deleteAfter = now + 7 d) | `pending`, `uploaded` | `source_operation_id`, `source_run_id` | Commit remote-workspaces.ts:782-784. Orphans become `delete_requested` by **TTL/age** 7 d from createdAt (remote-workspace-snapshot-retention.ts:9, :113-122), deferred while ops are running (:36) | none (age) | workspace-preservation | 2 | C |
 | Git restore point | `execution_workspace_restore_points.status` | git-restore-point-storage.ts:146-155 | `pending` (and `expiring`) | none (workspace + generation) | → `active` :187-188. → `superseded` :141-145, :182-185, :208-216 | none found | workspace-preservation | 1, 2 | C |
 | Git cache expiration intent | `execution_workspaces.metadata.gitCacheExpiration.completed` | git-workspace-expiration.ts:384-390, :401-406 | `completed=false` | `lockToken` (random, :384, :405). File locks git-workspace-expiration-locks.ts:72-76 | Deleting the key on completion git-workspace-expiration.ts:454. Restore/rehome deletes it (execution-workspaces.ts:4516, isolated-repository-rehome.ts:485) | Transaction advisory lock `execution_workspace_lifecycle:` (:302-303). Resumable by any worker using the same token (recover: git-workspace-expiration-locks.ts:57-63) | workspace-preservation | 1 | B, C |

server/src/__tests__/ownership-reconciliation-report.test.ts

@@ -12,6 +12,8 @@ import {
   companies,
   createDb,
   environmentLeases,
+  environments,
+  executionWorkspaces,
   heartbeatRunEvents,
   heartbeatRuns,
   issueComments,
@@ -34,6 +36,9 @@ import {
 } from "../services/recovery/ownership-invariant-report.js";
 import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js";
 import { trackRemoteWorkspaceSaveInvocationForTests } from "../services/remote-workspaces.js";
+import { readRemoteWorkspaceAuthority } from "@paperclipai/shared";
+import { preparedRemoteWorkspaceSource } from "../services/remote-workspace-lifecycle.js";
+import { readRemoteWorkspaceWriterHold } from "../services/remote-workspace-writer-hold.js";
 import { workspaceOperationService } from "../services/workspace-operations.js";
 
 const external = resolveExternalTestDatabaseUrl("PAPERCLIP_HISTORY_TEST_DATABASE_URL");
@@ -146,7 +151,7 @@ describe("report-only ownership reconciliation", () => {
     expect(new Set(names).size).toBe(names.length);
     expect(names.length).toBeGreaterThanOrEqual(40);
     expect(OWNABLE_KINDS.filter(kind => kind.settle !== null).map(kind => kind.id))
-      .toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "workspace_command_operation", "remote_snapshot_save", "execution_workspace_runtime_lease", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "legacy_interrupted_preparation_hold", "issue_execution_lock", "issue_checkout_lock", "agent_lifecycle_lease"]);
+      .toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "workspace_command_operation", "remote_snapshot_save", "execution_workspace_runtime_lease", "remote_workspace_unsaved_files", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "legacy_interrupted_preparation_hold", "issue_execution_lock", "issue_checkout_lock", "agent_lifecycle_lease"]);
     expect(OWNABLE_KINDS.filter(kind => !kind.listInProgress).every(kind => kind.coverageGap)).toBe(true);
   });
 
@@ -452,6 +457,76 @@ describe("report-only ownership reconciliation", () => {
     expect(liveRun.status).toBe("running");
   });
 
+  it("ends a dirty remote epoch whose writer died during preparation, and keeps every other writer's files", async () => {
+    const runs = { preparing: randomUUID(), resolved: randomUUID(), executing: randomUUID(), live: randomUUID(), saving: randomUUID(),
+      invoked: randomUUID(), spawned: randomUUID() };
+    const run = (id: string, status: string, extra: Record<string, unknown> = {}) => ({ id, companyId, agentId, status,
+      invocationSource: "manual", executionStage: "preparing", ...(status === "running" ? {} : { finishedAt: minutesAgo(2), errorCode: "process_lost" }), ...extra });
+    await db.insert(heartbeatRuns).values([
+      run(runs.preparing, "failed"),
+      run(runs.resolved, "failed", { runtimeMode: "native", runtimeModeResolvedAt: minutesAgo(3) }),
+      run(runs.executing, "failed", { executionStage: "executing" }),
+      run(runs.live, "running"),
+      run(runs.saving, "failed"),
+      run(runs.invoked, "failed"),
+      run(runs.spawned, "failed", { processPid: 4_194_303 }),
+    ]);
+    await db.insert(heartbeatRunEvents).values({ companyId, runId: runs.invoked, agentId, seq: 1, eventType: "adapter.invoke" });
+    const workspaceFor = Object.fromEntries(Object.keys(runs).map(name => [name, randomUUID()])) as Record<keyof typeof runs, string>;
+    await db.insert(executionWorkspaces).values(Object.entries(runs).map(([name, writerRunId]) => ({
+      id: workspaceFor[name as keyof typeof runs], companyId, mode: "isolated_workspace", strategyType: "test", name,
+      metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: null,
+        source: { kind: "git", repoUrl: "https://example.invalid/repo.git", ref: null }, dirty: true, writerRunId,
+        location: { environmentId: randomUUID(), leaseId: randomUUID(), providerLeaseId: "sandbox-1", remoteCwd: "/workspace/root" } } },
+    })));
+    await db.insert(workspaceOperations).values({ companyId, executionWorkspaceId: workspaceFor.saving, heartbeatRunId: runs.saving,
+      phase: "remote_workspace_snapshot", status: "running", metadata: { epoch: 1, boundary: "initial" } });
+    const authorities = async () => Object.fromEntries((await db.select().from(executionWorkspaces))
+      .filter(row => Object.values(workspaceFor).includes(row.id))
+      .map(row => [row.name, readRemoteWorkspaceAuthority(row.metadata)]));
+
+    const first = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
+    expect(kindReport(first, "remote_workspace_unsaved_files")).toMatchObject({ settled: 1 });
+    const after = await authorities();
+    expect(after.preparing).toMatchObject({ epoch: 2, dirty: false, writerRunId: null, location: null, snapshotId: null });
+    for (const kept of ["resolved", "executing", "live", "saving", "invoked", "spawned"] as const)
+      expect(after[kept], kept).toMatchObject({ epoch: 1, dirty: true, writerRunId: runs[kept] });
+    const unsavedFiles = OWNABLE_KINDS.find(kind => kind.id === "remote_workspace_unsaved_files")!;
+    expect((await recheckSettlingKinds(db, { kinds: [unsavedFiles] })).watched).toBe(2);
+    const [abandoned] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceFor.preparing));
+    expect(abandoned.metadata?.remoteWorkspaceAbandonedEpochs).toEqual([expect.objectContaining({ epoch: 1, runId: runs.preparing })]);
+  });
+
+  it("releases the lease and frees the workspace for the retry once a preparation-only epoch ends", async () => {
+    const runId = randomUUID(), issueId = randomUUID(), workspaceId = randomUUID(), leaseId = randomUUID();
+    await db.insert(issues).values({ id: issueId, companyId, title: "Remote task", status: "in_progress" });
+    const [environment] = await db.insert(environments).values({ name: `Sandbox ${randomUUID()}`, driver: "sandbox" }).returning();
+    const location = { environmentId: environment.id, leaseId, providerLeaseId: `sandbox-${randomUUID()}`, remoteCwd: "/workspace/task" };
+    await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, status: "failed", invocationSource: "manual",
+      executionStage: "preparing", finishedAt: minutesAgo(2), errorCode: "process_lost", contextSnapshot: { issueId },
+      runnerProfileJson: { remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", workspaceId, leaseId,
+        providerLeaseId: location.providerLeaseId, remoteCwd: location.remoteCwd, epoch: 1, state: "prepared", snapshotId: null } } });
+    await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId, mode: "isolated_workspace",
+      strategyType: "test", name: "Remote task workspace",
+      metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: null,
+        source: { kind: "git", repoUrl: "https://example.invalid/repo.git", ref: null }, dirty: true, writerRunId: runId, location } } });
+    const [lease] = await db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId: environment.id, issueId,
+      heartbeatRunId: runId, executionWorkspaceId: workspaceId, status: "active", leasePolicy: "ephemeral", provider: "daytona",
+      providerLeaseId: location.providerLeaseId, metadata: { remoteCwd: location.remoteCwd } }).returning();
+    const authority = async () => readRemoteWorkspaceAuthority((await db.select().from(executionWorkspaces)
+      .where(eq(executionWorkspaces.id, workspaceId)))[0]!.metadata)!;
+    expect(await preparedRemoteWorkspaceSource(db, lease)).not.toBeNull();
+    expect(await readRemoteWorkspaceWriterHold(db, { companyId, authority: await authority(), runId: randomUUID() }))
+      .toMatchObject({ kind: "held" });
+
+    await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
+    await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
+    expect(await preparedRemoteWorkspaceSource(db, lease)).toBeNull();
+    expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)))[0]!.status).not.toBe("active");
+    expect(await readRemoteWorkspaceWriterHold(db, { companyId, authority: await authority(), runId: randomUUID() }))
+      .toEqual({ kind: "free" });
+  });
+
   it("settles a workspace command whose process died mid-command, and keeps live or unprovable ones", async () => {
     const run = (id: string, status: string) => ({ id, companyId, agentId, status, invocationSource: "manual",
       ...(status === "running" ? {} : { finishedAt: minutesAgo(2) }) });

server/src/services/recovery/ownership-registry.ts

@@ -38,6 +38,7 @@ import {
 } from "../native-runtime/native-accepted-result-lease-settlement.js";
 import { settleClaimedWakeRequest } from "./wake-request-claimed-settlement.js";
 import { settleAbandonedRemoteSnapshotSave, settleAbandonedWorkspaceCommandOperation, settleAbandonedWorkspaceFinalizeAttempt } from "./workspace-finalize-attempt-settlement.js";
+import { settlePreparationOnlyRemoteWorkspaceEpoch } from "./remote-workspace-unsaved-files-settlement.js";
 import { LOCAL_WORKSPACE_COMMAND_PHASES, runtimeControlOperationOwnerStamp, settleRuntimeControlOperation } from "../workspace-operations.js";
 import { copybackOwnerHoldOwner, settleCopybackOwnerHold } from "./copyback-owner-hold.js";
 import { applyObservedFenceProviderProof, observedWorkspaceFenceOwner, settleObservedWorkspaceFence } from "./observed-workspace-fence.js";
@@ -930,7 +931,7 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [
   },
   {
     id: "remote_workspace_unsaved_files", title: "Remote workspace unsaved-files flag", obligation: "workspace_preservation",
-    storage: "database", settle: null, invariantChecks: [1, 2],
+    storage: "database", settle: settlePreparationOnlyRemoteWorkspaceEpoch, invariantChecks: [1, 2],
     listInProgress: async (db, limit, offset) => {
       const rows = await db.select({ id: executionWorkspaces.id, companyId: executionWorkspaces.companyId,
         metadata: executionWorkspaces.metadata, updatedAt: executionWorkspaces.updatedAt }).from(executionWorkspaces)

server/src/services/recovery/remote-workspace-unsaved-files-settlement.ts

@@ -0,0 +1,60 @@
+import { and, eq, inArray, like, or, sql } from "drizzle-orm";
+import { executionWorkspaces, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db";
+import { readRemoteWorkspaceAuthority } from "@paperclipai/shared";
+import type { OwnerProof } from "../owner-proof.js";
+import { abandonUnverifiedRemoteWorkspaceEpoch } from "../remote-workspace-epoch-abandonment.js";
+import { notifyTerminalWorkspaceWork } from "../terminal-workspace-work.js";
+import type { InProgressRecord, SettleOutcome } from "./ownership-registry.js";
+
+const TERMINAL_RUN_STATUSES = ["succeeded", "failed", "cancelled", "timed_out", "interrupted"];
+const WORK_STARTED_EVENT_TYPES = ["adapter.invoke", "adapter.progress"];
+const WORK_STARTED_EVENT_PREFIXES = ["turn.", "provider.", "native.process_"];
+
+export async function settlePreparationOnlyRemoteWorkspaceEpoch(db: Db, record: InProgressRecord,
+  context: { now: Date; proof: OwnerProof }): Promise<SettleOutcome> {
+  if (record.owner.kind !== "run" || !record.runId || !record.companyId) return { settled: false, reason: "owner_not_provable" };
+  const runId = record.runId, companyId = record.companyId;
+  return db.transaction(async (transaction): Promise<SettleOutcome> => {
+    const tx = transaction as unknown as Db;
+    const [run] = await tx.select({ status: heartbeatRuns.status, executionStage: heartbeatRuns.executionStage,
+      runtimeModeResolvedAt: heartbeatRuns.runtimeModeResolvedAt, processPid: heartbeatRuns.processPid,
+      processGroupId: heartbeatRuns.processGroupId, nativeSessionId: heartbeatRuns.nativeSessionId,
+      runnerInstanceId: heartbeatRuns.runnerInstanceId })
+      .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId))).for("update");
+    if (!run || !TERMINAL_RUN_STATUSES.includes(run.status)) return { settled: false, reason: "owner_run_active" };
+    if (run.runtimeModeResolvedAt || run.executionStage !== "preparing") return { settled: false, reason: "writer_reached_runtime", watch: false };
+    const [finalization] = await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
+      .where(and(eq(nativeRunFinalizations.companyId, companyId), eq(nativeRunFinalizations.runId, runId))).limit(1);
+    if (finalization || run.processPid || run.processGroupId || run.nativeSessionId || run.runnerInstanceId) {
+      return { settled: false, reason: "writer_reached_runtime", watch: false };
+    }
+    const [workStarted] = await tx.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
+      eq(heartbeatRunEvents.runId, runId), or(inArray(heartbeatRunEvents.eventType, WORK_STARTED_EVENT_TYPES),
+        ...WORK_STARTED_EVENT_PREFIXES.map(prefix => like(heartbeatRunEvents.eventType, `${prefix}%`))))).limit(1);
+    if (workStarted) return { settled: false, reason: "writer_reached_runtime", watch: false };
+    const [workspace] = await tx.select({ metadata: executionWorkspaces.metadata }).from(executionWorkspaces)
+      .where(and(eq(executionWorkspaces.companyId, companyId), eq(executionWorkspaces.id, record.recordId)));
+    const authority = readRemoteWorkspaceAuthority(workspace?.metadata);
+    if (!authority?.dirty || authority.writerRunId !== runId || !authority.location) {
+      return { settled: false, reason: "unsaved_files_no_longer_held" };
+    }
+    const [save] = await tx.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
+      eq(workspaceOperations.companyId, companyId), eq(workspaceOperations.executionWorkspaceId, record.recordId),
+      eq(workspaceOperations.phase, "remote_workspace_snapshot"), inArray(workspaceOperations.status, ["running", "pending"]),
+      sql`${workspaceOperations.metadata}->>'epoch' = ${String(authority.epoch)}`)).limit(1);
+    if (save) return { settled: false, reason: "save_in_flight" };
+    await notifyTerminalWorkspaceWork(tx, `workspace:${record.recordId}`);
+    try {
+      await abandonUnverifiedRemoteWorkspaceEpoch(transaction, {
+        companyId, workspaceId: record.recordId, runId, epoch: authority.epoch, snapshotId: authority.snapshotId,
+        location: authority.location,
+      });
+    } catch (error) {
+      if (error instanceof Error && error.message === "remote_workspace_authority_changed") {
+        return { settled: false, reason: "unsaved_files_no_longer_held" };
+      }
+      throw error;
+    }
+    return { settled: true, reason: context.proof.reason };
+  });
+}

server/src/services/remote-workspace-save-concurrency.test.ts

@@ -186,6 +186,66 @@ describe.skipIf(!support.supported)("remote workspace save parent lock order", (
     }
   }, 30_000);
 
+  it("lets a retry prepare the workspace again after its first writer was killed during preparation", async () => {
+    const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), retryId = randomUUID();
+    await db.insert(companies).values({ id: companyId, name: "Killed preparation", issuePrefix: `K${companyId.slice(0, 6)}` });
+    await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner" });
+    await db.insert(issues).values({ id: issueId, companyId, title: "Prepare twice", status: "in_progress", assigneeAgentId: agentId });
+    await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, status: "running", executionStage: "preparing",
+      contextSnapshot: { issueId } });
+    const [environment] = await db.insert(environments).values({ name: runId, driver: "sandbox" }).returning();
+    const storage = { ...createStorageService(createLocalDiskStorageProvider(path.join(root, runId, "objects"))), provider: "s3" as const };
+    const allocate = async (heartbeatRunId: string, directory: string, killSave = false) => {
+      const [lease] = await db.insert(environmentLeases).values({ companyId, issueId, heartbeatRunId, environmentId: environment.id,
+        provider: "daytona", providerLeaseId: randomUUID(), status: "active" }).returning();
+      const remoteCwd = path.join(root, runId, directory);
+      await mkdir(remoteCwd, { recursive: true });
+      const transfer: NonNullable<NonNullable<AdapterSandboxExecutionTarget["runner"]>["syncIn"]> = async operations => {
+        for (const operation of operations) for (const file of operation.files) {
+          await mkdir(path.dirname(file.targetPath), { recursive: true });
+          await cp(file.sourcePath, file.targetPath, { recursive: file.kind === "directory" });
+        }
+        return { operations: operations.map(operation => ({ operationId: operation.operationId,
+          filesTransferred: operation.files.length, bytesTransferred: 0 })) };
+      };
+      const target: AdapterSandboxExecutionTarget = { kind: "remote", transport: "sandbox", leaseId: lease.id, remoteCwd, timeoutMs: 30_000,
+        runner: { syncIn: transfer, syncOut: killSave ? async () => { throw new Error("controller killed"); } : transfer,
+          execute: async input => {
+          const result = await exec(input.command, input.args ?? [], { cwd: input.cwd ?? remoteCwd,
+            env: { ...process.env, ...input.env }, timeout: input.timeoutMs, maxBuffer: 8 * 1024 * 1024 });
+          return { exitCode: 0, stdout: result.stdout, stderr: result.stderr, signal: null, timedOut: false,
+            pid: process.pid, startedAt: new Date().toISOString() };
+        } } };
+      return { lease: lease as EnvironmentLease, target, remoteCwd };
+    };
+    const first = await allocate(runId, "first", true);
+    const reserved = await reserveRemoteWorkspace({ db, storage, companyId, issueId, runId, agentId, environmentId: environment.id,
+      projectId: null, projectWorkspaceId: null, existingWorkspaceId: null, source: { kind: "empty" } });
+    await expect(prepareRemoteWorkspace({ db, storage, workspace: reserved, runId, lease: first.lease, target: first.target }))
+      .rejects.toThrow();
+    const workspaceAuthority = async () => readRemoteWorkspaceAuthority((await db.select().from(executionWorkspaces)
+      .where(eq(executionWorkspaces.id, reserved.id)))[0]!.metadata)!;
+    expect(await workspaceAuthority()).toMatchObject({ epoch: 1, dirty: true, writerRunId: runId, snapshotId: null });
+
+    await db.update(heartbeatRuns).set({ status: "failed", errorCode: "process_lost", finishedAt: new Date() })
+      .where(eq(heartbeatRuns.id, runId));
+    await db.insert(heartbeatRuns).values({ id: retryId, companyId, agentId, issueId, status: "running", executionStage: "preparing",
+      contextSnapshot: { issueId, retryOfRunId: runId } });
+    await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
+    await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
+    expect(await workspaceAuthority()).toMatchObject({ epoch: 2, dirty: false, writerRunId: null, location: null, snapshotId: null });
+    expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, first.lease.id)))[0]!.status).not.toBe("active");
+
+    const second = await allocate(retryId, "second");
+    const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, reserved.id));
+    const retried = await reserveRemoteWorkspace({ db, storage, companyId, issueId, runId: retryId, agentId,
+      environmentId: environment.id, projectId: null, projectWorkspaceId: null, existingWorkspaceId: workspace.id, source: { kind: "empty" } });
+    const reprepared = await prepareRemoteWorkspace({ db, storage, workspace: retried, runId: retryId,
+      lease: second.lease, target: second.target });
+    expect(await workspaceAuthority()).toMatchObject({ epoch: 3, dirty: true, writerRunId: retryId, snapshotId: expect.any(String) });
+    expect(path.relative(first.remoteCwd, reprepared.remoteRoot).startsWith("..")).toBe(true);
+  }, 30_000);
+
   it("releases database locks while a provisioning command is in flight", async () => {
     const f = await preparedFailure();
     await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, f.runId));