fix(recovery): settle a remote workspace save that stopped without finishing (#20291)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshots.
Each save is a `workspace_operations` row with phase
`remote_workspace_snapshot`
> - The ownership registry lists that kind as in-progress work, but it
had no settle step
> - When a restart cut off a failing final save, the row stayed
`running` after its invocations had all stopped and its run had ended
> - The retained-source fallback refuses while any workspace operation
is running, so a task whose saved files could not be verified could
never continue from its last save
> - This pull request lets the ownership reconciler settle that row as
failed
> - The benefit is that the automatic fallback can continue such a task
without any user action

## Linked Issues or Issue Description

**What happened?**
A native run in a remote sandbox finished its turn. Its final workspace
save failed because the sandbox disk was full ("No space left on device"
in the snapshot copy). A graceful restart then interrupted the retry.
The run ended as `native_workspace_sync_out_retry_exhausted`. Its
`remote_workspace_snapshot` row kept `status: running` for six days,
with `saveWork: { activeIds: [], unknown: false, quiescedAt: <time> }`
and `exportCleanup: { state: "pending" }`. The retained-source fallback
checked the task and refused with `workspace_operation_running`, so the
task stayed blocked.

**Expected behavior**
A save with no active or unknown invocation, whose run is terminal, is
finished work. The reconciler must mark it failed, so the fallback can
continue from the last committed save. A save that might still be
running must stay untouched.

**Steps to reproduce**
1. Run a native task in a remote sandbox. Make its final save fail, then
restart the server during the retry, so the save's work is quiesced and
the operation stays `running`.
2. Let the task's recovery refuse with an unverifiable-source failure
(for example `allocation_ownership_ambiguous`).
3. The retained-source fallback refuses with
`workspace_operation_running` on every check, and the task never
continues.

**Paperclip version or commit**
`master` at f7c85bca18.

Refs #20282 (the other cause of the same stuck task: a false ownership
refusal).

## What Changed

- `settleAbandonedRemoteSnapshotSave`
(`recovery/workspace-finalize-attempt-settlement.ts`) settles a
`remote_workspace_snapshot` row in `running` or `pending` only when all
of these are true:
- its save work is quiesced (`activeIds` is empty and `unknown` is
false);
  - its run is terminal.
It sets `status: failed`, records `abandonedSave` provenance and
`lastFailure: remote_workspace_save_abandoned`, keeps `saveWork` and
`exportCleanup`, writes a `workspace_operation.reconciled` activity
entry, and wakes the workspace's terminal work.
- The ownership registry wires it as the settle of
`remote_snapshot_save`.
- In the same transaction it re-arms blocked fallbacks. For every open
hold whose fallback check last refused with
`workspace_operation_running` and whose source run used this save's
workspace, it clears the check and wakes the workspace-recovery queue. A
hold that had already spent its 12 checks (no next check) continues as
well.
- The settle uses the save path's own `readSaveWork`, now exported, so
it never settles save work that the save path would treat as invalid.
- Because the row now has a `lastFailure`, `retireQuiescedSaves` retires
it like any failed save when a later final save commits. A later retry
of the same save boundary reopens a failed row, as before.
- The ownership inventory row for this kind names the new settle.

## Verification

- Red before the fix, green after: `settles a remote snapshot save that
stopped without finishing after its run ended, and keeps open or live
ones` (`ownership-reconciliation-report.test.ts`). Before: `settled: 0`.
After: only the abandoned save settles. Saves with a live run, an active
invocation, or unknown save work stay `running`.
- Red before the fix, green after, end to end: `continues once the
reconciler settles a final save a restart left running after its run
ended` (`workspace-recovery-fallback.test.ts`). The fallback first
refuses with `workspace_operation_running`, and the test then spends its
check budget (12 checks, no next check), so the hold drops out of
discovery. After one ownership reconciliation pass, the save is `failed`
and the check is cleared. The worker then resolves the hold with
`retained_source_unverifiable_continued_from_last_saved`. Without the
re-arm, this test fails.
- The reconciler test also covers save work that is not quiesced
(`quiescedAt: null`), save work that is invalid (`quiescedAt: ""`), and
a `pending` save.
- `workspace-recovery-fallback.test.ts` (16 tests),
`ownership-reconciliation-report.test.ts`, and the remote workspace
lifecycle, save-concurrency, and snapshot-retention suites pass (80
tests, 1 environment-gated file skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.

## Risks

- A save in another process registers its invocation in `activeIds` in
the same row-locked transaction before it runs. The settle locks the row
and requires `activeIds` to be empty and `unknown` to be false, so it
cannot fail a save that is still working.
- A save whose process died with its invocation still in `activeIds`
stays `running`. That case needs an owner-death proof, which is out of
scope here.
- No schema or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only browser diagnostics).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/plans/2026-11-07-ownership-inventory.md

@@ -346,7 +346,7 @@ Scope: workspace operations (#19807 intents), runtime leases, runtime services,
 | Managed runtime-control operation (start/stop/restart/repair/run) | `workspace_operations.status` (+ `metadata.runtimeControlOwner`) | workspace-operations.ts:559-578 (stamp :539-550, 10 s heartbeat :586-612) | `running` | `{ownerId: random UUID per process, pid, heartbeatAt}`. No host, boot id, or process start time (:25-29, :69-74) | Owner settle :671-690, owner failure :701-728, own deadline 30 min / 4 h (:46, :53, :656-668). Sweep `sweepRuntimeControlOperations` :297-415, run at startup (index.ts:1087) and before each control (:448-454, routes/execution-workspaces.ts:484, :570) | Same ownerId and not in the in-process set → abandoned (:334-335). Otherwise `process.kill(pid,0)` (:126-135) or **TTL/age**: 60 s without a heartbeat (:38, :346). CAS on `updated_at` (:142-147, :379-383) | ownership | 1 (no host/boot), 2 | A, C |
 | Native workspace finalize attempt (copyback export) | `workspace_operations.status`, phase `workspace_finalize`, `metadata.owningService='native_workspace_finalizer'` | native-runtime/native-workspace-finalizer.ts:170 → workspace-operations.ts:559 | `running` (`pending` is also read as unresolved) | none on the row. Run id only. Physical owner is the `nativeWorkspaceFinalizationOwner` receipt (row below) | (a) Normal settle workspace-operations.ts:671-728. (b) #19807 supersession native-workspace-finalization-supersession.ts:21-91, called from native-workspace-finalizer.ts:94. (c) Offline maintenance offline-workspace-operation-reconciliation.ts:30-118, called from maintenance/ephemeral-migration-data.ts:52. (d) Online variant :152-157 (**no production caller**) | (b) Caller holds finalization ownership, plus the in-process registry `isNativeWorkspaceFinalizationOperationActive` (workspace-operations.ts:114-119) and a later terminal attempt. (c) Advisory lock :41, committed coordinator with no lease :52-57, finalized sync hash :58-60, accepted result :65-70, later succeeded receipt :83-92. Never age | workspace-preservation | 1, 2 | B, C |
 | Remote provision command intent | `workspace_operations.status` + `metadata.uncertain`, phase `remote_workspace_provision` | remote-workspaces.ts:973-975 (`running`, `uncertain:true`) | `running` / `uncertain=true` | `heartbeatRunId` + `metadata.epoch` (fenced by authority epoch/writerRunId :964) | Success only: remote-workspaces.ts:989-990. On a non-zero exit, a timeout (:980-982), or a crash, **no write settles the row** (grep finds no other writer) | none. Reservation for any other run is refused with `remote_workspace_provision_unverified` (:220-225). Offline migration counts it as a blocker (remote-workspace-migration-tree.ts:160, remote-workspace-migration.ts:237) | workspace-preservation | 1, 2, 3, 7 | B, C |
-| Remote snapshot save (initial/checkpoint/final) | `workspace_operations.status`, phase `remote_workspace_snapshot` | remote-workspaces.ts:729 | `running`, `pending` | `heartbeatRunId` + `metadata.epoch`. Fenced by `assertOwned` and the authority writerRunId/epoch (:776-779) | Owner settle in `saveRemoteWorkspace`. Earlier saves are retired only after a final save commits: `retireQuiescedSaves` :592-630 | Save-work quiescence (`readSaveWork` activeIds/quiescedAt :616-618) plus a committed final snapshot (:597-608). No owner-death proof | workspace-preservation | 1, 2 | B, C |
+| Remote snapshot save (initial/checkpoint/final) | `workspace_operations.status`, phase `remote_workspace_snapshot` | remote-workspaces.ts:729 | `running`, `pending` | `heartbeatRunId` + `metadata.epoch`. Fenced by `assertOwned` and the authority writerRunId/epoch (:776-779) | Owner settle in `saveRemoteWorkspace`. Earlier saves are retired only after a final save commits: `retireQuiescedSaves` :592-630 | Save-work quiescence (`readSaveWork` activeIds/quiescedAt :616-618) plus a committed final snapshot (:597-608). A quiesced save whose run is terminal is settled `failed` by the ownership reconciler (`settleAbandonedRemoteSnapshotSave`) | workspace-preservation | 1, 2 | B, C |
 | Remote workspace migration op | `workspace_operations.status`, phase `remote_workspace_migration` | remote-workspace-migration.ts:375 | `running` | none (workspace id) | Explicit publication or repair by migration (unverified line). The runtime-control sweep skips it on purpose (workspace-operations.ts:319-323) | none | workspace-preservation | 1, 2 | B, C |
 | Execution workspace runtime lease | `execution_workspace_runtime_leases` (row existence) | workspace-runtime-leases.ts:194-209 (create), :235-247 (renew), :255-273 (reclaim). Called only from routes/execution-workspaces.ts:567 | row present | `owner_key` `issue:`/`run:`/`agent:` (+ FK columns). No process identity | `release` :291-313. In production it is called only with `force` on archive (routes/execution-workspaces.ts:1270). Otherwise the lease is reclaimed lazily at the next claim (:250-283) | `evaluateStaleOwner` :85-119: issue hidden or terminal, run terminal, or **TTL/age** 30 min (:39, :90). An agent-scoped owner has TTL only (:116-118) | ownership | 1, 2 | A, C |
 | Workspace runtime service (local process) | `workspace_runtime_services.status` | workspace-runtime.ts (insert line unverified) | `provisioning`, `starting`, `running` | `startedByRunId`, `ownerAgentId`. The pid lives in a local service registry file, not on the row | Startup `reconcilePersistedRuntimeServicesOnStartup` workspace-runtime.ts:8636, which adopts or stops each service (index.ts:1101, not awaited) | Registry record match (workspace-runtime.ts:8781-8797) plus `isPidAlive` local-service-supervisor.ts:218, :245-250. Same host only | ownership | 1, 2 | A, C |

server/src/__tests__/ownership-reconciliation-report.test.ts

@@ -145,7 +145,7 @@ describe("report-only ownership reconciliation", () => {
     expect(new Set(names).size).toBe(names.length);
     expect(names.length).toBeGreaterThanOrEqual(40);
     expect(OWNABLE_KINDS.filter(kind => kind.settle !== null).map(kind => kind.id))
-      .toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "workspace_command_operation", "execution_workspace_runtime_lease", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "legacy_interrupted_preparation_hold", "issue_execution_lock", "issue_checkout_lock", "agent_lifecycle_lease"]);
+      .toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "environment_login_probe_lease", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "workspace_command_operation", "remote_snapshot_save", "execution_workspace_runtime_lease", "legacy_shutdown_settlement_fence", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "legacy_interrupted_preparation_hold", "issue_execution_lock", "issue_checkout_lock", "agent_lifecycle_lease"]);
     expect(OWNABLE_KINDS.filter(kind => !kind.listInProgress).every(kind => kind.coverageGap)).toBe(true);
   });
 
@@ -371,6 +371,43 @@ describe("report-only ownership reconciliation", () => {
     expect(doneRun.status).toBe("succeeded");
   });
 
+  it("settles a remote snapshot save that stopped without finishing after its run ended, and keeps open or live ones", async () => {
+    const run = (id: string, status: string) => ({ id, companyId, agentId, status, invocationSource: "manual",
+      ...(status === "running" ? {} : { finishedAt: minutesAgo(2) }) });
+    const runs = { failed: randomUUID(), live: randomUUID() };
+    await db.insert(heartbeatRuns).values([run(runs.failed, "failed"), run(runs.live, "running")]);
+    const quiesced = { schema: "paperclip.remote-workspace-save-work.v1", activeIds: [], unknown: false, quiescedAt: minutesAgo(30).toISOString() };
+    const ops = { abandoned: randomUUID(), pendingAbandoned: randomUUID(), liveRun: randomUUID(), activeInvocation: randomUUID(),
+      unknownWork: randomUUID(), notQuiesced: randomUUID(), invalidWork: randomUUID() };
+    const op = (id: string, runId: string, saveWork: Record<string, unknown>, status = "running") => ({ id, companyId, heartbeatRunId: runId,
+      phase: "remote_workspace_snapshot", status,
+      metadata: { epoch: 8, boundary: "final", generation: 9, saveWork, exportCleanup: { state: "pending" } } });
+    await db.insert(workspaceOperations).values([
+      op(ops.abandoned, runs.failed, quiesced),
+      op(ops.pendingAbandoned, runs.failed, quiesced, "pending"),
+      op(ops.liveRun, runs.live, quiesced),
+      op(ops.activeInvocation, runs.failed, { ...quiesced, activeIds: [randomUUID()], quiescedAt: null }),
+      op(ops.unknownWork, runs.failed, { ...quiesced, unknown: true }),
+      op(ops.notQuiesced, runs.failed, { ...quiesced, quiescedAt: null }),
+      op(ops.invalidWork, runs.failed, { ...quiesced, quiescedAt: "" }),
+    ]);
+    const statuses = async () => Object.fromEntries((await db.select({ id: workspaceOperations.id, status: workspaceOperations.status })
+      .from(workspaceOperations).where(eq(workspaceOperations.phase, "remote_workspace_snapshot")))
+      .map(row => [Object.keys(ops).find(name => ops[name as keyof typeof ops] === row.id), row.status]));
+
+    const first = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
+    expect(kindReport(first, "remote_snapshot_save")).toMatchObject({ inProgress: 7, settled: 2 });
+    expect(await statuses()).toEqual({ abandoned: "failed", pendingAbandoned: "failed", liveRun: "running", activeInvocation: "running",
+      unknownWork: "running", notQuiesced: "running", invalidWork: "running" });
+    const [settled] = await db.select().from(workspaceOperations).where(eq(workspaceOperations.id, ops.abandoned));
+    expect(settled.metadata).toMatchObject({ saveWork: quiesced, exportCleanup: { state: "pending" },
+      abandonedSave: { reason: "abandoned_remote_snapshot_save", runId: runs.failed, previousStatus: "running" },
+      lastFailure: { code: "remote_workspace_save_abandoned" } });
+    expect(settled.finishedAt).not.toBeNull();
+    const [failedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runs.failed));
+    expect(failedRun.status).toBe("failed");
+  });
+
   it("settles a workspace command whose process died mid-command, and keeps live or unprovable ones", async () => {
     const run = (id: string, status: string) => ({ id, companyId, agentId, status, invocationSource: "manual",
       ...(status === "running" ? {} : { finishedAt: minutesAgo(2) }) });

server/src/services/recovery/ownership-registry.ts

@@ -36,7 +36,7 @@ import {
   settleNativeAcceptedResultLease,
 } from "../native-runtime/native-accepted-result-lease-settlement.js";
 import { settleClaimedWakeRequest } from "./wake-request-claimed-settlement.js";
-import { settleAbandonedWorkspaceCommandOperation, settleAbandonedWorkspaceFinalizeAttempt } from "./workspace-finalize-attempt-settlement.js";
+import { settleAbandonedRemoteSnapshotSave, settleAbandonedWorkspaceCommandOperation, settleAbandonedWorkspaceFinalizeAttempt } from "./workspace-finalize-attempt-settlement.js";
 import { LOCAL_WORKSPACE_COMMAND_PHASES, runtimeControlOperationOwnerStamp, settleRuntimeControlOperation } from "../workspace-operations.js";
 import { copybackOwnerHoldOwner, settleCopybackOwnerHold } from "./copyback-owner-hold.js";
 import { applyObservedFenceProviderProof, observedWorkspaceFenceOwner, settleObservedWorkspaceFence } from "./observed-workspace-fence.js";
@@ -866,9 +866,9 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [
     and(eq(workspaceOperations.phase, "remote_workspace_provision"), or(eq(workspaceOperations.status, "running"),
       sql`(${workspaceOperations.metadata}->>'uncertain')::boolean is true`)),
     row => runOwner(row.heartbeatRunId)),
-  operationKind("remote_snapshot_save", "Remote snapshot save", "workspace_preservation",
+  { ...operationKind("remote_snapshot_save", "Remote snapshot save", "workspace_preservation",
     and(eq(workspaceOperations.phase, "remote_workspace_snapshot"), inArray(workspaceOperations.status, ["running", "pending"])),
-    row => runOwner(row.heartbeatRunId)),
+    row => runOwner(row.heartbeatRunId)), settle: settleAbandonedRemoteSnapshotSave },
   operationKind("remote_workspace_migration", "Remote workspace migration op", "workspace_preservation",
     and(eq(workspaceOperations.phase, "remote_workspace_migration"), eq(workspaceOperations.status, "running")),
     () => ({ kind: "none" })),

server/src/services/recovery/workspace-finalize-attempt-settlement.ts

@@ -1,8 +1,11 @@
-import { and, eq, inArray } from "drizzle-orm";
-import { heartbeatRuns, workspaceOperations, type Db } from "@paperclipai/db";
+import { and, eq, inArray, sql } from "drizzle-orm";
+import { heartbeatRuns, issueRecoveryActions, workspaceOperations, type Db } from "@paperclipai/db";
 import { persistActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
 import type { OwnerProof } from "../owner-proof.js";
+import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
+import { readSaveWork } from "../remote-workspaces.js";
 import { notifyTerminalWorkspaceWork } from "../terminal-workspace-work.js";
+import { RETAINED_SOURCE_FALLBACK_CHECK_KEY } from "../workspace-recovery-fallback-discovery.js";
 import { LOCAL_WORKSPACE_COMMAND_PHASES, isOwnerStampedWorkspaceOperationLive } from "../workspace-operations.js";
 import type { InProgressRecord, SettleOutcome } from "./ownership-registry.js";
 
@@ -103,3 +106,64 @@ function abandonedWorkspaceOperationSettle(spec: { phases: readonly string[]; st
     return outcome;
   };
 }
+
+export const ABANDONED_REMOTE_SNAPSHOT_SAVE_REASON = "abandoned_remote_snapshot_save";
+
+export async function settleAbandonedRemoteSnapshotSave(db: Db, record: InProgressRecord,
+  context: { now: Date; proof: OwnerProof }): Promise<SettleOutcome> {
+  let publication: ActivityPublication | undefined;
+  const outcome = await db.transaction(async (transaction): Promise<SettleOutcome> => {
+    const tx = transaction as unknown as Db;
+    const [operation] = await tx.select().from(workspaceOperations)
+      .where(and(eq(workspaceOperations.id, record.recordId), eq(workspaceOperations.phase, "remote_workspace_snapshot"),
+        inArray(workspaceOperations.status, ["running", "pending"])))
+      .for("update");
+    if (!operation) return { settled: false, reason: "attempt_no_longer_in_progress" };
+    const metadata = operation.metadata ?? {};
+    const work = readSaveWork(metadata);
+    if (!work || work.unknown || work.activeIds.length > 0 || !work.quiescedAt) return { settled: false, reason: "save_work_unsettled" };
+    const [run] = operation.heartbeatRunId ? await tx.select({ status: heartbeatRuns.status }).from(heartbeatRuns)
+      .where(and(eq(heartbeatRuns.id, operation.heartbeatRunId), inArray(heartbeatRuns.status, TERMINAL_RUN_STATUSES))) : [];
+    if (!run) return { settled: false, reason: "owner_run_active" };
+    const now = context.now;
+    const provenance = {
+      schema: "paperclip.remote-snapshot-save-abandonment.v1",
+      reason: ABANDONED_REMOTE_SNAPSHOT_SAVE_REASON,
+      proof: context.proof.reason,
+      previousStatus: operation.status,
+      runId: operation.heartbeatRunId,
+      quiescedAt: work.quiescedAt,
+      reconciledAt: now.toISOString(),
+    };
+    const workKey = operation.executionWorkspaceId ? `workspace:${operation.executionWorkspaceId}`
+      : operation.issueId ? `issue:${operation.issueId}` : null;
+    if (workKey) await notifyTerminalWorkspaceWork(tx, workKey);
+    const [row] = await tx.update(workspaceOperations).set({
+      status: "failed", exitCode: 1, finishedAt: now, updatedAt: now,
+      stderrExcerpt: `${operation.stderrExcerpt ?? ""}The workspace save stopped without finishing, and its run has ended.\n`.slice(-4096),
+      metadata: { ...metadata, abandonedSave: provenance,
+        lastFailure: { code: "remote_workspace_save_abandoned", at: now.toISOString() } },
+    }).where(and(eq(workspaceOperations.id, operation.id), eq(workspaceOperations.status, operation.status)))
+      .returning({ id: workspaceOperations.id });
+    if (!row) return { settled: false, reason: "attempt_no_longer_in_progress" };
+    if (operation.executionWorkspaceId) {
+      await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery);
+      await tx.update(issueRecoveryActions).set({
+        evidence: sql`${issueRecoveryActions.evidence} - ${RETAINED_SOURCE_FALLBACK_CHECK_KEY}::text`,
+      }).where(and(eq(issueRecoveryActions.companyId, operation.companyId), inArray(issueRecoveryActions.status, ["active", "escalated"]),
+        sql`${issueRecoveryActions.evidence}->${RETAINED_SOURCE_FALLBACK_CHECK_KEY}->>'reason' = 'workspace_operation_running'`,
+        sql`exists (select 1 from heartbeat_runs source where source.company_id = ${issueRecoveryActions.companyId}
+          and source.id::text = ${issueRecoveryActions.evidence}->>'runId'
+          and source.runner_profile_json#>>'{remoteWorkspace,workspaceId}' = ${operation.executionWorkspaceId})`));
+    }
+    const activity = await persistActivity(tx, {
+      companyId: operation.companyId, actorType: "system", actorId: "ownership-reconciler",
+      action: "workspace_operation.reconciled", entityType: "workspace_operation", entityId: operation.id,
+      runId: operation.heartbeatRunId, issueId: operation.issueId, details: provenance,
+    });
+    publication = activity.publication;
+    return { settled: true, reason: context.proof.reason };
+  });
+  if (outcome.settled && publication) publishActivity(publication);
+  return outcome;
+}

server/src/services/remote-workspaces.ts

@@ -69,7 +69,7 @@ type RemoteWorkspaceSaveTiming = RemoteWorkspaceSavePhaseTiming & {
 };
 const RETIRED_SAVE_SCHEMA = "paperclip.remote-workspace-retired-save.v1";
 type SaveWork = { schema: typeof SAVE_WORK_SCHEMA; activeIds: string[]; unknown: boolean; quiescedAt: string | null };
-function readSaveWork(metadata: Record<string, unknown> | null | undefined): SaveWork | null {
+export function readSaveWork(metadata: Record<string, unknown> | null | undefined): SaveWork | null {
   const work = object(metadata?.saveWork);
   return work.schema === SAVE_WORK_SCHEMA && Array.isArray(work.activeIds)
     && work.activeIds.every(id => typeof id === "string") && new Set(work.activeIds).size === work.activeIds.length

server/src/services/workspace-recovery-fallback.test.ts

@@ -17,6 +17,7 @@ import { prepareRemoteWorkspace, reserveRemoteWorkspace, saveRemoteWorkspace } f
 import { readSystemRecovery } from "./system-recovery-state.js";
 import { reconcileRetainedWorkspaces, recoverRetainedWorkspace, retainedWorkspaceCandidates } from "./workspace-recovery.js";
 import { nextWorkspaceRecoveryAt } from "./system-recovery.js";
+import { createOwnershipReportState, runOwnershipReconciliationPass } from "./recovery/ownership-invariant-report.js";
 import { RETAINED_SOURCE_FALLBACK_OUTCOME, RETAINED_SOURCE_QUARANTINE_PHASE } from "./workspace-recovery-fallback.js";
 import { hasVerifiedWorkspaceRepair, readWorkspaceRecovery } from "./workspace-recovery-receipt.js";
 import { acceptedRemoteExport, action, connectionString, db, run, useWorkspaceRecoveryFixture } from "./__tests__/workspace-recovery-fixture.js";
@@ -339,6 +340,30 @@ describe("retained source fallback", () => {
     expect(await action(f.actionId)).toMatchObject({ status: "resolved", resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME });
   }, 60_000);
 
+  it("continues once the reconciler settles a final save a restart left running after its run ended", async () => {
+    const f = await unverifiableRetainedSource();
+    const [save] = await db.insert(workspaceOperations).values({ companyId: f.companyId, executionWorkspaceId: f.workspaceId,
+      heartbeatRunId: f.runId, issueId: f.issueId, phase: "remote_workspace_snapshot", status: "running",
+      metadata: { epoch: 8, boundary: "final", generation: 99, exportCleanup: { state: "pending" },
+        saveWork: { schema: "paperclip.remote-workspace-save-work.v1", activeIds: [], unknown: false, quiescedAt: new Date(Date.now() - 60_000).toISOString() } },
+    }).returning();
+    await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => undefined, true) ?? []);
+    const deferred = await action(f.actionId);
+    expect(deferred.status).toBe("active");
+    expect(deferred.evidence.retainedSourceFallbackCheck).toMatchObject({ reason: "workspace_operation_running" });
+    // Twelve refusals spend the check budget: no further check is scheduled.
+    await db.update(issueRecoveryActions).set({ evidence: { ...deferred.evidence, retainedSourceFallbackCheck: {
+      ...(deferred.evidence.retainedSourceFallbackCheck as object), checks: 12, nextCheckAt: null } } }).where(eq(issueRecoveryActions.id, f.actionId));
+    expect((await retainedWorkspaceCandidates(db, undefined, false, true)).map(row => row.action.id)).not.toContain(f.actionId);
+
+    await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() });
+    expect((await db.select().from(workspaceOperations).where(eq(workspaceOperations.id, save.id)))[0])
+      .toMatchObject({ status: "failed", metadata: { abandonedSave: { reason: "abandoned_remote_snapshot_save" } } });
+    expect((await action(f.actionId)).evidence.retainedSourceFallbackCheck).toBeUndefined();
+    await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => undefined, true) ?? []);
+    expect(await action(f.actionId)).toMatchObject({ status: "resolved", resolutionNote: RETAINED_SOURCE_FALLBACK_OUTCOME });
+  }, 60_000);
+
   it("settles a stale exhausted hold as recovered when a sibling action saved the run's files", async () => {
     const f = await acceptedRemoteExport();
     await recoverRetainedWorkspace(f.input);