feat(recovery): settle observed-workspace preservation fences on the shared path (#20022)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native run on a remote sandbox fails mid-turn with unsaved
files, Paperclip pins an observed-workspace preservation fence. The
sandbox is stopped and retained, never deleted, and the run is settled
only after the provider confirms the stop
> - The fence is held by the run's expired coordinator lease. Its
release proof is the provider's stop receipt, and only the
environment-settlement worker acted on it
> - The ownership reconciler could not see these fences at all: they sit
on the preserved-workspace recovery action, which is resolved (no
replay) while the fence still waits
> - The durable ownership plan has every in-progress record visible to
the one reconciler, and settled from it once its owner is proven gone
> - This pull request makes the fence visible, proves it by the provider
stop, and settles it through the same function the worker already uses
> - The benefit is that a fence whose sandbox stopped can no longer sit
unseen, and expiry alone never releases one

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory:
"Observed-run workspace preservation fence").

**What happened?**
`recoveryActionKind` lists only unresolved recovery actions, so
`observed_workspace_preservation_fence` reported nothing: its fence
lives on a resolved action. The kind registered no settle, and its fence
owner never carried the provider proof.

**Expected behavior**
Every `stop_pending` fence is listed. The fence is `gone`
(`provider_stopped`) once its run's one remote lease has a termination
receipt and a confirmed stop-and-retain. The reconciler then settles it
with `settleStoppedObservedWorkspace`, which re-checks every fence field
under its own locks. An expired coordinator lease without that receipt
is `unprovable` and is never released.

## What Changed

- `recovery/ownership-registry.ts`:
- `recoveryActionKind` takes `includeResolved`, which only the observed
fence uses.
- `resolveFenceHolders` adds `provider: "stopped"` for observed fences
whose run lease carries the stop proof.
- The kind registers a settle that calls
`settleStoppedObservedWorkspace`. A source-busy lock conflict is kept
for a later pass.
- `native-observed-workspace-preservation.ts`: pinning a fence wakes the
ownership worker in the same transaction.
- The settling-kinds list in `ownership-reconciliation-report.test.ts`
is updated.

## Verification

- `heartbeat-observed-workspace-preservation.test.ts`: 29 pass. The new
test "settles the fence from the shared reconciler once the provider
stop is proven, never before" fails on `master` and passes here:
- a pinned fence with an expired lease is
`fence_expired_without_provider_proof` and is not settled;
- after the provider stop, a read-only pass sees `provider_stopped` and
changes nothing;
- the settle moves the fence to `stopped` and the coordinator to
`terminal_failure`, and the saved files are untouched.

All 15 existing "keeps unsafe observed source held" cases and the
stale-evidence cases still pass.
- `ownership-reconciliation-report.test.ts` (25) passes.
- The server package typecheck passes.

## Risks

- The kind's listing now includes resolved recovery actions that carry a
`stop_pending` fence. Like every recovery-action kind, it is an
evidence-predicate scan; no new index is added.
- The settle is the existing worker's function with all its fence
checks. The reconciler is a backstop for that worker, not a new release
rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/__tests__/heartbeat-observed-workspace-preservation.test.ts

@@ -25,6 +25,9 @@ import { hasVerifiedWorkspaceRepair } from "../services/workspace-recovery-recei
 import { pinFailedObservedWorkspace, settleStoppedObservedWorkspace, ObservedWorkspacePreservationBusyError } from "../services/native-runtime/native-observed-workspace-preservation.js";
 import { claimNativeRestartRecoveries } from "../services/native-runtime/native-restart-recovery.js";
 import { hasRemoteTerminationReceipt } from "../services/remote-execution-termination.js";
+import { hasConfirmedSandboxStopAndRetain } from "../services/sandbox-stop-and-retain.js";
+import { createOwnershipReportState, runOwnershipReconciliationPass, settleOwnableKind } from "../services/recovery/ownership-invariant-report.js";
+import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js";
 import { describeEmbeddedPostgres, nativeRestartFixture } from "./helpers/native-restart.js";
 
 const execute = promisify(execFile);
@@ -326,6 +329,35 @@ describeEmbeddedPostgres("stopped observed native workspace preservation", () =>
     }
   });
 
+  it("settles the fence from the shared reconciler once the provider stop is proven, never before", async () => {
+    const f = await seedSource("SHARED-RECONCILER");
+    const kind = OWNABLE_KINDS.find(entry => entry.id === "observed_workspace_preservation_fence")!;
+    const pass = (record = true) => runOwnershipReconciliationPass(f.db, { state: createOwnershipReportState(), kinds: [kind], record });
+    const settleThis = () => settleOwnableKind(f.db, kind.id, { runIds: [f.runId] });
+    const fenceOf = async () => (await f.db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, f.issueId),
+      eq(issueRecoveryActions.fingerprint, `native-workspace:${f.runId}`))))[0].evidence.observedWorkspacePreservation as { state: string };
+    expect(await pinFailedObservedWorkspace(f.db, companyId, f.runId)).toBe(true);
+    const held = await pass(false);
+    expect(held.kinds[0]!.reasons.fence_expired_without_provider_proof).toBeGreaterThanOrEqual(1);
+    expect(await settleThis()).toEqual([]);
+    expect(await fenceOf()).toMatchObject({ state: "stop_pending" });
+
+    const heartbeat = heartbeatService(f.db);
+    try {
+      await heartbeat.sweepPendingCleanupLeases({ backoffMs: 0 });
+      const { lease } = await readSource(f);
+      expect(hasRemoteTerminationReceipt(lease) && hasConfirmedSandboxStopAndRetain(lease)).toBe(true);
+      const readOnly = await pass(false);
+      expect(readOnly.kinds[0]!.reasons.provider_stopped).toBeGreaterThanOrEqual(1);
+      expect(await fenceOf()).toMatchObject({ state: "stop_pending" });
+
+      expect(await settleThis()).toMatchObject([{ kind: kind.id, runId: f.runId, reason: "provider_stopped" }]);
+      expect(await fenceOf()).toMatchObject({ state: "stopped" });
+      expect((await readSource(f)).owner).toMatchObject({ phase: "terminal_failure", leaseOwner: null });
+      expect(await readFile(path.join(f.remoteCwd, "saved.txt"), "utf8")).toBe("unfinished work with an unknown provider effect\n");
+    } finally { await heartbeat.drainActiveRunExecutions(); }
+  });
+
   it("rechecks a new physical allocation owner after pin before the actual Stop dispatch", async () => {
     const f = await seedSource("ARRIVING-OWNER");
     expect(await pinFailedObservedWorkspace(f.db, companyId, f.runId)).toBe(true);

server/src/__tests__/ownership-reconciliation-report.test.ts

@@ -143,7 +143,7 @@ describe("report-only ownership reconciliation", () => {
     expect(new Set(names).size).toBe(names.length);
     expect(names.length).toBeGreaterThanOrEqual(40);
     expect(OWNABLE_KINDS.filter(kind => kind.settle !== null).map(kind => kind.id))
-      .toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "execution_workspace_runtime_lease", "copyback_owner_hold", "wake_request_claimed", "legacy_controller_lease", "issue_execution_lock", "issue_checkout_lock"]);
+      .toEqual(["native_accepted_result_lease", "native_finalizer_lease", "native_cleanup_lease", "sandbox_creation_intent", "environment_run_lease_active", "environment_teardown_claim", "environment_retained_orphan_sandbox", "workspace_runtime_control_operation", "native_workspace_finalize_attempt", "execution_workspace_runtime_lease", "copyback_owner_hold", "observed_workspace_preservation_fence", "wake_request_claimed", "legacy_controller_lease", "issue_execution_lock", "issue_checkout_lock"]);
     expect(OWNABLE_KINDS.filter(kind => !kind.listInProgress).every(kind => kind.coverageGap)).toBe(true);
   });
 

server/src/services/native-runtime/native-observed-workspace-preservation.ts

@@ -174,6 +174,7 @@ export async function pinFailedObservedWorkspace(db: Db, companyId: string, runI
       && lease.metadata?.workspaceRecoveryCleanup == null;
     if (!active) return false;
     await notifyNativeEnvironmentSettlement(tx, { companyId, runId, issueId: task.id });
+    await notifyDeliveryWork(tx, DELIVERY_QUEUES.ownershipInvariantReport, JSON.stringify({ companyId, runId }));
     const retained = await recordRequiredWorkspaceRestoreFailure(tx, run, { workspaceRestoreFailure: "restore_failed" }, lease);
     if (!retained || !hasRequiredWorkspaceRecovery(retained.resultJson)) throw new Error("observed_workspace_retention_missing");
     const [action] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, companyId),

server/src/services/recovery/copyback-owner-hold.ts

@@ -0,0 +1,32 @@
+import type { Db } from "@paperclipai/db";
+import { copybackOwnerStamp } from "../native-runtime/native-workspace-owner-stop-proof.js";
+import type { OwnerProof } from "../owner-proof.js";
+import type { InProgressRecord, RecordOwner, SettleOutcome } from "./ownership-registry.js";
+
+const object = (value: unknown): Record<string, unknown> =>
+  value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
+const text = (value: unknown) => typeof value === "string" && value ? value : null;
+
+export function copybackOwnerHoldOwner(evidence: unknown): RecordOwner {
+  const owner = object(object(object(evidence).workspaceOwnerRecovery).owner);
+  const pid = typeof owner.pid === "number" && Number.isInteger(owner.pid) && owner.pid > 0 ? owner.pid : null;
+  const hostname = text(owner.hostname);
+  const processStartedAt = text(owner.processStartedAt);
+  if (!pid || !hostname || !processStartedAt) return { kind: "none" };
+  const controllerBootId = text(owner.controllerBootId);
+  const hostHash = text(owner.hostHash);
+  return { kind: "process", stamp: { ...copybackOwnerStamp({ hostname, pid, processStartedAt,
+    ...(controllerBootId ? { controllerBootId } : {}), ...(hostHash ? { hostHash } : {}) }), scope: "copyback_owner_hold" } };
+}
+
+export async function settleCopybackOwnerHold(db: Db, record: InProgressRecord,
+  context: { now: Date; proof: OwnerProof }): Promise<SettleOutcome> {
+  if (record.owner.kind !== "process" || !record.companyId) return { settled: false, reason: "owner_not_provable" };
+  const { wakeCopybackOwnerRecheck } = await import("../workspace-recovery.js");
+  const stamp = record.owner.stamp;
+  const woke = await wakeCopybackOwnerRecheck(db, { companyId: record.companyId, actionId: record.recordId,
+    now: context.now, owner: { pid: stamp.pid, processStartedAt: stamp.processStartedAt, bootId: stamp.bootId,
+      processSpace: stamp.processSpace } });
+  return woke ? { settled: false, reason: "copyback_recheck_woken" }
+    : { settled: false, reason: "copyback_recheck_not_waiting", watch: false };
+}

server/src/services/recovery/observed-workspace-fence.ts

@@ -0,0 +1,55 @@
+import { inArray } from "drizzle-orm";
+import { environmentLeases, type Db } from "@paperclipai/db";
+import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
+import { hasConfirmedSandboxStopAndRetain } from "../sandbox-stop-and-retain.js";
+import type { InProgressRecord, RecordOwner, SettleOutcome } from "./ownership-registry.js";
+
+export const OBSERVED_WORKSPACE_FENCE_SCOPE = "observed_workspace_preservation_fence";
+
+const object = (value: unknown): Record<string, unknown> =>
+  value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
+
+export function observedWorkspaceFenceOwner(evidence: unknown): RecordOwner {
+  const fence = object(object(evidence).observedWorkspacePreservation);
+  const token = typeof fence.leaseOwner === "string" && fence.leaseOwner ? fence.leaseOwner : null;
+  const generation = typeof fence.controllerGeneration === "number" && Number.isSafeInteger(fence.controllerGeneration)
+    ? fence.controllerGeneration : null;
+  return token ? { kind: "fence", stamp: { kind: "remote", scope: OBSERVED_WORKSPACE_FENCE_SCOPE, generation, fenceToken: token },
+    fence: { currentFenceToken: null, currentGeneration: null, leaseExpiresAt: null } } : { kind: "none" };
+}
+
+export async function providerStoppedObservedRuns(db: Db, runIds: readonly string[]): Promise<Set<string>> {
+  const stopped = new Set<string>();
+  if (runIds.length === 0) return stopped;
+  const leases = await db.select().from(environmentLeases).where(inArray(environmentLeases.heartbeatRunId, [...runIds]));
+  const byRun = new Map<string, typeof leases>();
+  for (const lease of leases) byRun.set(lease.heartbeatRunId!, [...(byRun.get(lease.heartbeatRunId!) ?? []), lease]);
+  for (const [runId, runLeases] of byRun) {
+    if (runLeases.length === 1 && hasRemoteTerminationReceipt(runLeases[0]!) && hasConfirmedSandboxStopAndRetain(runLeases[0]!)) {
+      stopped.add(runId);
+    }
+  }
+  return stopped;
+}
+
+export async function applyObservedFenceProviderProof(db: Db, records: readonly InProgressRecord[]): Promise<void> {
+  const observed = records.filter(record => record.owner.kind === "fence" && record.runId
+    && record.owner.stamp.scope === OBSERVED_WORKSPACE_FENCE_SCOPE);
+  const stopped = await providerStoppedObservedRuns(db, [...new Set(observed.map(record => record.runId!))]);
+  for (const record of observed) {
+    if (record.owner.kind === "fence" && stopped.has(record.runId!)) record.owner.fence = { ...record.owner.fence, provider: "stopped" };
+  }
+}
+
+export async function settleObservedWorkspaceFence(db: Db, record: InProgressRecord): Promise<SettleOutcome> {
+  if (!record.companyId || !record.runId) return { settled: false, reason: "owner_not_provable" };
+  const { ObservedWorkspacePreservationBusyError, settleStoppedObservedWorkspace } =
+    await import("../native-runtime/native-observed-workspace-preservation.js");
+  try {
+    return await settleStoppedObservedWorkspace(db, record.companyId, record.runId)
+      ? { settled: true, reason: "provider_stopped" } : { settled: false, reason: "fence_not_settleable" };
+  } catch (error) {
+    if (error instanceof ObservedWorkspacePreservationBusyError) return { settled: false, reason: "source_busy" };
+    throw error;
+  }
+}

server/src/services/recovery/ownership-registry.ts

@@ -34,7 +34,8 @@ import {
 import { settleClaimedWakeRequest } from "./wake-request-claimed-settlement.js";
 import { settleAbandonedWorkspaceFinalizeAttempt } from "./workspace-finalize-attempt-settlement.js";
 import { runtimeControlOperationOwnerStamp, settleRuntimeControlOperation } from "../workspace-operations.js";
-import { copybackOwnerStamp } from "../native-runtime/native-workspace-owner-stop-proof.js";
+import { copybackOwnerHoldOwner, settleCopybackOwnerHold } from "./copyback-owner-hold.js";
+import { applyObservedFenceProviderProof, observedWorkspaceFenceOwner, settleObservedWorkspaceFence } from "./observed-workspace-fence.js";
 import { nativeControllerOwnerStamp, runProcessOwnerStamp } from "../native-runtime/native-owner-stamps.js";
 import { settleIssueLock } from "./issue-lock-settlement.js";
 import { legacyControllerOwnerStamp, settleLegacyControllerLease } from "../legacy-controller-lease.js";
@@ -461,14 +462,16 @@ type RecoveryActionRow = Pick<typeof issueRecoveryActions.$inferSelect, "id" | "
 
 function recoveryActionKind(id: string, title: string, obligation: Obligation, where: SQL | undefined,
   owner: (row: RecoveryActionRow) => RecordOwner, invariantChecks: readonly number[] = [1, 2, 7],
-  runIdOf: (row: RecoveryActionRow) => string | null = () => null): OwnableKind {
+  runIdOf: (row: RecoveryActionRow) => string | null = () => null,
+  options: { includeResolved?: boolean } = {}): OwnableKind {
   return {
     id, title, obligation, storage: "database", settle: null, invariantChecks,
     listInProgress: async (db, limit, offset) => {
       const rows = await db.select({ id: issueRecoveryActions.id, companyId: issueRecoveryActions.companyId,
         sourceIssueId: issueRecoveryActions.sourceIssueId, ownerType: issueRecoveryActions.ownerType,
         evidence: issueRecoveryActions.evidence, updatedAt: issueRecoveryActions.updatedAt })
-        .from(issueRecoveryActions).where(and(isNull(issueRecoveryActions.resolvedAt), where)).orderBy(asc(issueRecoveryActions.id)).limit(limit + 1).offset(offset);
+        .from(issueRecoveryActions).where(and(options.includeResolved ? undefined : isNull(issueRecoveryActions.resolvedAt), where))
+        .orderBy(asc(issueRecoveryActions.id)).limit(limit + 1).offset(offset);
       const result = page(rows, limit);
       return { truncated: result.truncated, records: result.rows.map(row => ({
         recordId: row.id, companyId: row.companyId, runId: runIdOf(row), issueId: row.sourceIssueId,
@@ -850,46 +853,18 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [
     sql`${issueRecoveryActions.evidence}->'systemRecovery'->>'state' in ('scheduled', 'running', 'awaiting_evidence')`,
     () => ({ kind: "none" })),
   {
-    // Shared path since Phase 2: receipts record their PID space, so the shared
-    // prover judges the owner (older receipts name none and stay unprovable).
-    // A proven-gone owner wakes the waiting recheck; the worker still proves the
-    // copyback process group before it takes ownership, so this never settles
-    // the hold itself.
     ...recoveryActionKind("copyback_owner_hold", "Copyback-owner hold (RC-50)", "ownership",
       sql`${issueRecoveryActions.evidence}->'workspaceOwnerRecovery'->>'state' = 'operator_required'`,
-      row => {
-        const owner = json(json(json(row.evidence).workspaceOwnerRecovery).owner);
-        const pid = int(owner.pid);
-        const hostname = str(owner.hostname);
-        const processStartedAt = str(owner.processStartedAt);
-        if (!pid || !hostname || !processStartedAt) return { kind: "none" };
-        return { kind: "process", stamp: { ...copybackOwnerStamp({ hostname, pid, processStartedAt,
-          ...(str(owner.controllerBootId) ? { controllerBootId: str(owner.controllerBootId)! } : {}),
-          ...(str(owner.hostHash) ? { hostHash: str(owner.hostHash)! } : {}) }), scope: "copyback_owner_hold" } };
-      }),
-    settle: async (db, record, context) => {
-      if (record.owner.kind !== "process" || !record.companyId) return { settled: false, reason: "owner_not_provable" };
-      const { wakeCopybackOwnerRecheck } = await import("../workspace-recovery.js");
-      const stamp = record.owner.stamp;
-      const woke = await wakeCopybackOwnerRecheck(db, { companyId: record.companyId, actionId: record.recordId,
-        now: context.now, owner: { pid: stamp.pid, processStartedAt: stamp.processStartedAt, bootId: stamp.bootId,
-          processSpace: stamp.processSpace } });
-      // A hold whose recheck is not waiting (exhausted, past its deadline or
-      // moved on) is the board's: keep it visible, but stop watching it.
-      return woke ? { settled: false, reason: "copyback_recheck_woken" }
-        : { settled: false, reason: "copyback_recheck_not_waiting", watch: false };
-    },
+      row => copybackOwnerHoldOwner(row.evidence)),
+    settle: settleCopybackOwnerHold,
+  },
+  {
+    ...recoveryActionKind("observed_workspace_preservation_fence", "Observed-run workspace preservation fence", "workspace_preservation",
+      sql`${issueRecoveryActions.evidence}->'observedWorkspacePreservation'->>'state' = 'stop_pending'`,
+      row => observedWorkspaceFenceOwner(row.evidence), [1, 2, 7],
+      row => str(json(json(row.evidence).observedWorkspacePreservation).runId), { includeResolved: true }),
+    settle: settleObservedWorkspaceFence,
   },
-  recoveryActionKind("observed_workspace_preservation_fence", "Observed-run workspace preservation fence", "workspace_preservation",
-    sql`${issueRecoveryActions.evidence}->'observedWorkspacePreservation'->>'state' = 'stop_pending'`,
-    row => {
-      const fence = json(json(row.evidence).observedWorkspacePreservation);
-      const token = str(fence.leaseOwner);
-      // The current holder is read in the report pass (`resolveFenceHolders`).
-      return token ? { kind: "fence", stamp: { kind: "remote", scope: "observed_workspace_preservation_fence",
-        generation: int(fence.controllerGeneration), fenceToken: token },
-        fence: { currentFenceToken: null, currentGeneration: null, leaseExpiresAt: null } } : { kind: "none" };
-    }, [1, 2, 7], row => str(json(json(row.evidence).observedWorkspacePreservation).runId)),
 
   // Accounting and budgets
   runKind("run_accounting_debt", "Run accounting debt", "accounting",
@@ -1018,4 +993,5 @@ export async function resolveFenceHolders(db: Db, records: InProgressRecord[]):
     record.owner.fence = { currentFenceToken: holder?.leaseOwner ?? null,
       currentGeneration: holder?.generation ?? null, leaseExpiresAt: holder?.leaseExpiresAt ?? null };
   }
+  await applyObservedFenceProviderProof(db, pending);
 }