feat(recovery): settle observed-workspace preservation fences on the shared path (#20022)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native run on a remote sandbox fails mid-turn with unsaved
files, Paperclip pins an observed-workspace preservation fence. The
sandbox is stopped and retained, never deleted, and the run is settled
only after the provider confirms the stop
> - The fence is held by the run's expired coordinator lease. Its
release proof is the provider's stop receipt, and only the
environment-settlement worker acted on it
> - The ownership reconciler could not see these fences at all: they sit
on the preserved-workspace recovery action, which is resolved (no
replay) while the fence still waits
> - The durable ownership plan has every in-progress record visible to
the one reconciler, and settled from it once its owner is proven gone
> - This pull request makes the fence visible, proves it by the provider
stop, and settles it through the same function the worker already uses
> - The benefit is that a fence whose sandbox stopped can no longer sit
unseen, and expiry alone never releases one

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory:
"Observed-run workspace preservation fence").

**What happened?**
`recoveryActionKind` lists only unresolved recovery actions, so
`observed_workspace_preservation_fence` reported nothing: its fence
lives on a resolved action. The kind registered no settle, and its fence
owner never carried the provider proof.

**Expected behavior**
Every `stop_pending` fence is listed. The fence is `gone`
(`provider_stopped`) once its run's one remote lease has a termination
receipt and a confirmed stop-and-retain. The reconciler then settles it
with `settleStoppedObservedWorkspace`, which re-checks every fence field
under its own locks. An expired coordinator lease without that receipt
is `unprovable` and is never released.

## What Changed

- `recovery/ownership-registry.ts`:
- `recoveryActionKind` takes `includeResolved`, which only the observed
fence uses.
- `resolveFenceHolders` adds `provider: "stopped"` for observed fences
whose run lease carries the stop proof.
- The kind registers a settle that calls
`settleStoppedObservedWorkspace`. A source-busy lock conflict is kept
for a later pass.
- `native-observed-workspace-preservation.ts`: pinning a fence wakes the
ownership worker in the same transaction.
- The settling-kinds list in `ownership-reconciliation-report.test.ts`
is updated.

## Verification

- `heartbeat-observed-workspace-preservation.test.ts`: 29 pass. The new
test "settles the fence from the shared reconciler once the provider
stop is proven, never before" fails on `master` and passes here:
- a pinned fence with an expired lease is
`fence_expired_without_provider_proof` and is not settled;
- after the provider stop, a read-only pass sees `provider_stopped` and
changes nothing;
- the settle moves the fence to `stopped` and the coordinator to
`terminal_failure`, and the saved files are untouched.

All 15 existing "keeps unsafe observed source held" cases and the
stale-evidence cases still pass.
- `ownership-reconciliation-report.test.ts` (25) passes.
- The server package typecheck passes.

## Risks

- The kind's listing now includes resolved recovery actions that carry a
`stop_pending` fence. Like every recovery-action kind, it is an
evidence-predicate scan; no new index is added.
- The settle is the existing worker's function with all its fence
checks. The reconciler is a backstop for that worker, not a new release
rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-08 05:56:08 -06:00
1 parent 7c56147bb3
commit 82a1fda9b1
6 files changed
+137 -41

No files matched your search

@@ -174,6 +174,7 @@ export async function pinFailedObservedWorkspace(db: Db, companyId: string, runI
&& lease.metadata?.workspaceRecoveryCleanup == null;
if (!active) return false;
await notifyNativeEnvironmentSettlement(tx, { companyId, runId, issueId: task.id });
await notifyDeliveryWork(tx, DELIVERY_QUEUES.ownershipInvariantReport, JSON.stringify({ companyId, runId }));
const retained = await recordRequiredWorkspaceRestoreFailure(tx, run, { workspaceRestoreFailure: "restore_failed" }, lease);
if (!retained || !hasRequiredWorkspaceRecovery(retained.resultJson)) throw new Error("observed_workspace_retention_missing");
const [action] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, companyId),