feat(recovery): settle workspace runtime leases of proven-gone owners on the shared path (#20015)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that starts, stops or repairs a workspace's runtime services
takes a runtime lease, so two tasks never drive the same workspace at
once
> - A lease whose owner task or run has ended must be released. Until
now that happened only when the next claimant arrived and checked, or
when 30 minutes of inactivity passed
> - The durable ownership plan releases a record when its owner is
proven gone, by identity and never by age, through the shared reconciler
> - This pull request registers the runtime lease with the reconciler.
Proof releases a lease; the 30-minute idle rule stays only as a fairness
policy at claim time
> - The benefit is that a workspace locked by a finished task frees
itself at once, and a live task's lease is never taken by age

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 2 sections A and C), #19838 (inventory:
"Execution workspace runtime lease", and the "Released only by TTL/age"
list).

**What happened?**
`execution_workspace_runtime_lease` was a report-only kind. A lease
whose issue or run had ended stayed held until another claim came, or
until its 30-minute TTL expired. The claim path also mixed the TTL with
the ownership check.

**Expected behavior**
An issue- or run-owned lease is released when its owner is proven gone:
the issue is missing, hidden or terminal, or the run is missing or
terminal. Age is not proof. An agent-scoped owner has no identity to
prove, so the lease's expiry plus the next claim's locked takeover is
the fenced handoff.

## What Changed

- New `workspace-runtime-lease-settlement.ts`:
`settleExecutionWorkspaceRuntimeLease` locks the row, requires the same
owner key, re-proves that owner gone, and only then deletes the lease
(compare-and-swap).
- `workspace-runtime-leases.ts`: the new
`proveWorkspaceRuntimeLeaseOwnerGone` holds the identity proof. The
claim path checks it first, then applies the 30-minute idle reclaim as a
separate fairness policy.
- `recovery/ownership-registry.ts`: the kind registers the settle.
Owners are now `issue`, `run`, or a fence on `expires_at` for
agent-scoped keys. `RecordOwner` gains an `issue` kind.
- `recovery/ownership-invariant-report.ts`: issue owners are classified
(missing, hidden or terminal means gone).

Salvaged from the stopped Phase 2 workspace agent's uncommitted work. I
resolved it against current `master` and verified it.

## Verification

- New `workspace-runtime-lease-ownership.test.ts`: 8 pass. Six fail on
`master` and pass here:
  - a crash after the claim;
  - hidden or missing issues, and terminal or missing runs;
  - an agent-scoped lease never settles;
  - a read-only pass changes nothing;
  - a lease taken over after the proof is kept;
  - the release reason names the ownership proof, not the idle policy.

Two guards pass on both: a live owner is never released by age, and the
idle reclaim still works at claim time.
- `ownership-reconciliation-report.test.ts` (25),
`workspace-runtime-leases.test.ts` (10) and
`execution-workspace-runtime-lease-route.test.ts` (4) pass.
- The server package typecheck passes.

## Risks

- The reconciler now deletes leases whose owner is proven gone. It
always re-proves under the row lock against the exact owner key.
- The idle reclaim behaves as before at claim time.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/recovery/ownership-invariant-report.ts

@@ -128,6 +128,7 @@ export function createOwnershipReportState(): OwnershipReportState {
 }
 
 const TERMINAL_RUN_STATUSES = ["succeeded", "failed", "cancelled", "timed_out", "interrupted"];
+const TERMINAL_ISSUE_STATUSES = ["done", "cancelled"];
 const ACTIVE_RUN_STATUSES = ["queued", "running", "scheduled_retry"];
 /** An in-progress issue must show a next action within this time. */
 const NEXT_ACTION_GRACE_MS = 5 * 60_000;
@@ -156,6 +157,21 @@ async function classifyRunOwners(db: Db, records: InProgressRecord[]): Promise<M
         : { verdict: "alive", reason: "owner_run_active" });
     }
   }
+  // Issue owners share the map under an `issue:` key (run ids are bare UUIDs).
+  const issueIds = [...new Set(records.flatMap(record => record.owner.kind === "issue" ? [record.owner.issueId] : []))];
+  for (let index = 0; index < issueIds.length; index += 500) {
+    const batch = issueIds.slice(index, index + 500);
+    const rows = await db.select({ id: issues.id, status: issues.status, hiddenAt: issues.hiddenAt }).from(issues)
+      .where(inArray(issues.id, batch));
+    const byId = new Map(rows.map(row => [row.id, row]));
+    for (const issueId of batch) {
+      const row = byId.get(issueId);
+      proofs.set(`issue:${issueId}`, !row ? { verdict: "gone", reason: "owner_issue_missing" }
+        : row.hiddenAt ? { verdict: "gone", reason: "owner_issue_hidden" }
+        : TERMINAL_ISSUE_STATUSES.includes(row.status) ? { verdict: "gone", reason: "owner_issue_terminal" }
+        : { verdict: "alive", reason: "owner_issue_active" });
+    }
+  }
   return proofs;
 }
 
@@ -165,6 +181,7 @@ async function classify(owner: RecordOwner, runProofs: Map<string, Classificatio
     case "process": return proveOwnerGone(owner.stamp, probe);
     case "fence": return proveOwnerGone(owner.stamp, owner.fence, now);
     case "run": return runProofs.get(owner.runId) ?? { verdict: "unprovable", reason: "owner_run_unread" };
+    case "issue": return runProofs.get(`issue:${owner.issueId}`) ?? { verdict: "unprovable", reason: "owner_issue_unread" };
     case "operator": return { verdict: "alive", reason: "operator_owned" };
     // A queue is not a process: it is never proven gone. A deadline long past
     // means the queue is not acting, which is a gap to report, not a release.

server/src/services/recovery/ownership-registry.ts

@@ -57,6 +57,8 @@ import {
   releasableRetainedOrphanCondition,
   settleRetainedOrphanSandbox,
 } from "../environment-retained-orphan-settlement.js";
+import { settleExecutionWorkspaceRuntimeLease } from "../workspace-runtime-lease-settlement.js";
+import { parseWorkspaceRuntimeLeaseOwnerKey } from "../workspace-runtime-leases.js";
 
 /**
  * Registry of ownable record kinds (plan section C).
@@ -82,6 +84,8 @@ export type RecordOwner =
   | { kind: "fence"; stamp: RemoteOwnerStamp; fence: RemoteFenceState }
   /** A run id only. The record's owner is that run's execution. */
   | { kind: "run"; runId: string }
+  /** An issue id only. Gone when the issue is missing, hidden or terminal. */
+  | { kind: "issue"; issueId: string }
   /** A person or agent assignment, such as an operator-owned recovery action. */
   | { kind: "operator"; ownerType: string | null }
   /** A durable work queue that acts on the record by its saved deadline. The
@@ -511,6 +515,21 @@ function spoolKind(id: string, title: string, obligation: Obligation, directory:
   };
 }
 
+// ----------------------------------------------------------------------------
+// execution_workspace_runtime_leases
+
+/** The lease's owner key names its owner: an issue or run is proven by
+ * identity; an agent-scoped (or unparseable) owner is a fence on the lease
+ * expiry that only a claim's locked takeover moves. */
+function runtimeLeaseOwner(row: typeof executionWorkspaceRuntimeLeases.$inferSelect): RecordOwner {
+  const owner = parseWorkspaceRuntimeLeaseOwnerKey(row.ownerKey);
+  if (owner?.kind === "issue") return { kind: "issue", issueId: owner.id };
+  if (owner?.kind === "run") return { kind: "run", runId: owner.id };
+  return { kind: "fence", stamp: { kind: "remote", scope: "execution_workspace_runtime_lease", generation: null,
+    fenceToken: row.ownerKey }, fence: { currentFenceToken: row.ownerKey, currentGeneration: null,
+    leaseExpiresAt: row.expiresAt } };
+}
+
 // ----------------------------------------------------------------------------
 // The registry
 
@@ -715,15 +734,18 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [
     and(eq(workspaceOperations.phase, "remote_workspace_migration"), eq(workspaceOperations.status, "running")),
     () => ({ kind: "none" })),
   {
+    // Shared path since Phase 2: a lease whose issue or run owner is proven
+    // gone (by identity, never age) is released here and at the next claim.
+    // An agent-scoped owner is a fence on `expires_at`: alive while held,
+    // unprovable after; only the next claim's locked takeover moves it.
     id: "execution_workspace_runtime_lease", title: "Execution workspace runtime lease", obligation: "ownership",
-    storage: "database", settle: null, invariantChecks: [1, 2],
+    storage: "database", settle: settleExecutionWorkspaceRuntimeLease, invariantChecks: [1, 2],
     listInProgress: async (db, limit, offset) => {
       const rows = await db.select().from(executionWorkspaceRuntimeLeases).orderBy(asc(executionWorkspaceRuntimeLeases.id)).limit(limit + 1).offset(offset);
       const result = page(rows, limit);
       return { truncated: result.truncated, records: result.rows.map(row => ({
         recordId: row.id, companyId: row.companyId, runId: row.ownerRunId, issueId: row.ownerIssueId,
-        owner: row.ownerRunId ? runOwner(row.ownerRunId) : { kind: "operator", ownerType: row.ownerKey.split(":")[0] ?? null },
-        updatedAt: row.updatedAt,
+        owner: runtimeLeaseOwner(row), updatedAt: row.updatedAt,
       })) };
     },
   },

server/src/services/workspace-runtime-lease-settlement.ts

@@ -0,0 +1,49 @@
+import { and, eq } from "drizzle-orm";
+import { executionWorkspaceRuntimeLeases, type Db } from "@paperclipai/db";
+import { persistActivity, publishActivity, type ActivityPublication } from "./activity-log.js";
+import type { OwnerProof } from "./owner-proof.js";
+import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
+import { proveWorkspaceRuntimeLeaseOwnerGone } from "./workspace-runtime-leases.js";
+
+/**
+ * Settle of the `execution_workspace_runtime_lease` ownership kind (plan Phase 2).
+ *
+ * The shared reconciler calls this only after it proved the lease's issue or run
+ * owner gone (issue terminal, hidden or missing; run terminal or missing). Settle
+ * releases exactly that lease: under the row lock it re-reads the row, requires
+ * the same owner key, and re-proves the owner gone, so a lease renewed or taken
+ * over since the proof is never released. Age plays no part. Agent-scoped leases
+ * are never settled here: no identity proves such an owner gone, and their expiry
+ * plus the next claim's locked takeover is the fenced handoff.
+ */
+export async function settleExecutionWorkspaceRuntimeLease(db: Db, record: InProgressRecord,
+  _context: { now: Date; proof: OwnerProof }): Promise<SettleOutcome> {
+  const ownerKey = record.owner.kind === "issue" ? `issue:${record.owner.issueId}`
+    : record.owner.kind === "run" ? `run:${record.owner.runId}` : null;
+  if (!ownerKey) return { settled: false, reason: "owner_not_provable" };
+  let publication: ActivityPublication | undefined;
+  const outcome = await db.transaction(async transaction => {
+    const tx = transaction as unknown as Db;
+    const [row] = await tx.select().from(executionWorkspaceRuntimeLeases)
+      .where(and(eq(executionWorkspaceRuntimeLeases.id, record.recordId),
+        eq(executionWorkspaceRuntimeLeases.ownerKey, ownerKey)))
+      .for("update");
+    if (!row) return { settled: false, reason: "lease_no_longer_held" };
+    const gone = await proveWorkspaceRuntimeLeaseOwnerGone(tx, row);
+    if (!gone) return { settled: false, reason: "owner_changed_since_proof" };
+    const deleted = await tx.delete(executionWorkspaceRuntimeLeases)
+      .where(and(eq(executionWorkspaceRuntimeLeases.id, row.id), eq(executionWorkspaceRuntimeLeases.ownerKey, ownerKey)))
+      .returning({ id: executionWorkspaceRuntimeLeases.id });
+    if (!deleted.length) return { settled: false, reason: "lease_no_longer_held" };
+    // The lease row is gone; the activity entry keeps who held it and why it was released.
+    publication = (await persistActivity(tx, {
+      companyId: row.companyId, actorType: "system", actorId: "ownership-reconciler",
+      action: "execution_workspace.runtime_lease_released", entityType: "execution_workspace",
+      entityId: row.executionWorkspaceId, runId: row.ownerRunId, issueId: row.ownerIssueId,
+      details: { leaseId: row.id, previousOwnerKey: row.ownerKey, lastAction: row.lastAction, proof: gone },
+    })).publication;
+    return { settled: true, reason: gone };
+  });
+  if (outcome.settled && publication) publishActivity(publication);
+  return outcome;
+}

server/src/services/workspace-runtime-leases.ts

@@ -31,10 +31,12 @@ const TERMINAL_HEARTBEAT_RUN_STATUSES = new Set([
 export const LEASED_WORKSPACE_RUNTIME_ACTIONS: readonly string[] = ["start", "stop", "restart", "repair"];
 
 /**
- * Upper bound on how long a lease survives without the owner touching it. Recovery
- * from a stale owner is otherwise driven by owner eligibility, which is cheaper and
- * more precise; the TTL only bounds the cases eligibility cannot see (for example an
- * agent-scoped owner with no run or issue identity).
+ * Idle-reclaim policy window. A lease untouched this long may be taken over by the
+ * next claimant at claim time. This is a fairness policy, not an ownership proof:
+ * dead owners (terminal, hidden or missing issue; terminal or missing run) are
+ * released by identity, with no age, both at claim time and by the shared
+ * ownership reconciler. For agent-scoped owners, which no identity can prove gone,
+ * expiry plus the locked takeover is the fenced handoff.
  */
 export const WORKSPACE_RUNTIME_LEASE_TTL_MS = 30 * 60 * 1000;
 
@@ -72,7 +74,7 @@ export function buildWorkspaceRuntimeLeaseOwnerKey(owner: WorkspaceRuntimeLeaseO
   return null;
 }
 
-function parseOwnerKey(ownerKey: string): { kind: "issue" | "run" | "agent"; id: string } | null {
+export function parseWorkspaceRuntimeLeaseOwnerKey(ownerKey: string): { kind: "issue" | "run" | "agent"; id: string } | null {
   const separator = ownerKey.indexOf(":");
   if (separator <= 0) return null;
   const kind = ownerKey.slice(0, separator);
@@ -82,14 +84,18 @@ function parseOwnerKey(ownerKey: string): { kind: "issue" | "run" | "agent"; id:
   return { kind, id };
 }
 
-async function evaluateStaleOwner(
-  tx: LeaseTx,
-  lease: ExecutionWorkspaceRuntimeLeaseRow,
-  now: Date,
-): Promise<WorkspaceRuntimeLeaseStaleReason | null> {
-  if (lease.expiresAt.getTime() <= now.getTime()) return "lease_expired";
-
-  const owner = parseOwnerKey(lease.ownerKey);
+/**
+ * Proves by identity that an issue- or run-owned lease's owner is gone: the issue
+ * is missing, hidden or terminal, or the run is missing or terminal. Returns null
+ * while the owner lives, and for agent-scoped or unparseable owners, which have no
+ * identity to prove. The shared ownership reconciler's settle and the claim path
+ * both use it, so the lane is released on the same proof wherever it is checked.
+ */
+export async function proveWorkspaceRuntimeLeaseOwnerGone(
+  tx: Pick<Db, "select">,
+  lease: Pick<ExecutionWorkspaceRuntimeLeaseRow, "ownerKey" | "companyId">,
+): Promise<Exclude<WorkspaceRuntimeLeaseStaleReason, "lease_expired"> | null> {
+  const owner = parseWorkspaceRuntimeLeaseOwnerKey(lease.ownerKey);
   if (owner?.kind === "issue") {
     const ownerIssue = await tx
       .select({ status: issues.status, hiddenAt: issues.hiddenAt })
@@ -113,11 +119,26 @@ async function evaluateStaleOwner(
     return null;
   }
 
-  // Agent-scoped and unparseable owners have no lifecycle to inspect, so the TTL
-  // checked above is their only recovery path.
   return null;
 }
 
+async function evaluateStaleOwner(
+  tx: LeaseTx,
+  lease: ExecutionWorkspaceRuntimeLeaseRow,
+  now: Date,
+): Promise<WorkspaceRuntimeLeaseStaleReason | null> {
+  // Ownership proof first: a gone issue or run releases the lane with no age
+  // involved (the same proof the shared reconciler settles on).
+  const gone = await proveWorkspaceRuntimeLeaseOwnerGone(tx, lease);
+  if (gone) return gone;
+  // Idle reclaim: a POLICY, not an ownership proof. A lease untouched for the
+  // TTL is handed to the next claimant even when its issue or run still lives
+  // (fairness between issues sharing a workspace). For agent-scoped owners,
+  // which no identity can prove gone, expiry plus this locked takeover is the
+  // fenced handoff. The reconciler never releases a lease on this rule.
+  return lease.expiresAt.getTime() <= now.getTime() ? "lease_expired" : null;
+}
+
 function throwLeaseConflict(input: {
   lease: ExecutionWorkspaceRuntimeLeaseRow;
   requestedAction: string;