refactor(recovery): one process boot id and a cached controller identity (#19851)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Restart recovery decides when a record owned by a dead server
process may be released
> - One server process minted three different boot UUIDs, one per module
(`controller-process-identity.ts:13`, `native-restart-recovery.ts:284`,
`legacy-controller-lease.ts:8`)
> - The controller identity also ran `ps -o lstart` on macOS at every
native claim, preparation check and lease renewal, which is on the path
before `turn.start`
> - The durable ownership plan (Phase 1, step 1) needs one process
identity before it can add one owner stamp and one prover
> - This pull request makes the three modules stamp one boot id and
reads the process start time from the OS once
> - The benefit is one meaning of "owned by this process" across tables,
and no `ps` spawn on the turn path

## Linked Issues or Issue Description

Refs #19831 (plan:
`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`, Phase
1). Refs #19846 (Phase 0 inventory, finding 1, and the TTFT baseline row
"Controller identity").

**Problem**
One process has three boot ids, so the same fact ("this record is owned
by this process") has three encodings. The uncached controller identity
spawns `ps` on macOS on every claim and renewal.

**Expected behavior**
One boot id per process. The OS start-time read runs once per process.

**Impact**
Phase 1 needs one identity to build `OwnerStamp` and `proveOwnerGone`
on. The turn path loses one subprocess spawn per claim and renewal.

## What Changed

- `controller-process-identity.ts` owns the single process boot id. It
adds `currentProcessStartedAtMs()`, which caches a successful OS
start-time read for the life of the process and retries a failed read.
It also exports `currentProcessSpaceHash()` for the coming prover.
- `native-restart-recovery.ts`: `currentNativeControllerIdentity()` and
`isCurrentNativeControllerProcess()` use the shared boot id and the
cached start time. Its own `randomUUID()` boot id is removed. The
fallback to the server-info start time is kept.
- `legacy-controller-lease.ts`: `legacyControllerBootId` is now the
shared boot id.
- Tests in `native-controller-identity.test.ts`: one boot id across
native claims, legacy leases and creation intents; one OS read across
many claims and renewals; a failed read is retried, not cached.

## Verification

- `npx vitest run
src/services/native-runtime/native-controller-identity.test.ts
src/services/legacy-controller-lease.test.ts
src/__tests__/heartbeat-pending-cleanup-sweep.test.ts
src/__tests__/sandbox-creation-operations.test.ts
src/services/heartbeat/upgrade-readiness-post-result.test.ts
src/services/heartbeat/recovery.test.ts` (in `server/`): 6 files pass.
- `npx tsc --noEmit -p server`: no errors.
- I checked every reader of the three ids. Each one compares a stored
column only with the id of the module that writes that column
(`heartbeat_runs.controller_boot_id` with the legacy id,
`native_run_finalizations.controller_boot_id` with the native id,
metadata `bootId` with the creation id). No reader compares the ids of
two modules, so making them equal changes no decision inside one
process.

## Risks

- Low risk. No schema change and no persisted shape change.
- Previous-release story (`rules/upgrade-safe-run-state.mdc`): rows
saved by an older release carry that release's random per-module ids.
They never equal the new process's id, exactly as a row from any earlier
boot never did. They carry forward with no migration.
- The start-time cache is correct because a process's own start time
never changes. A failed read is not cached, so a transient `ps` failure
does not pin the fallback value.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/services/controller-process-identity.ts

@@ -10,6 +10,15 @@ import { readProcessStartedAt } from "./hot-restart.js";
  */
 export type ControllerProcess = { bootId: string; pid: number; hostHash: string; processStartedAt: string | null };
 
+/**
+ * The one boot id of this server process. Native coordinator claims
+ * (`native_run_finalizations.controller_boot_id`), legacy controller leases
+ * (`heartbeat_runs.controller_boot_id`), sandbox creation intents and teardown
+ * claims all stamp this value, so "owned by this process" means the same thing
+ * in every table. Records written by an older release carry that release's
+ * random per-module ids; those never equal this value, exactly as a record
+ * from any earlier boot never did.
+ */
 const bootId = randomUUID();
 const UNKNOWN_PROCESS_SPACE = "unknown";
 
@@ -38,7 +47,6 @@ function processSpaceHash(): string {
 }
 
 const hostHash = processSpaceHash();
-let processStartedAt: Promise<string | null> | undefined;
 
 async function observedProcessStartedAt(pid: number): Promise<number | null> {
   try {
@@ -49,14 +57,38 @@ async function observedProcessStartedAt(pid: number): Promise<number | null> {
   }
 }
 
+// This process's own start time never changes, so it is read from the OS once.
+// On macOS the read runs `ps -o lstart`; uncached, it ran on every native
+// claim, preparation check and lease renewal. A failed read is not cached, so
+// a transient failure is retried by the next caller.
+let ownStartedAt: Promise<number | null> | undefined;
+
+/** OS-observed start time of this server process (epoch ms), read at most once
+ * successfully per process. Null when the OS cannot report it. */
+export function currentProcessStartedAtMs(): Promise<number | null> {
+  if (!ownStartedAt) {
+    const read = observedProcessStartedAt(process.pid);
+    ownStartedAt = read;
+    void read.then(value => {
+      if (value === null && ownStartedAt === read) ownStartedAt = undefined;
+    });
+  }
+  return ownStartedAt;
+}
+
+/** Test seam: forget the cached start time so a test can stub the OS read. */
+export function resetCurrentProcessStartedAtForTests(): void {
+  ownStartedAt = undefined;
+}
+
 export function currentControllerBootId(): string {
   return bootId;
 }
 
 export async function currentControllerProcess(): Promise<ControllerProcess> {
-  processStartedAt ??= observedProcessStartedAt(process.pid)
-    .then(value => value === null ? null : new Date(value).toISOString());
-  return { bootId, pid: process.pid, hostHash, processStartedAt: await processStartedAt };
+  const startedAt = await currentProcessStartedAtMs();
+  return { bootId, pid: process.pid, hostHash,
+    processStartedAt: startedAt === null ? null : new Date(startedAt).toISOString() };
 }
 
 export function readControllerProcess(value: unknown): ControllerProcess | null {

server/src/services/legacy-controller-lease.ts

@@ -1,11 +1,13 @@
 import { notifyActiveExecutionRecovery, notifyLiveRunRecovery } from "./live-run-recovery-notifications.js";
-import { randomUUID } from "node:crypto";
 import { and, eq, gt, lte, sql } from "drizzle-orm";
 import { heartbeatRuns, livenessDb, type Db } from "@paperclipai/db";
 import type { LegacyControllerLeaseLossCause, LegacyControllerLeaseLossTiming } from "./run-cancellation.js";
+import { randomUUID } from "node:crypto";
+import { currentControllerBootId } from "./controller-process-identity.js";
 
-// A boot UUID has meaning across containers; a numeric PID does not.
-export const legacyControllerBootId = randomUUID();
+// A boot UUID has meaning across containers; a numeric PID does not. Legacy
+// leases stamp the single process boot id shared with native claims.
+export const legacyControllerBootId = currentControllerBootId();
 export const LEGACY_CONTROLLER_LEASE_MS = 60_000;
 export const LEGACY_CONTROLLER_RENEW_MS = 10_000;
 

server/src/services/native-runtime/native-controller-identity.test.ts

@@ -2,7 +2,13 @@ import { afterEach, expect, it, vi } from "vitest";
 import * as childProcess from "node:child_process";
 import * as hotRestart from "../hot-restart.js";
 import { getServerInfoSnapshot, resetServerInfoCacheForTests } from "../../server-info.js";
-import { currentNativeControllerIdentity } from "./native-restart-recovery.js";
+import { currentNativeControllerIdentity, isCurrentNativeControllerProcess } from "./native-restart-recovery.js";
+import {
+  currentControllerBootId,
+  currentControllerProcess,
+  resetCurrentProcessStartedAtForTests,
+} from "../controller-process-identity.js";
+import { legacyControllerBootId } from "../legacy-controller-lease.js";
 
 vi.mock("node:child_process", async importOriginal => ({
   ...await importOriginal<typeof import("node:child_process")>(),
@@ -14,6 +20,7 @@ vi.mock("node:child_process", async importOriginal => ({
 afterEach(() => {
   vi.restoreAllMocks();
   resetServerInfoCacheForTests();
+  resetCurrentProcessStartedAtForTests();
 });
 
 it.each([true, false])("keeps controller identity independent of Git commands (OS start available=%s)", async available => {
@@ -35,3 +42,39 @@ it.each([true, false])("keeps controller identity independent of Git commands (O
   expect(first.processStartedAt.toISOString()).toBe(available ? observed : fallback);
   expect(subprocess).not.toHaveBeenCalled();
 });
+
+it("stamps one boot id for native claims, legacy leases and creation intents", async () => {
+  vi.spyOn(hotRestart, "readProcessStartedAt").mockResolvedValue("2026-11-02T12:00:00.000Z");
+  const native = await currentNativeControllerIdentity();
+  const controller = await currentControllerProcess();
+
+  expect(native.bootId).toBe(currentControllerBootId());
+  expect(legacyControllerBootId).toBe(currentControllerBootId());
+  expect(controller.bootId).toBe(currentControllerBootId());
+  expect(controller.processStartedAt).toBe(native.processStartedAt.toISOString());
+  expect(isCurrentNativeControllerProcess({ controllerBootId: legacyControllerBootId, controllerPid: process.pid })).toBe(true);
+});
+
+it("reads this process's start time from the OS once across claims and renewals", async () => {
+  const read = vi.spyOn(hotRestart, "readProcessStartedAt").mockResolvedValue("2026-11-02T12:00:00.000Z");
+
+  await Promise.all(Array.from({ length: 5 }, () => currentNativeControllerIdentity()));
+  for (let i = 0; i < 5; i++) {
+    await currentNativeControllerIdentity();
+    await currentControllerProcess();
+  }
+
+  expect(read).toHaveBeenCalledTimes(1);
+});
+
+it("retries a failed start-time read instead of caching the failure", async () => {
+  const read = vi.spyOn(hotRestart, "readProcessStartedAt")
+    .mockRejectedValueOnce(new Error("ps unavailable"))
+    .mockResolvedValue("2026-11-02T12:00:00.000Z");
+
+  expect((await currentControllerProcess()).processStartedAt).toBeNull();
+  expect((await currentControllerProcess()).processStartedAt).toBe("2026-11-02T12:00:00.000Z");
+  await currentNativeControllerIdentity();
+
+  expect(read).toHaveBeenCalledTimes(2);
+});

server/src/services/native-runtime/native-restart-recovery.ts

@@ -35,7 +35,7 @@ import {
 } from "@paperclipai/db";
 import { REMOTE_CONTROLLER_LEASE_MS } from "./native-runner-recovery-contract.js";
 import { readProcessStartedAt } from "../hot-restart.js";
-import { processStartsDefinitelyDiffer } from "../controller-process-identity.js";
+import { currentControllerBootId, currentProcessStartedAtMs, processStartsDefinitelyDiffer } from "../controller-process-identity.js";
 import { getServerProcessStartedAt } from "../../server-info.js";
 import { redactSensitiveText } from "../../redaction.js";
 import { isUniqueViolation } from "../../db-errors.js";
@@ -282,7 +282,6 @@ export async function restartResumeKeepsAttempt(tx: Pick<Db, "select">, input: {
   return work !== undefined;
 }
 
-const controllerBootId = randomUUID();
 
 function processIsAlive(pid: number | null | undefined): boolean {
   if (!pid || !Number.isInteger(pid) || pid <= 0) return false;
@@ -324,14 +323,17 @@ async function observedProcessStart(pid: number): Promise<Date | null> {
   }
 }
 
+/** The native controller stamp of this process. The boot id is the single
+ * process boot id (`controller-process-identity.ts`) and the start time is read
+ * from the OS once, so claims and renewals on the turn path spawn no `ps`. */
 export async function currentNativeControllerIdentity(): Promise<NativeControllerIdentity> {
-  const observed = await observedProcessStart(process.pid);
+  const observed = await currentProcessStartedAtMs();
   const fallback = new Date(getServerProcessStartedAt());
   return {
-    bootId: controllerBootId,
+    bootId: currentControllerBootId(),
     pid: process.pid,
     processStartedAt:
-      observed ?? (Number.isNaN(fallback.getTime()) ? new Date() : fallback),
+      observed !== null ? new Date(observed) : (Number.isNaN(fallback.getTime()) ? new Date() : fallback),
   };
 }
 
@@ -339,7 +341,7 @@ export function isCurrentNativeControllerProcess(owner: {
   controllerBootId: string | null;
   controllerPid: number | null;
 }): boolean {
-  return owner.controllerBootId === controllerBootId && owner.controllerPid === process.pid;
+  return owner.controllerBootId === currentControllerBootId() && owner.controllerPid === process.pid;
 }
 
 function sameProcessStart(left: Date | null, right: Date | null): boolean {