From fa86162790cb17aba8a08eb2c38c213a7a632877 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 7 Nov 2026 10:49:12 -0600 Subject: [PATCH] refactor(recovery): one process boot id and a cached controller identity (#19851) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Restart recovery decides when a record owned by a dead server process may be released > - One server process minted three different boot UUIDs, one per module (`controller-process-identity.ts:13`, `native-restart-recovery.ts:284`, `legacy-controller-lease.ts:8`) > - The controller identity also ran `ps -o lstart` on macOS at every native claim, preparation check and lease renewal, which is on the path before `turn.start` > - The durable ownership plan (Phase 1, step 1) needs one process identity before it can add one owner stamp and one prover > - This pull request makes the three modules stamp one boot id and reads the process start time from the OS once > - The benefit is one meaning of "owned by this process" across tables, and no `ps` spawn on the turn path ## Linked Issues or Issue Description Refs #19831 (plan: `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`, Phase 1). Refs #19846 (Phase 0 inventory, finding 1, and the TTFT baseline row "Controller identity"). **Problem** One process has three boot ids, so the same fact ("this record is owned by this process") has three encodings. The uncached controller identity spawns `ps` on macOS on every claim and renewal. **Expected behavior** One boot id per process. The OS start-time read runs once per process. **Impact** Phase 1 needs one identity to build `OwnerStamp` and `proveOwnerGone` on. The turn path loses one subprocess spawn per claim and renewal. ## What Changed - `controller-process-identity.ts` owns the single process boot id. It adds `currentProcessStartedAtMs()`, which caches a successful OS start-time read for the life of the process and retries a failed read. It also exports `currentProcessSpaceHash()` for the coming prover. - `native-restart-recovery.ts`: `currentNativeControllerIdentity()` and `isCurrentNativeControllerProcess()` use the shared boot id and the cached start time. Its own `randomUUID()` boot id is removed. The fallback to the server-info start time is kept. - `legacy-controller-lease.ts`: `legacyControllerBootId` is now the shared boot id. - Tests in `native-controller-identity.test.ts`: one boot id across native claims, legacy leases and creation intents; one OS read across many claims and renewals; a failed read is retried, not cached. ## Verification - `npx vitest run src/services/native-runtime/native-controller-identity.test.ts src/services/legacy-controller-lease.test.ts src/__tests__/heartbeat-pending-cleanup-sweep.test.ts src/__tests__/sandbox-creation-operations.test.ts src/services/heartbeat/upgrade-readiness-post-result.test.ts src/services/heartbeat/recovery.test.ts` (in `server/`): 6 files pass. - `npx tsc --noEmit -p server`: no errors. - I checked every reader of the three ids. Each one compares a stored column only with the id of the module that writes that column (`heartbeat_runs.controller_boot_id` with the legacy id, `native_run_finalizations.controller_boot_id` with the native id, metadata `bootId` with the creation id). No reader compares the ids of two modules, so making them equal changes no decision inside one process. ## Risks - Low risk. No schema change and no persisted shape change. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): rows saved by an older release carry that release's random per-module ids. They never equal the new process's id, exactly as a row from any earlier boot never did. They carry forward with no migration. - The start-time cache is correct because a process's own start time never changes. A failed read is not cached, so a transient `ps` failure does not pin the fallback value. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../services/controller-process-identity.ts | 40 +++++++++++++++-- .../src/services/legacy-controller-lease.ts | 8 ++-- .../native-controller-identity.test.ts | 45 ++++++++++++++++++- .../native-runtime/native-restart-recovery.ts | 14 +++--- 4 files changed, 93 insertions(+), 14 deletions(-) diff --git a/server/src/services/controller-process-identity.ts b/server/src/services/controller-process-identity.ts index 2bf79735f6f..1a537a1eb0b 100644 --- a/server/src/services/controller-process-identity.ts +++ b/server/src/services/controller-process-identity.ts @@ -10,6 +10,15 @@ import { readProcessStartedAt } from "./hot-restart.js"; */ export type ControllerProcess = { bootId: string; pid: number; hostHash: string; processStartedAt: string | null }; +/** + * The one boot id of this server process. Native coordinator claims + * (`native_run_finalizations.controller_boot_id`), legacy controller leases + * (`heartbeat_runs.controller_boot_id`), sandbox creation intents and teardown + * claims all stamp this value, so "owned by this process" means the same thing + * in every table. Records written by an older release carry that release's + * random per-module ids; those never equal this value, exactly as a record + * from any earlier boot never did. + */ const bootId = randomUUID(); const UNKNOWN_PROCESS_SPACE = "unknown"; @@ -38,7 +47,6 @@ function processSpaceHash(): string { } const hostHash = processSpaceHash(); -let processStartedAt: Promise | undefined; async function observedProcessStartedAt(pid: number): Promise { try { @@ -49,14 +57,38 @@ async function observedProcessStartedAt(pid: number): Promise { } } +// This process's own start time never changes, so it is read from the OS once. +// On macOS the read runs `ps -o lstart`; uncached, it ran on every native +// claim, preparation check and lease renewal. A failed read is not cached, so +// a transient failure is retried by the next caller. +let ownStartedAt: Promise | undefined; + +/** OS-observed start time of this server process (epoch ms), read at most once + * successfully per process. Null when the OS cannot report it. */ +export function currentProcessStartedAtMs(): Promise { + if (!ownStartedAt) { + const read = observedProcessStartedAt(process.pid); + ownStartedAt = read; + void read.then(value => { + if (value === null && ownStartedAt === read) ownStartedAt = undefined; + }); + } + return ownStartedAt; +} + +/** Test seam: forget the cached start time so a test can stub the OS read. */ +export function resetCurrentProcessStartedAtForTests(): void { + ownStartedAt = undefined; +} + export function currentControllerBootId(): string { return bootId; } export async function currentControllerProcess(): Promise { - processStartedAt ??= observedProcessStartedAt(process.pid) - .then(value => value === null ? null : new Date(value).toISOString()); - return { bootId, pid: process.pid, hostHash, processStartedAt: await processStartedAt }; + const startedAt = await currentProcessStartedAtMs(); + return { bootId, pid: process.pid, hostHash, + processStartedAt: startedAt === null ? null : new Date(startedAt).toISOString() }; } export function readControllerProcess(value: unknown): ControllerProcess | null { diff --git a/server/src/services/legacy-controller-lease.ts b/server/src/services/legacy-controller-lease.ts index a6a281db059..ae7656429b9 100644 --- a/server/src/services/legacy-controller-lease.ts +++ b/server/src/services/legacy-controller-lease.ts @@ -1,11 +1,13 @@ import { notifyActiveExecutionRecovery, notifyLiveRunRecovery } from "./live-run-recovery-notifications.js"; -import { randomUUID } from "node:crypto"; import { and, eq, gt, lte, sql } from "drizzle-orm"; import { heartbeatRuns, livenessDb, type Db } from "@paperclipai/db"; import type { LegacyControllerLeaseLossCause, LegacyControllerLeaseLossTiming } from "./run-cancellation.js"; +import { randomUUID } from "node:crypto"; +import { currentControllerBootId } from "./controller-process-identity.js"; -// A boot UUID has meaning across containers; a numeric PID does not. -export const legacyControllerBootId = randomUUID(); +// A boot UUID has meaning across containers; a numeric PID does not. Legacy +// leases stamp the single process boot id shared with native claims. +export const legacyControllerBootId = currentControllerBootId(); export const LEGACY_CONTROLLER_LEASE_MS = 60_000; export const LEGACY_CONTROLLER_RENEW_MS = 10_000; diff --git a/server/src/services/native-runtime/native-controller-identity.test.ts b/server/src/services/native-runtime/native-controller-identity.test.ts index fd11591da9a..fbd899aa498 100644 --- a/server/src/services/native-runtime/native-controller-identity.test.ts +++ b/server/src/services/native-runtime/native-controller-identity.test.ts @@ -2,7 +2,13 @@ import { afterEach, expect, it, vi } from "vitest"; import * as childProcess from "node:child_process"; import * as hotRestart from "../hot-restart.js"; import { getServerInfoSnapshot, resetServerInfoCacheForTests } from "../../server-info.js"; -import { currentNativeControllerIdentity } from "./native-restart-recovery.js"; +import { currentNativeControllerIdentity, isCurrentNativeControllerProcess } from "./native-restart-recovery.js"; +import { + currentControllerBootId, + currentControllerProcess, + resetCurrentProcessStartedAtForTests, +} from "../controller-process-identity.js"; +import { legacyControllerBootId } from "../legacy-controller-lease.js"; vi.mock("node:child_process", async importOriginal => ({ ...await importOriginal(), @@ -14,6 +20,7 @@ vi.mock("node:child_process", async importOriginal => ({ afterEach(() => { vi.restoreAllMocks(); resetServerInfoCacheForTests(); + resetCurrentProcessStartedAtForTests(); }); it.each([true, false])("keeps controller identity independent of Git commands (OS start available=%s)", async available => { @@ -35,3 +42,39 @@ it.each([true, false])("keeps controller identity independent of Git commands (O expect(first.processStartedAt.toISOString()).toBe(available ? observed : fallback); expect(subprocess).not.toHaveBeenCalled(); }); + +it("stamps one boot id for native claims, legacy leases and creation intents", async () => { + vi.spyOn(hotRestart, "readProcessStartedAt").mockResolvedValue("2026-11-02T12:00:00.000Z"); + const native = await currentNativeControllerIdentity(); + const controller = await currentControllerProcess(); + + expect(native.bootId).toBe(currentControllerBootId()); + expect(legacyControllerBootId).toBe(currentControllerBootId()); + expect(controller.bootId).toBe(currentControllerBootId()); + expect(controller.processStartedAt).toBe(native.processStartedAt.toISOString()); + expect(isCurrentNativeControllerProcess({ controllerBootId: legacyControllerBootId, controllerPid: process.pid })).toBe(true); +}); + +it("reads this process's start time from the OS once across claims and renewals", async () => { + const read = vi.spyOn(hotRestart, "readProcessStartedAt").mockResolvedValue("2026-11-02T12:00:00.000Z"); + + await Promise.all(Array.from({ length: 5 }, () => currentNativeControllerIdentity())); + for (let i = 0; i < 5; i++) { + await currentNativeControllerIdentity(); + await currentControllerProcess(); + } + + expect(read).toHaveBeenCalledTimes(1); +}); + +it("retries a failed start-time read instead of caching the failure", async () => { + const read = vi.spyOn(hotRestart, "readProcessStartedAt") + .mockRejectedValueOnce(new Error("ps unavailable")) + .mockResolvedValue("2026-11-02T12:00:00.000Z"); + + expect((await currentControllerProcess()).processStartedAt).toBeNull(); + expect((await currentControllerProcess()).processStartedAt).toBe("2026-11-02T12:00:00.000Z"); + await currentNativeControllerIdentity(); + + expect(read).toHaveBeenCalledTimes(2); +}); diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index f33e728ec2f..54c4087d299 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -35,7 +35,7 @@ import { } from "@paperclipai/db"; import { REMOTE_CONTROLLER_LEASE_MS } from "./native-runner-recovery-contract.js"; import { readProcessStartedAt } from "../hot-restart.js"; -import { processStartsDefinitelyDiffer } from "../controller-process-identity.js"; +import { currentControllerBootId, currentProcessStartedAtMs, processStartsDefinitelyDiffer } from "../controller-process-identity.js"; import { getServerProcessStartedAt } from "../../server-info.js"; import { redactSensitiveText } from "../../redaction.js"; import { isUniqueViolation } from "../../db-errors.js"; @@ -282,7 +282,6 @@ export async function restartResumeKeepsAttempt(tx: Pick, input: { return work !== undefined; } -const controllerBootId = randomUUID(); function processIsAlive(pid: number | null | undefined): boolean { if (!pid || !Number.isInteger(pid) || pid <= 0) return false; @@ -324,14 +323,17 @@ async function observedProcessStart(pid: number): Promise { } } +/** The native controller stamp of this process. The boot id is the single + * process boot id (`controller-process-identity.ts`) and the start time is read + * from the OS once, so claims and renewals on the turn path spawn no `ps`. */ export async function currentNativeControllerIdentity(): Promise { - const observed = await observedProcessStart(process.pid); + const observed = await currentProcessStartedAtMs(); const fallback = new Date(getServerProcessStartedAt()); return { - bootId: controllerBootId, + bootId: currentControllerBootId(), pid: process.pid, processStartedAt: - observed ?? (Number.isNaN(fallback.getTime()) ? new Date() : fallback), + observed !== null ? new Date(observed) : (Number.isNaN(fallback.getTime()) ? new Date() : fallback), }; } @@ -339,7 +341,7 @@ export function isCurrentNativeControllerProcess(owner: { controllerBootId: string | null; controllerPid: number | null; }): boolean { - return owner.controllerBootId === controllerBootId && owner.controllerPid === process.pid; + return owner.controllerBootId === currentControllerBootId() && owner.controllerPid === process.pid; } function sameProcessStart(left: Date | null, right: Date | null): boolean {