mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 22:56:58 +01:00
refactor(recovery): one process boot id and a cached controller identity (#19851)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Restart recovery decides when a record owned by a dead server process may be released > - One server process minted three different boot UUIDs, one per module (`controller-process-identity.ts:13`, `native-restart-recovery.ts:284`, `legacy-controller-lease.ts:8`) > - The controller identity also ran `ps -o lstart` on macOS at every native claim, preparation check and lease renewal, which is on the path before `turn.start` > - The durable ownership plan (Phase 1, step 1) needs one process identity before it can add one owner stamp and one prover > - This pull request makes the three modules stamp one boot id and reads the process start time from the OS once > - The benefit is one meaning of "owned by this process" across tables, and no `ps` spawn on the turn path ## Linked Issues or Issue Description Refs #19831 (plan: `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`, Phase 1). Refs #19846 (Phase 0 inventory, finding 1, and the TTFT baseline row "Controller identity"). **Problem** One process has three boot ids, so the same fact ("this record is owned by this process") has three encodings. The uncached controller identity spawns `ps` on macOS on every claim and renewal. **Expected behavior** One boot id per process. The OS start-time read runs once per process. **Impact** Phase 1 needs one identity to build `OwnerStamp` and `proveOwnerGone` on. The turn path loses one subprocess spawn per claim and renewal. ## What Changed - `controller-process-identity.ts` owns the single process boot id. It adds `currentProcessStartedAtMs()`, which caches a successful OS start-time read for the life of the process and retries a failed read. It also exports `currentProcessSpaceHash()` for the coming prover. - `native-restart-recovery.ts`: `currentNativeControllerIdentity()` and `isCurrentNativeControllerProcess()` use the shared boot id and the cached start time. Its own `randomUUID()` boot id is removed. The fallback to the server-info start time is kept. - `legacy-controller-lease.ts`: `legacyControllerBootId` is now the shared boot id. - Tests in `native-controller-identity.test.ts`: one boot id across native claims, legacy leases and creation intents; one OS read across many claims and renewals; a failed read is retried, not cached. ## Verification - `npx vitest run src/services/native-runtime/native-controller-identity.test.ts src/services/legacy-controller-lease.test.ts src/__tests__/heartbeat-pending-cleanup-sweep.test.ts src/__tests__/sandbox-creation-operations.test.ts src/services/heartbeat/upgrade-readiness-post-result.test.ts src/services/heartbeat/recovery.test.ts` (in `server/`): 6 files pass. - `npx tsc --noEmit -p server`: no errors. - I checked every reader of the three ids. Each one compares a stored column only with the id of the module that writes that column (`heartbeat_runs.controller_boot_id` with the legacy id, `native_run_finalizations.controller_boot_id` with the native id, metadata `bootId` with the creation id). No reader compares the ids of two modules, so making them equal changes no decision inside one process. ## Risks - Low risk. No schema change and no persisted shape change. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): rows saved by an older release carry that release's random per-module ids. They never equal the new process's id, exactly as a row from any earlier boot never did. They carry forward with no migration. - The start-time cache is correct because a process's own start time never changes. A failed read is not cached, so a transient `ps` failure does not pin the fallback value. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6cfb69b4b0
commit
fa86162790
4 files changed
+93
-14
No files matched your search
@@ -2,7 +2,13 @@ import { afterEach, expect, it, vi } from "vitest";
|
||||
import * as childProcess from "node:child_process";
|
||||
import * as hotRestart from "../hot-restart.js";
|
||||
import { getServerInfoSnapshot, resetServerInfoCacheForTests } from "../../server-info.js";
|
||||
import { currentNativeControllerIdentity } from "./native-restart-recovery.js";
|
||||
import { currentNativeControllerIdentity, isCurrentNativeControllerProcess } from "./native-restart-recovery.js";
|
||||
import {
|
||||
currentControllerBootId,
|
||||
currentControllerProcess,
|
||||
resetCurrentProcessStartedAtForTests,
|
||||
} from "../controller-process-identity.js";
|
||||
import { legacyControllerBootId } from "../legacy-controller-lease.js";
|
||||
|
||||
vi.mock("node:child_process", async importOriginal => ({
|
||||
...await importOriginal<typeof import("node:child_process")>(),
|
||||
@@ -14,6 +20,7 @@ vi.mock("node:child_process", async importOriginal => ({
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
resetServerInfoCacheForTests();
|
||||
resetCurrentProcessStartedAtForTests();
|
||||
});
|
||||
|
||||
it.each([true, false])("keeps controller identity independent of Git commands (OS start available=%s)", async available => {
|
||||
@@ -35,3 +42,39 @@ it.each([true, false])("keeps controller identity independent of Git commands (O
|
||||
expect(first.processStartedAt.toISOString()).toBe(available ? observed : fallback);
|
||||
expect(subprocess).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("stamps one boot id for native claims, legacy leases and creation intents", async () => {
|
||||
vi.spyOn(hotRestart, "readProcessStartedAt").mockResolvedValue("2026-11-02T12:00:00.000Z");
|
||||
const native = await currentNativeControllerIdentity();
|
||||
const controller = await currentControllerProcess();
|
||||
|
||||
expect(native.bootId).toBe(currentControllerBootId());
|
||||
expect(legacyControllerBootId).toBe(currentControllerBootId());
|
||||
expect(controller.bootId).toBe(currentControllerBootId());
|
||||
expect(controller.processStartedAt).toBe(native.processStartedAt.toISOString());
|
||||
expect(isCurrentNativeControllerProcess({ controllerBootId: legacyControllerBootId, controllerPid: process.pid })).toBe(true);
|
||||
});
|
||||
|
||||
it("reads this process's start time from the OS once across claims and renewals", async () => {
|
||||
const read = vi.spyOn(hotRestart, "readProcessStartedAt").mockResolvedValue("2026-11-02T12:00:00.000Z");
|
||||
|
||||
await Promise.all(Array.from({ length: 5 }, () => currentNativeControllerIdentity()));
|
||||
for (let i = 0; i < 5; i++) {
|
||||
await currentNativeControllerIdentity();
|
||||
await currentControllerProcess();
|
||||
}
|
||||
|
||||
expect(read).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("retries a failed start-time read instead of caching the failure", async () => {
|
||||
const read = vi.spyOn(hotRestart, "readProcessStartedAt")
|
||||
.mockRejectedValueOnce(new Error("ps unavailable"))
|
||||
.mockResolvedValue("2026-11-02T12:00:00.000Z");
|
||||
|
||||
expect((await currentControllerProcess()).processStartedAt).toBeNull();
|
||||
expect((await currentControllerProcess()).processStartedAt).toBe("2026-11-02T12:00:00.000Z");
|
||||
await currentNativeControllerIdentity();
|
||||
|
||||
expect(read).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
@@ -35,7 +35,7 @@ import {
|
||||
} from "@paperclipai/db";
|
||||
import { REMOTE_CONTROLLER_LEASE_MS } from "./native-runner-recovery-contract.js";
|
||||
import { readProcessStartedAt } from "../hot-restart.js";
|
||||
import { processStartsDefinitelyDiffer } from "../controller-process-identity.js";
|
||||
import { currentControllerBootId, currentProcessStartedAtMs, processStartsDefinitelyDiffer } from "../controller-process-identity.js";
|
||||
import { getServerProcessStartedAt } from "../../server-info.js";
|
||||
import { redactSensitiveText } from "../../redaction.js";
|
||||
import { isUniqueViolation } from "../../db-errors.js";
|
||||
@@ -282,7 +282,6 @@ export async function restartResumeKeepsAttempt(tx: Pick<Db, "select">, input: {
|
||||
return work !== undefined;
|
||||
}
|
||||
|
||||
const controllerBootId = randomUUID();
|
||||
|
||||
function processIsAlive(pid: number | null | undefined): boolean {
|
||||
if (!pid || !Number.isInteger(pid) || pid <= 0) return false;
|
||||
@@ -324,14 +323,17 @@ async function observedProcessStart(pid: number): Promise<Date | null> {
|
||||
}
|
||||
}
|
||||
|
||||
/** The native controller stamp of this process. The boot id is the single
|
||||
* process boot id (`controller-process-identity.ts`) and the start time is read
|
||||
* from the OS once, so claims and renewals on the turn path spawn no `ps`. */
|
||||
export async function currentNativeControllerIdentity(): Promise<NativeControllerIdentity> {
|
||||
const observed = await observedProcessStart(process.pid);
|
||||
const observed = await currentProcessStartedAtMs();
|
||||
const fallback = new Date(getServerProcessStartedAt());
|
||||
return {
|
||||
bootId: controllerBootId,
|
||||
bootId: currentControllerBootId(),
|
||||
pid: process.pid,
|
||||
processStartedAt:
|
||||
observed ?? (Number.isNaN(fallback.getTime()) ? new Date() : fallback),
|
||||
observed !== null ? new Date(observed) : (Number.isNaN(fallback.getTime()) ? new Date() : fallback),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -339,7 +341,7 @@ export function isCurrentNativeControllerProcess(owner: {
|
||||
controllerBootId: string | null;
|
||||
controllerPid: number | null;
|
||||
}): boolean {
|
||||
return owner.controllerBootId === controllerBootId && owner.controllerPid === process.pid;
|
||||
return owner.controllerBootId === currentControllerBootId() && owner.controllerPid === process.pid;
|
||||
}
|
||||
|
||||
function sameProcessStart(left: Date | null, right: Date | null): boolean {
|
||||
|
||||
Reference in new issue
Block a user