From 9fdcefcc3415eaae327cf78192fb088bcf014c80 Mon Sep 17 00:00:00 2001 From: Dotta Date: Tue, 6 Oct 2026 13:52:35 -0500 Subject: [PATCH] fix: record Linux process birth from kernel ticks Co-Authored-By: Paperclip --- .../2026-10-02-pi-production-readiness.md | 50 ++++++++++++++++--- .../paperclip-runner/docs/durable-recovery.md | 8 +++ packages/paperclip-runner/src/index.ts | 1 + packages/paperclip-runner/src/live/index.ts | 1 + .../src/live/linux-process-start.test.ts | 41 +++++++++++++++ .../src/live/linux-process-start.ts | 40 +++++++++++++++ .../src/live/runnerd-codex-transport.ts | 3 +- server/src/services/hot-restart.test.ts | 16 +++--- server/src/services/hot-restart.ts | 8 ++- ...erver-restart-preservation-process.test.ts | 3 +- 10 files changed, 149 insertions(+), 22 deletions(-) create mode 100644 packages/paperclip-runner/src/live/linux-process-start.test.ts create mode 100644 packages/paperclip-runner/src/live/linux-process-start.ts diff --git a/doc/plans/2026-10-02-pi-production-readiness.md b/doc/plans/2026-10-02-pi-production-readiness.md index f1ae84589d4..511f65e34ba 100644 --- a/doc/plans/2026-10-02-pi-production-readiness.md +++ b/doc/plans/2026-10-02-pi-production-readiness.md @@ -5,7 +5,36 @@ set of release gates. Pi remains a candidate until every required gate passes. Do not expand this work to additional models, widgets, images, Cursor or Copilot qualification. The existing draft stack must be reviewed in dependency order. -## Frozen corrected-set measurements — 2026-10-06 +## Linux process-birth correction — 2026-10-06 + +The local pending-question restart fails before controller shutdown because the +unchanged ownership guard rejects the public run's daemon birth receipt. Its +canonical failed grade and passed cleanup remain preserved. Actual SDK inspection +proves the launcher and temporary fixture root absent before releasing the claim. + +A zero-provider Linux control reproduces a concrete mismatch: production records +`/proc/PID` directory ctime, while the strict restart guard observes actual birth +through `ps`. The installed daemon's ctime is `18:46:34.102Z`, but its kernel +birth is `18:46:33.800Z`. The same unchanged guard rejects the former and accepts +the latter. It also rejects a deliberately wrong birth receipt. The daemon's +identity stays stable through delayed inspection and its process is retired. +The proof is hash-bound at +`32580fe4f97a9799049ae887ff5652f4b5243cc53c3c2e77404bb13e0c40f32e`. + +The correction shares a kernel-tick birth reader between server process receipts +and retained Runner maintenance. Malformed metadata fails closed. PID, group, +ancestry, role and cleanup checks stay intact. Parser, hot-restart, native-recovery +and process-owner checks pass 94 tests; one Linux-only parser integration is +skipped on Mac and the separate installed Linux daemon proof passes. Runner, +server and Product E2E typechecks pass. No paid attempt follows this correction +yet. It changes shipping inputs: rebuild all public packages and the image, +then freeze a new set starting at Product **0/26** and Runner **0/7**. The prior +set's 1/26 and 6/7 passes remain historical. Its canonical full Linux tests are +still confirmed live on their original handle; do not stop or duplicate them. +The missing-LF memory failure and finish-task stream timeout still need a proven +correction. Production remains held. + +## Previous frozen-set measurements — 2026-10-06 The rebuilt `7f66a30ad` package/image set has Product **1/26** and Runner **6/7** verified passes. The Daytona controller-restart case passes all six @@ -584,13 +613,13 @@ local tests and release review are still required. | Gate | Acceptance evidence | Current result | | --- | --- | --- | -| Runtime identity and admission | Exact runtime/profile/model; verified effective thinking; startup below the 60-second admission limit; unsupported configuration fails before a prompt | Frozen `7f66a30ad` packages/image pass normal profile-15 offline admission on ARM Mac, Intel x64 under Rosetta and Linux. Live model/low and all 33 cases still need proof on this set. Historical admission retains its original source. | -| Live lifecycle | Pending native question survives controller restart in the original process; one creation and resolution; stale answer after provider death rejected; Stop retires owned processes; three warm turns keep process identity | The corrected sandbox GitHub path passes both actual Linux observer controls and all 35 affected Linux tests. The old-set restart remains failed with incomplete terminal evidence and failed canonical cleanup, despite independent resource absence. A new exact live restart measurement is required. | -| Product workflows | All 26 explicit Pi Product E2E cells (13 local, 13 Daytona); screenshots, public state, independent artifacts, terminal and cleanup evidence | Current set: 1/26. The corrected Daytona controller-restart case passes exact same-request/answer, terminal, retirement and cleanup grades. The new local memory measurement fails at 120 seconds; independent output shows 32 bytes without LF. Preserve all failures, hold unchanged retries, and complete every cell on this freeze. | -| Runner protocol | Pi roster passes through the native packaged runner and authenticated mock control plane; lifecycle/denial/control cases remain distinct from Product tests | Current set: 6/7. Task context, context-before-action, document creation, human confirmation, context/document/progress and governed-wait workflows pass on the new frozen artifacts. Private definitions `dc97fdcb`, profile 15, model and low thinking stay pinned. Finish-task stays held pending a concrete correction to its historical terminal timeout. | -| Installed distribution | Public CLI/server tars on ARM Mac, Intel Mac and Linux; normal Pi setup; exact Linux companion and immutable Daytona image imported without binary override | All 18 public tars are repacked at `7f66a30ad`. Image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:8e744211ded7c19015aececeb87da4e1ee994ca9d85c6fbbebcd916682425231` passes anonymous import and normal Linux installation. Normal ARM Mac and x64 under Rosetta install/setup/admission pass. Physical Intel hardware remains unverified. The new Linux controller passes imported companion, browser, complete consumer/plugin graph audits, offline native controls and collection of all 26 cases. | -| Governance and spend | Company isolation, human-only permission, duplicate/stale answers, Stop and budget hard stop; pricing estimates never become claimed bills | Historical invariant checks keep their source labels. Fresh live shipping qualification remains required. The dedicated key has a $5 lifetime cap, zero BYOK usage and a $100 campaign ceiling. Last provisional read-only accounting observes $0.34323782 used and $4.65676218 remaining. Delayed settlement is possible. The key cap does not prove Paperclip budget enforcement. | -| Integration and rollout | Review each prerequisite; final-head typecheck, tests, build and CI pass; no unresolved review; exact artifacts; rollback recorded | At `7f66a30ad`, all 1,847 Product fixture tests, harness typecheck, full build and recursive typecheck pass. CI has 53 successes and two skips; Greptile is 5/5 with zero open #14956 threads. Documentation head `99e823164` also has 53 successful checks, two skips and zero open #14956 threads. The old full test command is terminal with an npm-PATH failure; its exact packaging test passes after the owned tooling correction. Current-source typecheck and build pass; full tests remain on their recorded live handle. Current-source full workspace command evidence remains required. The valid #14921 premature-qualification finding stays open until all live qualification passes. Later docs heads need current CI/review. No merge or release is authorized. | +| Runtime identity and admission | Exact runtime/profile/model; verified effective thinking; startup below the 60-second admission limit; unsupported configuration fails before a prompt | The Linux process-birth correction changes shipping inputs. Fresh rebuilt packages/image and normal profile-15 admission are required. Prior `7f66a30ad` ARM, Rosetta and Linux proofs remain historical. | +| Live lifecycle | Pending native question survives controller restart in the original process; one creation and resolution; stale answer after provider death rejected; Stop retires owned processes; three warm turns keep process identity | The prior-set Daytona restart passes. The local restart fails daemon-birth admission before shutdown. A zero-provider installed Linux daemon control proves the kernel-birth correction against the unchanged ownership guard, including wrong-birth rejection and actual retirement. Fresh live lifecycle measurements remain required. | +| Product workflows | All 26 explicit Pi Product E2E cells (13 local, 13 Daytona); screenshots, public state, independent artifacts, terminal and cleanup evidence | Rebuild pending; new-set count 0/26. Prior `7f66a30ad` has 1/26. Its local memory and local restart failures remain unchanged. Memory still reaches 120 seconds after an independently observed 32-byte file without LF. No unchanged paid retries. | +| Runner protocol | Pi roster passes through the native packaged runner and authenticated mock control plane; lifecycle/denial/control cases remain distinct from Product tests | Rebuild pending; new-set count 0/7. Prior `7f66a30ad` has 6/7. Private definitions `dc97fdcb`, profile 15, exact model and low thinking stay pinned. Finish-task remains held pending a proven correction to the historical terminal timeout. | +| Installed distribution | Public CLI/server tars on ARM Mac, Intel Mac and Linux; normal Pi setup; exact Linux companion and immutable Daytona image imported without binary override | Rebuild all 18 public packages and the immutable Linux image for the Linux birth-reader change. Repeat normal installation and admission on the new set. Prior ARM Mac, Rosetta and Linux receipts remain historical. Physical Intel hardware remains unverified. | +| Governance and spend | Company isolation, human-only permission, duplicate/stale answers, Stop and budget hard stop; pricing estimates never become claimed bills | Company isolation, human authority, duplicate/stale answers, Stop and Paperclip budget hard stop still need fresh shipping qualification. Keep the dedicated $5 lifetime key and $100 campaign caps with no reset or BYOK. Last provisional accounting is $0.357317585 used; delayed settlement is possible. | +| Integration and rollout | Review each prerequisite; final-head typecheck, tests, build and CI pass; no unresolved review; exact artifacts; rollback recorded | The birth-reader correction passes 94 targeted tests, with one Linux-only skip on Mac, plus Runner, server and E2E typechecks and the installed Linux daemon proof. Full build, recursive typecheck, full tests and latest-head CI remain required. The prior-source full Linux command remains confirmed live with passed build/typecheck. Prerequisite #14921 still has a valid premature-admission finding; draft-stack review and all qualification must finish before release. No merge or release is authorized. | ## Bounded execution @@ -1595,6 +1624,8 @@ reviews pass. This goal does not publish, merge or deploy the candidate. current company configuration and the exact retained package set. Run `paperclipai db:backup --json` against the intended instance. Retain the reported backup path and size. Record the backup file SHA256 before updating. + Drain active Linux native runs before crossing between ctime-based and + kernel-birth receipts. Never rewrite an active identity to force adoption. 2. Use the exact qualified published version. For a managed npm installation, preview and apply the pinned update below. The operator must supply `PI_RELEASE_VERSION` after publication. Keep the default pre-update backup. @@ -1638,6 +1669,9 @@ stop new Pi dispatch and retire active work through the control-plane Stop path. For a managed installation, verify that the preview names the recorded prior payload before applying rollback: +Complete the owned Stop path before rolling Linux runs back across the process +birth format change. Preserve their terminal and cleanup evidence. + ```sh paperclipai update --rollback --dry-run --json paperclipai update --rollback diff --git a/packages/paperclip-runner/docs/durable-recovery.md b/packages/paperclip-runner/docs/durable-recovery.md index 8ee0b22c975..6986e6a00c5 100644 --- a/packages/paperclip-runner/docs/durable-recovery.md +++ b/packages/paperclip-runner/docs/durable-recovery.md @@ -305,6 +305,14 @@ the lease TTL to match its required revocation bound. ## Restart and reconciliation +Local Linux process receipts use the kernel's `/proc/PID/stat` start ticks, +`/proc/stat` boot time and `CLK_TCK` to record process birth. The server and +retained Runner maintenance use the same reader. `/proc/PID` directory ctime +can differ from birth and must not grant process ownership. Invalid or missing +birth metadata fails closed. PID, process group and owner checks remain required. +Drain active Linux runs before updating or rolling back across the previous +ctime-based receipt format; do not rewrite a live receipt to force adoption. + A socket drop keeps the same Rust process and in-memory lease. The process reconnects and reloads the mock-core command and event cursor. diff --git a/packages/paperclip-runner/src/index.ts b/packages/paperclip-runner/src/index.ts index 2494e177e9c..b199a2b7109 100644 --- a/packages/paperclip-runner/src/index.ts +++ b/packages/paperclip-runner/src/index.ts @@ -66,6 +66,7 @@ export { type RunnerdCodexTransportOptions, } from "./live/runnerd-codex-transport.js"; export * from "./live/workspace-file-reference.js"; +export { readLinuxProcessStartedAt, type LinuxProcessStartOptions } from "./live/linux-process-start.js"; export * from "./protocol/replay-contract.js"; export * from "./protocol/replay-loader.js"; export * from "./protocol/result-normalization.js"; diff --git a/packages/paperclip-runner/src/live/index.ts b/packages/paperclip-runner/src/live/index.ts index d6abeefc91d..be23acc0c42 100644 --- a/packages/paperclip-runner/src/live/index.ts +++ b/packages/paperclip-runner/src/live/index.ts @@ -4,5 +4,6 @@ export * from "./live-session.js"; export * from "./durable-live-session-store.js"; export * from "./runnerd-codex-transport.js"; export * from "./turn-stream.js"; +export * from "./linux-process-start.js"; export { probeAcpxClaudeInstallation, probeAcpxGrokInstallation, probeAcpxPiInstallation } from "../drivers/acpx/installation-integrity.js"; diff --git a/packages/paperclip-runner/src/live/linux-process-start.test.ts b/packages/paperclip-runner/src/live/linux-process-start.test.ts new file mode 100644 index 00000000000..4cc8a684444 --- /dev/null +++ b/packages/paperclip-runner/src/live/linux-process-start.test.ts @@ -0,0 +1,41 @@ +import { spawn } from "node:child_process"; +import { expect, it } from "vitest"; +import { readLinuxProcessStartedAt } from "./linux-process-start.js"; + +const stat = (ticks: string, command = "runner") => `123 (${command}) S ${Array(18).fill("0").join(" ")} ${ticks}\n`; +const options = (processStat = stat("123"), systemStat = "btime 1785546000\n", clockTicksPerSecond = 100) => ({ + clockTicksPerSecond, + readFile: (path: string) => path === "/proc/123/stat" ? processStat : systemStat, +}); + +it("uses kernel birth ticks even when the command contains spaces and parentheses", () => { + expect(readLinuxProcessStartedAt(123, options(stat("123", "a ) b ( c")))) + .toBe(new Date(1785546001230).toISOString()); + expect(readLinuxProcessStartedAt(123, options(stat("124")))) + .not.toBe(readLinuxProcessStartedAt(123, options())); +}); + +it.each([ + [stat("bad"), "btime 1785546000\n", 100], + [stat("123").replace("123 (", "124 ("), "btime 1785546000\n", 100], + ["123 (runner) S", "btime 1785546000\n", 100], + [stat("123"), "btime invalid\n", 100], + [stat("123"), "btime 1785546000\n", 0], + [stat("123"), "btime 1785546000\n", Number.NaN], + [stat("123"), "btime 999999999999999999999\n", 100], +])("fails closed on malformed birth metadata or clock frequency", (processStat, systemStat, frequency) => { + expect(() => readLinuxProcessStartedAt(123, options(processStat, systemStat, frequency))).toThrow(); +}); + +it.skipIf(process.platform !== "linux")("keeps the actual owned child's birth identity stable through a delayed observation", async () => { + const child = spawn("/bin/sleep", ["10"], { stdio: "ignore" }); + const closed = new Promise(resolve => child.once("close", () => resolve())); + try { + expect(child.pid).toBeGreaterThan(0); + const first = readLinuxProcessStartedAt(child.pid!); + await new Promise(resolve => setTimeout(resolve, 1100)); + expect(readLinuxProcessStartedAt(child.pid!)).toBe(first); + child.kill("SIGTERM"); await closed; + expect(() => readLinuxProcessStartedAt(child.pid!)).toThrow(); + } finally { child.kill("SIGTERM"); await closed; } +}, 5000); diff --git a/packages/paperclip-runner/src/live/linux-process-start.ts b/packages/paperclip-runner/src/live/linux-process-start.ts new file mode 100644 index 00000000000..66abe1322dc --- /dev/null +++ b/packages/paperclip-runner/src/live/linux-process-start.ts @@ -0,0 +1,40 @@ +import { execFileSync } from "node:child_process"; +import { readFileSync } from "node:fs"; + +let clockTicksPerSecond: number | undefined; + +export interface LinuxProcessStartOptions { + readFile?: (path: string) => string; + clockTicksPerSecond?: number; +} + +/** /proc/PID directory ctime is lookup metadata, not the process's birth time. */ +export function readLinuxProcessStartedAt(pid: number, options: LinuxProcessStartOptions = {}): string { + if (!Number.isSafeInteger(pid) || pid <= 0) throw new Error("Invalid Linux process PID"); + const readFile = options.readFile ?? ((path: string) => readFileSync(path, "utf8")); + const processStat = readFile(`/proc/${pid}/stat`); + // comm can contain spaces and parentheses. Fields after its final ')' start + // at field 3 (state), so field 22 (starttime) is index 19 in this suffix. + const end = processStat.lastIndexOf(") "); + const fields = end >= 0 ? processStat.slice(end + 2).trim().split(/\s+/) : []; + const startTicks = fields[19]; + const bootSeconds = /^btime (\d+)$/m.exec(readFile("/proc/stat"))?.[1]; + if (!processStat.startsWith(`${pid} (`) || !/^\d+$/.test(startTicks ?? "") || !bootSeconds) { + throw new Error("Invalid Linux process birth metadata"); + } + let ticks = options.clockTicksPerSecond; + if (ticks === undefined) { + clockTicksPerSecond ??= Number(execFileSync("getconf", ["CLK_TCK"], { + encoding: "utf8", timeout: 1_500, windowsHide: true, + env: { PATH: "/usr/bin:/bin", LANG: "C", LC_ALL: "C" }, + }).trim()); + ticks = clockTicksPerSecond; + } + if (!Number.isSafeInteger(ticks) || ticks <= 0) throw new Error("Invalid Linux clock tick frequency"); + const milliseconds = BigInt(bootSeconds) * 1_000n + BigInt(startTicks!) * 1_000n / BigInt(ticks); + const value = Number(milliseconds); + if (!Number.isSafeInteger(value) || !Number.isFinite(new Date(value).getTime())) { + throw new Error("Invalid Linux process birth timestamp"); + } + return new Date(value).toISOString(); +} diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index 4385a42b818..ebac6ff38ae 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -5,6 +5,7 @@ import { waitForWarmAttachmentReadiness } from "./warm-attachment-readiness.js"; import { codexExecutableReadOnlyRoots } from "../drivers/codex/codex-security-config.js"; import { isCanonicalProviderEventType } from "../provider-events.js"; import { execFileSync } from "node:child_process"; +import { readLinuxProcessStartedAt } from "./linux-process-start.js"; import { createHash, randomUUID } from "node:crypto"; import { appendFileSync, @@ -96,7 +97,7 @@ function readLocalProcessStartedAt(pid: number): string | null { if (!Number.isInteger(pid) || pid <= 0) return null; try { if (process.platform === "linux") { - return new Date(statSync(`/proc/${pid}`).ctimeMs).toISOString(); + return readLinuxProcessStartedAt(pid); } if ( ["darwin", "freebsd", "openbsd", "aix", "sunos"].includes( diff --git a/server/src/services/hot-restart.test.ts b/server/src/services/hot-restart.test.ts index 7b53fc84055..41c6a375fb2 100644 --- a/server/src/services/hot-restart.test.ts +++ b/server/src/services/hot-restart.test.ts @@ -89,18 +89,20 @@ describe("hot-restart path compatibility", () => { ).not.toBeNull(); }); - it("reads Linux process start time from proc metadata", async () => { + it("reads Linux process birth from kernel ticks rather than proc directory metadata", async () => { await expect( readProcessStartedAt(123, { platform: "linux", - stat: async (target) => { - expect(target).toBe("/proc/123"); - return { - ctimeMs: Date.parse("2026-08-01T01:00:00.123Z"), - }; + linuxProcessStart: { + clockTicksPerSecond: 100, + readFile: (target) => target === "/proc/stat" + ? `btime ${Date.parse("2026-08-01T01:00:00.000Z") / 1000}\n` + : target === "/proc/123/stat" + ? `123 (runner) S ${Array(18).fill("0").join(" ")} 123\n` + : (() => { throw new Error("Unexpected proc read"); })(), }, }), - ).resolves.toBe("2026-08-01T01:00:00.123Z"); + ).resolves.toBe("2026-08-01T01:00:01.230Z"); }); it("reads macOS process start time through ps", async () => { diff --git a/server/src/services/hot-restart.ts b/server/src/services/hot-restart.ts index 30bf71e40f0..a0406c553ee 100644 --- a/server/src/services/hot-restart.ts +++ b/server/src/services/hot-restart.ts @@ -1,6 +1,7 @@ import { execFile } from "node:child_process"; import fs from "node:fs/promises"; import path from "node:path"; +import { readLinuxProcessStartedAt, type LinuxProcessStartOptions } from "@paperclipai/paperclip-runner"; import { resolvePaperclipHomeDir, resolvePaperclipInstanceId, @@ -14,7 +15,6 @@ const HOT_RESTART_LOCK_STALE_MS = 30_000; const HOT_RESTART_LOCK_TIMEOUT_MS = 10_000; type ProcessCommandRunner = (command: string, args: string[]) => Promise; -type ProcessStatReader = (target: string) => Promise<{ ctimeMs: number }>; export type HotRestartIntentRun = { runId: string; @@ -178,17 +178,15 @@ export async function readProcessStartedAt( pid: number, options: { platform?: NodeJS.Platform; - stat?: ProcessStatReader; + linuxProcessStart?: LinuxProcessStartOptions; runCommand?: ProcessCommandRunner; } = {}, ) { const platform = options.platform ?? process.platform; - const stat = options.stat ?? fs.stat; const runCommand = options.runCommand ?? runProcessCommand; if (platform === "linux") { - const processStat = await stat(`/proc/${pid}`); - return new Date(processStat.ctimeMs).toISOString(); + return readLinuxProcessStartedAt(pid, options.linuxProcessStart); } if (["darwin", "freebsd", "openbsd", "aix", "sunos"].includes(platform)) { diff --git a/tests/runner-e2e/server-restart-preservation-process.test.ts b/tests/runner-e2e/server-restart-preservation-process.test.ts index 10b1e720635..6315f690549 100644 --- a/tests/runner-e2e/server-restart-preservation-process.test.ts +++ b/tests/runner-e2e/server-restart-preservation-process.test.ts @@ -7,6 +7,7 @@ import { expect, it } from "vitest"; import { createRunnerE2EServerStopper } from "./server-stop.js"; import { createProcessTreeOwner } from "./process-tree-owner.js"; import { readProcessTable } from "./process-tree.js"; +import { readLinuxProcessStartedAt } from "../../packages/paperclip-runner/src/live/linux-process-start.js"; it.skipIf(process.platform === "win32")("keeps the real admitted daemon and child alive across controller exit, then drains both controller generations", async () => { const directory = await mkdtemp(path.join(await realpath(os.tmpdir()), "pc-restart-owner-")); @@ -57,7 +58,7 @@ it.skipIf(process.platform === "win32")("keeps the real admitted daemon and chil const descendants = before!.filter(row => row.parentPid === runner.pid); expect(descendants.length).toBeGreaterThan(0); await stop.forRestart(first.child, { processPid: runner.pid, processGroupId: runner.processGroupId, - processStartedAt: new Date(runner.started).toISOString() }); + processStartedAt: process.platform === "linux" ? readLinuxProcessStartedAt(runner.pid) : new Date(runner.started).toISOString() }); const after = await readProcessTable(); expect(after!.find(row => row.pid === runner.pid)?.started).toBe(runner.started); for (const descendant of descendants) expect(after!.find(row => row.pid === descendant.pid)?.started).toBe(descendant.started);