From 49550814fdd0c8491d275a8580a555b70fe56dc3 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 12 Nov 2026 12:29:40 -0600 Subject: [PATCH] fix(runner-e2e): re-read a process listing that fails once during an owned stop (#21152) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner end-to-end harness stops the Paperclip server and its launcher through a process tree owner that reads `ps` > - The owner already waits out a briefly uncertain process group, but one failed `ps` listing ended the stop at once > - The stop then fell back to the direct child and always reported `Owned tree cleanup incomplete`, even when the launcher exited normally > - CI showed this as `Owned tree cleanup incomplete: Error: Could not inspect owned process identities` in `server-stop.test.ts` on 5 unrelated branches > - This pull request re-reads a failed listing a bounded number of times inside the same deadline, and names why `ps` failed > - The benefit is that one transient listing failure no longer fails a server stop, and a persistent failure is diagnosable ## Linked Issues or Issue Description **What happened?** `tests/runner-e2e/server-stop.test.ts > lets launcher cancellation join wrapper shutdown without signaling the server twice` (job "Runner support suite", project `runner-e2e-support`) fails intermittently with `Error: Owned tree cleanup incomplete: Error: Could not inspect owned process identities`, thrown from `tests/runner-e2e/process-tree-owner.ts:265`. It failed 6 times on 5 unrelated branches in 48 hours. **Steps to reproduce** 1. The failing inspection is `readProcessTable()` returning `null` (non-zero `ps` exit, `ps` timeout, or spawn error). The CI logs do not say which, because `ps` stderr was discarded. 2. Locally on macOS, 40 plain runs pass (the Linux `ps` failure does not occur on this host). 3. Inject the fault: make `readProcessTable()` return `null` for 15% of calls. 26 of 30 runs then fail with the exact CI error. **Paperclip version or commit** `master` at the base of this branch (CI on unrelated pull requests, 2026-11-10 to 2026-11-12). **Expected behavior** A failed listing grants no authority to signal anything. The stop reads the table again within its deadline, as it already does for an uncertain group, and it still falls back to the direct child when the listing keeps failing. **Actual behavior** `settleUncertainIdentity` only retried the uncertain-identity error. The first failed listing went straight to `incompleteTreeFallback`, which always throws. ## What Changed - `process-tree-owner.ts`: `settleUncertainIdentity` also re-reads after `Could not inspect owned process identities`, up to 3 attempts per step. It does not start a new read when less than one poll interval (100 ms) is left before the step deadline. No signal is sent between attempts. A persistent failure keeps the old fallback, so the direct child still gets its grace period. - `process-tree.ts`: `readProcessTable` takes an optional `onFailure(reason)` callback and keeps `ps` stderr. The owner adds the reason (exit code, signal, timeout, or stderr tail) to the inspection error. - `process-tree-owner.test.ts`: four regression tests. A listing that fails once during a stop no longer fails it. A listing that always fails still falls back to the direct child after 3 reads. No read starts at the deadline. A failed listing reports its reason. ## Verification - Red/green: both new tests fail on master. The first fails with the exact CI message `Owned tree cleanup incomplete: Error: Could not inspect owned process identities`. Both pass with this change. - Loop of the flaky test, 30 runs each, with a temporary local fault injection (`readProcessTable` returns `null` for 15% of calls, not committed): - Before: 4 pass, 26 fail (24 with `Owned tree cleanup incomplete: Error: Could not inspect owned process identities`). - After: 27 pass, 3 fail. In a 20-run recheck, the only failures (2) came from the test's own preliminary `await owner.observe()` at line 166, outside the stop path. The 15% injected rate is far above the real rate. - Without injection: 40 of 40 pass on master locally. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/process-tree-owner.test.ts tests/runner-e2e/server-stop.test.ts`: 41 tests pass. - `tsc --noEmit -p tests/runner-e2e/tsconfig.json`: no errors in the changed files. ## Risks - Low risk. This is test harness code. Each step can now wait up to 2 extra 100 ms polls when `ps` fails. No new read starts within 100 ms of the deadline. As before this change, one `ps` read that is already running can still take up to its 5 s timeout. The exact cause of the Linux `ps` failure is not proven yet. The new error text will name it if it happens again. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- tests/runner-e2e/process-tree-owner.test.ts | 73 +++++++++++++++++++++ tests/runner-e2e/process-tree-owner.ts | 15 ++++- tests/runner-e2e/process-tree.ts | 27 +++++--- 3 files changed, 104 insertions(+), 11 deletions(-) diff --git a/tests/runner-e2e/process-tree-owner.test.ts b/tests/runner-e2e/process-tree-owner.test.ts index 862ff8ba998..924a3592a8a 100644 --- a/tests/runner-e2e/process-tree-owner.test.ts +++ b/tests/runner-e2e/process-tree-owner.test.ts @@ -280,3 +280,76 @@ it.skipIf(process.platform === "win32")("keeps an uncertainty that starts at SIG expect(signals.filter(([, signal]) => signal === "SIGKILL")).toEqual([]); } finally { owner.stopObserving(); } }); + +it.skipIf(process.platform === "win32")("re-reads a process listing that fails once while the launcher stops", async () => { + const self = row(process.pid, 1, 10); + let table: ProcessObservation[] = [self, row(100, process.pid, 100)]; + let failedReads = 0; + let stopping = false; + const signals: Array<[number, NodeJS.Signals]> = []; + const child = { pid: 100, exitCode: null as number | null, signalCode: null as NodeJS.Signals | null, + kill: () => { throw new Error("must not fall back to the direct child"); } }; + const owner = createProcessTreeOwner(child as unknown as ChildProcess, { + readTable: async () => { + if (stopping && failedReads === 0) { failedReads += 1; return null; } + return table; + }, + signalGroup: (pid, signal) => { + signals.push([pid, signal]); + child.exitCode = 0; table = [self]; + }, + }); + try { + await owner.observe(); + stopping = true; + await stopOwnedProcessTree(child as unknown as ChildProcess, owner, 2_000, 500); + expect(failedReads).toBe(1); + expect(signals).toEqual([[100, "SIGTERM"]]); + } finally { owner.stopObserving(); } +}); + +it.skipIf(process.platform === "win32")("reports why the process listing failed", async () => { + const reasons: string[] = []; + const inspector = spawn(process.execPath, ["-e", "process.stderr.write('listing unavailable'); process.exit(3)"], { + stdio: ["ignore", "pipe", "pipe"], + }); + expect(await readProcessTable(() => inspector, reason => reasons.push(reason))).toBeNull(); + expect(reasons).toEqual(["ps exited code=3 signal=null: listing unavailable"]); +}); + +it.skipIf(process.platform === "win32")("still falls back to the direct child when every process listing fails", async () => { + let reads = 0; + const signals: NodeJS.Signals[] = []; + const child = { pid: 100, exitCode: null as number | null, signalCode: null as NodeJS.Signals | null, + kill: (signal: NodeJS.Signals) => { signals.push(signal); child.signalCode = signal; return true; } }; + const owner = createProcessTreeOwner(child as unknown as ChildProcess, { + readTable: async () => { reads += 1; return null; }, + signalGroup: () => { throw new Error("must not signal an unobserved group"); }, + }); + try { + await expect(owner.observe()).rejects.toThrow("Could not inspect owned process identities"); + reads = 0; + const started = Date.now(); + await expect(stopOwnedProcessTree(child as unknown as ChildProcess, owner, 2_000, 500)) + .rejects.toThrow("Owned tree cleanup incomplete: Error: Could not inspect owned process identities"); + expect(reads).toBeGreaterThanOrEqual(3); + expect(signals).toEqual(["SIGTERM"]); + expect(Date.now() - started).toBeLessThan(1_000); + } finally { owner.stopObserving(); } +}); + +it.skipIf(process.platform === "win32")("does not re-read a failed process listing past the stop deadline", async () => { + let reads = 0; + const child = { pid: 100, exitCode: null as number | null, signalCode: null as NodeJS.Signals | null, + kill: (signal: NodeJS.Signals) => { child.signalCode = signal; return true; } }; + const owner = createProcessTreeOwner(child as unknown as ChildProcess, { + readTable: async () => { reads += 1; return null; }, + signalGroup: () => { throw new Error("must not signal an unobserved group"); }, + }); + try { + await expect(owner.observe()).rejects.toThrow("Could not inspect owned process identities"); + reads = 0; + await expect(stopOwnedProcessTree(child as unknown as ChildProcess, owner, 50, 500)).rejects.toThrow("Owned tree cleanup incomplete"); + expect(reads).toBe(1); + } finally { owner.stopObserving(); } +}); diff --git a/tests/runner-e2e/process-tree-owner.ts b/tests/runner-e2e/process-tree-owner.ts index 8fcbed6aae5..6687b65ffc4 100644 --- a/tests/runner-e2e/process-tree-owner.ts +++ b/tests/runner-e2e/process-tree-owner.ts @@ -29,6 +29,8 @@ export function restartProcessStartMatches(observed: string, expected: string, p } export const UNCERTAIN_GROUP_IDENTITY = "Owned process group identity became uncertain"; +export const INSPECTION_UNAVAILABLE = "Could not inspect owned process identities"; +const INSPECTION_ATTEMPTS = 3; /** * Under load or during a graceful drain a whole group can turn over between @@ -39,10 +41,15 @@ export const UNCERTAIN_GROUP_IDENTITY = "Owned process group identity became unc * meanwhile; a straggler that outlives the deadline keeps the original error. */ export async function settleUncertainIdentity(step: () => Promise, until: () => number, pollMs = 100): Promise { + let inspectionFailures = 0; for (;;) { try { return await step(); } catch (error) { - if (!String(error).includes(UNCERTAIN_GROUP_IDENTITY) || Date.now() >= until()) throw error; + const text = String(error); + const transient = text.includes(UNCERTAIN_GROUP_IDENTITY) + || (text.includes(INSPECTION_UNAVAILABLE) && ++inspectionFailures < INSPECTION_ATTEMPTS); + const remaining = until() - Date.now(); + if (!transient || remaining <= pollMs) throw error; await new Promise(resolve => setTimeout(resolve, pollMs)); } } @@ -59,7 +66,8 @@ export function createProcessTreeOwner(root: ChildProcess, options: { let restartRunner: ProcessObservation | undefined; let observing: Promise | undefined; const covered = new Set(); - const readTable = options.readTable ?? readProcessTable; + let inspectionFailure = ""; + const readTable = options.readTable ?? (() => readProcessTable(undefined, reason => { inspectionFailure = reason; })); const signalGroup = options.signalGroup ?? ((pid, signal) => process.kill(-pid, signal)); const exited = () => root.exitCode !== null || root.signalCode !== null || !root.pid; const running = (candidate: ProcessObservation) => !candidate.state?.startsWith("Z"); @@ -68,8 +76,9 @@ export function createProcessTreeOwner(root: ChildProcess, options: { if (process.platform === "win32") return Promise.resolve(); if (observing) return observing; observing = (async () => { + inspectionFailure = ""; const next = await readTable(); - if (!next) throw new Error("Could not inspect owned process identities"); + if (!next) throw new Error(inspectionFailure ? `${INSPECTION_UNAVAILABLE} (${inspectionFailure})` : INSPECTION_UNAVAILABLE); const callerGroup = next.find(row => row.pid === process.pid)?.processGroupId; if (callerGroup === undefined) throw new Error("Cleanup caller group identity is unavailable"); const rootRow = next.find(row => row.pid === root.pid); diff --git a/tests/runner-e2e/process-tree.ts b/tests/runner-e2e/process-tree.ts index 7a21642d9cd..80e627f4d3c 100644 --- a/tests/runner-e2e/process-tree.ts +++ b/tests/runner-e2e/process-tree.ts @@ -167,7 +167,10 @@ export function diagnosticProcessKind(command: string, platform: NodeJS.Platform return diagnosticProcessKinds.has(name) ? name : "other"; } -export async function readProcessTable(startInspector?: () => ChildProcess): Promise { +export async function readProcessTable( + startInspector?: () => ChildProcess, + onFailure?: (reason: string) => void, +): Promise { if (process.platform === "win32") { return null; } @@ -177,7 +180,7 @@ export async function readProcessTable(startInspector?: () => ChildProcess): Pro ["-e", "-o", "pid=,ppid=,pgid=,lstart=,stat=,comm="], { env: { PATH: "/usr/bin:/bin", LANG: "C", LC_ALL: "C" }, - stdio: ["ignore", "pipe", "ignore"], + stdio: ["ignore", "pipe", "pipe"], // A wrapper-group shutdown must not interrupt its own bounded audit. detached: true, }, @@ -185,29 +188,37 @@ export async function readProcessTable(startInspector?: () => ChildProcess): Pro let output = ""; let settled = false; let truncated = false; - const finish = (value: ProcessObservation[] | null) => { + let errorOutput = ""; + const finish = (value: ProcessObservation[] | null, reason?: string) => { if (settled) return; settled = true; clearTimeout(inspectionTimeout); + if (value === null && reason) onFailure?.(reason); resolve(value); }; const inspectionTimeout = setTimeout(() => { inspector.kill("SIGKILL"); - finish(null); + finish(null, "ps did not finish within 5000ms"); }, 5_000); inspectionTimeout.unref(); inspector.stdout?.setEncoding("utf8").on("data", (chunk: string) => { if (output.length + chunk.length > 1024 * 1024) { truncated = true; inspector.kill("SIGKILL"); - finish(null); + finish(null, "ps output exceeded 1 MiB"); return; } output += chunk; }); - inspector.once("error", () => finish(null)); - inspector.once("close", (code) => { - if (code !== 0 || truncated) { finish(null); return; } + inspector.stderr?.setEncoding("utf8").on("data", (chunk: string) => { + errorOutput = (errorOutput + chunk).slice(-512); + }); + inspector.once("error", (error) => finish(null, `ps could not start: ${error.message}`)); + inspector.once("close", (code, signal) => { + if (code !== 0 || truncated) { + finish(null, `ps exited code=${code} signal=${signal}${errorOutput.trim() ? `: ${errorOutput.trim()}` : ""}`); + return; + } const observations = output .split(/\r?\n/) .map((line) =>