fix(runner-e2e): re-read a process listing that fails once during an owned stop (#21152)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner end-to-end harness stops the Paperclip server and its
launcher through a process tree owner that reads `ps`
> - The owner already waits out a briefly uncertain process group, but
one failed `ps` listing ended the stop at once
> - The stop then fell back to the direct child and always reported
`Owned tree cleanup incomplete`, even when the launcher exited normally
> - CI showed this as `Owned tree cleanup incomplete: Error: Could not
inspect owned process identities` in `server-stop.test.ts` on 5
unrelated branches
> - This pull request re-reads a failed listing a bounded number of
times inside the same deadline, and names why `ps` failed
> - The benefit is that one transient listing failure no longer fails a
server stop, and a persistent failure is diagnosable

## Linked Issues or Issue Description

**What happened?**
`tests/runner-e2e/server-stop.test.ts > lets launcher cancellation join
wrapper shutdown without signaling the server twice` (job "Runner
support suite", project `runner-e2e-support`) fails intermittently with
`Error: Owned tree cleanup incomplete: Error: Could not inspect owned
process identities`, thrown from
`tests/runner-e2e/process-tree-owner.ts:265`. It failed 6 times on 5
unrelated branches in 48 hours.

**Steps to reproduce**
1. The failing inspection is `readProcessTable()` returning `null`
(non-zero `ps` exit, `ps` timeout, or spawn error). The CI logs do not
say which, because `ps` stderr was discarded.
2. Locally on macOS, 40 plain runs pass (the Linux `ps` failure does not
occur on this host).
3. Inject the fault: make `readProcessTable()` return `null` for 15% of
calls. 26 of 30 runs then fail with the exact CI error.

**Paperclip version or commit**
`master` at the base of this branch (CI on unrelated pull requests,
2026-11-10 to 2026-11-12).

**Expected behavior**
A failed listing grants no authority to signal anything. The stop reads
the table again within its deadline, as it already does for an uncertain
group, and it still falls back to the direct child when the listing
keeps failing.

**Actual behavior**
`settleUncertainIdentity` only retried the uncertain-identity error. The
first failed listing went straight to `incompleteTreeFallback`, which
always throws.

## What Changed

- `process-tree-owner.ts`: `settleUncertainIdentity` also re-reads after
`Could not inspect owned process identities`, up to 3 attempts per step.
It does not start a new read when less than one poll interval (100 ms)
is left before the step deadline. No signal is sent between attempts. A
persistent failure keeps the old fallback, so the direct child still
gets its grace period.
- `process-tree.ts`: `readProcessTable` takes an optional
`onFailure(reason)` callback and keeps `ps` stderr. The owner adds the
reason (exit code, signal, timeout, or stderr tail) to the inspection
error.
- `process-tree-owner.test.ts`: four regression tests. A listing that
fails once during a stop no longer fails it. A listing that always fails
still falls back to the direct child after 3 reads. No read starts at
the deadline. A failed listing reports its reason.

## Verification

- Red/green: both new tests fail on master. The first fails with the
exact CI message `Owned tree cleanup incomplete: Error: Could not
inspect owned process identities`. Both pass with this change.
- Loop of the flaky test, 30 runs each, with a temporary local fault
injection (`readProcessTable` returns `null` for 15% of calls, not
committed):
- Before: 4 pass, 26 fail (24 with `Owned tree cleanup incomplete:
Error: Could not inspect owned process identities`).
- After: 27 pass, 3 fail. In a 20-run recheck, the only failures (2)
came from the test's own preliminary `await owner.observe()` at line
166, outside the stop path. The 15% injected rate is far above the real
rate.
- Without injection: 40 of 40 pass on master locally.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/process-tree-owner.test.ts
tests/runner-e2e/server-stop.test.ts`: 41 tests pass.
- `tsc --noEmit -p tests/runner-e2e/tsconfig.json`: no errors in the
changed files.

## Risks

- Low risk. This is test harness code. Each step can now wait up to 2
extra 100 ms polls when `ps` fails. No new read starts within 100 ms of
the deadline. As before this change, one `ps` read that is already
running can still take up to its 5 s timeout. The exact cause of the
Linux `ps` failure is not proven yet. The new error text will name it if
it happens again.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 12:29:40 -06:00
1 parent c5e4b46d3f
commit 49550814fd
3 files changed
+104 -11

No files matched your search

@@ -280,3 +280,76 @@ it.skipIf(process.platform === "win32")("keeps an uncertainty that starts at SIG
expect(signals.filter(([, signal]) => signal === "SIGKILL")).toEqual([]);
} finally { owner.stopObserving(); }
});
it.skipIf(process.platform === "win32")("re-reads a process listing that fails once while the launcher stops", async () => {
const self = row(process.pid, 1, 10);
let table: ProcessObservation[] = [self, row(100, process.pid, 100)];
let failedReads = 0;
let stopping = false;
const signals: Array<[number, NodeJS.Signals]> = [];
const child = { pid: 100, exitCode: null as number | null, signalCode: null as NodeJS.Signals | null,
kill: () => { throw new Error("must not fall back to the direct child"); } };
const owner = createProcessTreeOwner(child as unknown as ChildProcess, {
readTable: async () => {
if (stopping && failedReads === 0) { failedReads += 1; return null; }
return table;
},
signalGroup: (pid, signal) => {
signals.push([pid, signal]);
child.exitCode = 0; table = [self];
},
});
try {
await owner.observe();
stopping = true;
await stopOwnedProcessTree(child as unknown as ChildProcess, owner, 2_000, 500);
expect(failedReads).toBe(1);
expect(signals).toEqual([[100, "SIGTERM"]]);
} finally { owner.stopObserving(); }
});
it.skipIf(process.platform === "win32")("reports why the process listing failed", async () => {
const reasons: string[] = [];
const inspector = spawn(process.execPath, ["-e", "process.stderr.write('listing unavailable'); process.exit(3)"], {
stdio: ["ignore", "pipe", "pipe"],
});
expect(await readProcessTable(() => inspector, reason => reasons.push(reason))).toBeNull();
expect(reasons).toEqual(["ps exited code=3 signal=null: listing unavailable"]);
});
it.skipIf(process.platform === "win32")("still falls back to the direct child when every process listing fails", async () => {
let reads = 0;
const signals: NodeJS.Signals[] = [];
const child = { pid: 100, exitCode: null as number | null, signalCode: null as NodeJS.Signals | null,
kill: (signal: NodeJS.Signals) => { signals.push(signal); child.signalCode = signal; return true; } };
const owner = createProcessTreeOwner(child as unknown as ChildProcess, {
readTable: async () => { reads += 1; return null; },
signalGroup: () => { throw new Error("must not signal an unobserved group"); },
});
try {
await expect(owner.observe()).rejects.toThrow("Could not inspect owned process identities");
reads = 0;
const started = Date.now();
await expect(stopOwnedProcessTree(child as unknown as ChildProcess, owner, 2_000, 500))
.rejects.toThrow("Owned tree cleanup incomplete: Error: Could not inspect owned process identities");
expect(reads).toBeGreaterThanOrEqual(3);
expect(signals).toEqual(["SIGTERM"]);
expect(Date.now() - started).toBeLessThan(1_000);
} finally { owner.stopObserving(); }
});
it.skipIf(process.platform === "win32")("does not re-read a failed process listing past the stop deadline", async () => {
let reads = 0;
const child = { pid: 100, exitCode: null as number | null, signalCode: null as NodeJS.Signals | null,
kill: (signal: NodeJS.Signals) => { child.signalCode = signal; return true; } };
const owner = createProcessTreeOwner(child as unknown as ChildProcess, {
readTable: async () => { reads += 1; return null; },
signalGroup: () => { throw new Error("must not signal an unobserved group"); },
});
try {
await expect(owner.observe()).rejects.toThrow("Could not inspect owned process identities");
reads = 0;
await expect(stopOwnedProcessTree(child as unknown as ChildProcess, owner, 50, 500)).rejects.toThrow("Owned tree cleanup incomplete");
expect(reads).toBe(1);
} finally { owner.stopObserving(); }
});
+12 -3
View File
@@ -29,6 +29,8 @@ export function restartProcessStartMatches(observed: string, expected: string, p
}
export const UNCERTAIN_GROUP_IDENTITY = "Owned process group identity became uncertain";
export const INSPECTION_UNAVAILABLE = "Could not inspect owned process identities";
const INSPECTION_ATTEMPTS = 3;
/**
* Under load or during a graceful drain a whole group can turn over between
@@ -39,10 +41,15 @@ export const UNCERTAIN_GROUP_IDENTITY = "Owned process group identity became unc
* meanwhile; a straggler that outlives the deadline keeps the original error.
*/
export async function settleUncertainIdentity<T>(step: () => Promise<T>, until: () => number, pollMs = 100): Promise<T> {
let inspectionFailures = 0;
for (;;) {
try { return await step(); }
catch (error) {
if (!String(error).includes(UNCERTAIN_GROUP_IDENTITY) || Date.now() >= until()) throw error;
const text = String(error);
const transient = text.includes(UNCERTAIN_GROUP_IDENTITY)
|| (text.includes(INSPECTION_UNAVAILABLE) && ++inspectionFailures < INSPECTION_ATTEMPTS);
const remaining = until() - Date.now();
if (!transient || remaining <= pollMs) throw error;
await new Promise(resolve => setTimeout(resolve, pollMs));
}
}
@@ -59,7 +66,8 @@ export function createProcessTreeOwner(root: ChildProcess, options: {
let restartRunner: ProcessObservation | undefined;
let observing: Promise<void> | undefined;
const covered = new Set<number>();
const readTable = options.readTable ?? readProcessTable;
let inspectionFailure = "";
const readTable = options.readTable ?? (() => readProcessTable(undefined, reason => { inspectionFailure = reason; }));
const signalGroup = options.signalGroup ?? ((pid, signal) => process.kill(-pid, signal));
const exited = () => root.exitCode !== null || root.signalCode !== null || !root.pid;
const running = (candidate: ProcessObservation) => !candidate.state?.startsWith("Z");
@@ -68,8 +76,9 @@ export function createProcessTreeOwner(root: ChildProcess, options: {
if (process.platform === "win32") return Promise.resolve();
if (observing) return observing;
observing = (async () => {
inspectionFailure = "";
const next = await readTable();
if (!next) throw new Error("Could not inspect owned process identities");
if (!next) throw new Error(inspectionFailure ? `${INSPECTION_UNAVAILABLE} (${inspectionFailure})` : INSPECTION_UNAVAILABLE);
const callerGroup = next.find(row => row.pid === process.pid)?.processGroupId;
if (callerGroup === undefined) throw new Error("Cleanup caller group identity is unavailable");
const rootRow = next.find(row => row.pid === root.pid);
+19 -8
View File
@@ -167,7 +167,10 @@ export function diagnosticProcessKind(command: string, platform: NodeJS.Platform
return diagnosticProcessKinds.has(name) ? name : "other";
}
export async function readProcessTable(startInspector?: () => ChildProcess): Promise<ProcessObservation[] | null> {
export async function readProcessTable(
startInspector?: () => ChildProcess,
onFailure?: (reason: string) => void,
): Promise<ProcessObservation[] | null> {
if (process.platform === "win32") {
return null;
}
@@ -177,7 +180,7 @@ export async function readProcessTable(startInspector?: () => ChildProcess): Pro
["-e", "-o", "pid=,ppid=,pgid=,lstart=,stat=,comm="],
{
env: { PATH: "/usr/bin:/bin", LANG: "C", LC_ALL: "C" },
stdio: ["ignore", "pipe", "ignore"],
stdio: ["ignore", "pipe", "pipe"],
// A wrapper-group shutdown must not interrupt its own bounded audit.
detached: true,
},
@@ -185,29 +188,37 @@ export async function readProcessTable(startInspector?: () => ChildProcess): Pro
let output = "";
let settled = false;
let truncated = false;
const finish = (value: ProcessObservation[] | null) => {
let errorOutput = "";
const finish = (value: ProcessObservation[] | null, reason?: string) => {
if (settled) return;
settled = true;
clearTimeout(inspectionTimeout);
if (value === null && reason) onFailure?.(reason);
resolve(value);
};
const inspectionTimeout = setTimeout(() => {
inspector.kill("SIGKILL");
finish(null);
finish(null, "ps did not finish within 5000ms");
}, 5_000);
inspectionTimeout.unref();
inspector.stdout?.setEncoding("utf8").on("data", (chunk: string) => {
if (output.length + chunk.length > 1024 * 1024) {
truncated = true;
inspector.kill("SIGKILL");
finish(null);
finish(null, "ps output exceeded 1 MiB");
return;
}
output += chunk;
});
inspector.once("error", () => finish(null));
inspector.once("close", (code) => {
if (code !== 0 || truncated) { finish(null); return; }
inspector.stderr?.setEncoding("utf8").on("data", (chunk: string) => {
errorOutput = (errorOutput + chunk).slice(-512);
});
inspector.once("error", (error) => finish(null, `ps could not start: ${error.message}`));
inspector.once("close", (code, signal) => {
if (code !== 0 || truncated) {
finish(null, `ps exited code=${code} signal=${signal}${errorOutput.trim() ? `: ${errorOutput.trim()}` : ""}`);
return;
}
const observations = output
.split(/\r?\n/)
.map((line) =>