Files
PaperClipAI/tests/runner-e2e/process-tree.ts
T
DottaandClaude Opus 5.5 49550814fd fix(runner-e2e): re-read a process listing that fails once during an owned stop (#21152)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner end-to-end harness stops the Paperclip server and its
launcher through a process tree owner that reads `ps`
> - The owner already waits out a briefly uncertain process group, but
one failed `ps` listing ended the stop at once
> - The stop then fell back to the direct child and always reported
`Owned tree cleanup incomplete`, even when the launcher exited normally
> - CI showed this as `Owned tree cleanup incomplete: Error: Could not
inspect owned process identities` in `server-stop.test.ts` on 5
unrelated branches
> - This pull request re-reads a failed listing a bounded number of
times inside the same deadline, and names why `ps` failed
> - The benefit is that one transient listing failure no longer fails a
server stop, and a persistent failure is diagnosable

## Linked Issues or Issue Description

**What happened?**
`tests/runner-e2e/server-stop.test.ts > lets launcher cancellation join
wrapper shutdown without signaling the server twice` (job "Runner
support suite", project `runner-e2e-support`) fails intermittently with
`Error: Owned tree cleanup incomplete: Error: Could not inspect owned
process identities`, thrown from
`tests/runner-e2e/process-tree-owner.ts:265`. It failed 6 times on 5
unrelated branches in 48 hours.

**Steps to reproduce**
1. The failing inspection is `readProcessTable()` returning `null`
(non-zero `ps` exit, `ps` timeout, or spawn error). The CI logs do not
say which, because `ps` stderr was discarded.
2. Locally on macOS, 40 plain runs pass (the Linux `ps` failure does not
occur on this host).
3. Inject the fault: make `readProcessTable()` return `null` for 15% of
calls. 26 of 30 runs then fail with the exact CI error.

**Paperclip version or commit**
`master` at the base of this branch (CI on unrelated pull requests,
2026-11-10 to 2026-11-12).

**Expected behavior**
A failed listing grants no authority to signal anything. The stop reads
the table again within its deadline, as it already does for an uncertain
group, and it still falls back to the direct child when the listing
keeps failing.

**Actual behavior**
`settleUncertainIdentity` only retried the uncertain-identity error. The
first failed listing went straight to `incompleteTreeFallback`, which
always throws.

## What Changed

- `process-tree-owner.ts`: `settleUncertainIdentity` also re-reads after
`Could not inspect owned process identities`, up to 3 attempts per step.
It does not start a new read when less than one poll interval (100 ms)
is left before the step deadline. No signal is sent between attempts. A
persistent failure keeps the old fallback, so the direct child still
gets its grace period.
- `process-tree.ts`: `readProcessTable` takes an optional
`onFailure(reason)` callback and keeps `ps` stderr. The owner adds the
reason (exit code, signal, timeout, or stderr tail) to the inspection
error.
- `process-tree-owner.test.ts`: four regression tests. A listing that
fails once during a stop no longer fails it. A listing that always fails
still falls back to the direct child after 3 reads. No read starts at
the deadline. A failed listing reports its reason.

## Verification

- Red/green: both new tests fail on master. The first fails with the
exact CI message `Owned tree cleanup incomplete: Error: Could not
inspect owned process identities`. Both pass with this change.
- Loop of the flaky test, 30 runs each, with a temporary local fault
injection (`readProcessTable` returns `null` for 15% of calls, not
committed):
- Before: 4 pass, 26 fail (24 with `Owned tree cleanup incomplete:
Error: Could not inspect owned process identities`).
- After: 27 pass, 3 fail. In a 20-run recheck, the only failures (2)
came from the test's own preliminary `await owner.observe()` at line
166, outside the stop path. The 15% injected rate is far above the real
rate.
- Without injection: 40 of 40 pass on master locally.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/process-tree-owner.test.ts
tests/runner-e2e/server-stop.test.ts`: 41 tests pass.
- `tsc --noEmit -p tests/runner-e2e/tsconfig.json`: no errors in the
changed files.

## Risks

- Low risk. This is test harness code. Each step can now wait up to 2
extra 100 ms polls when `ps` fails. No new read starts within 100 ms of
the deadline. As before this change, one `ps` read that is already
running can still take up to its 5 s timeout. The exact cause of the
Linux `ps` failure is not proven yet. The new error text will name it if
it happens again.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:29:40 -06:00

248 lines
7.9 KiB
TypeScript

import { spawn, type ChildProcess } from "node:child_process";
import path from "node:path";
export interface ProcessObservation {
pid: number;
parentPid: number;
processGroupId: number;
started: string;
kind: string;
state?: string;
}
export interface ObservedProcessGroup {
processGroupId: number;
depth: number;
members: Array<Pick<ProcessObservation, "pid" | "started">>;
}
export interface ObservedProcessTreeMember {
process: ProcessObservation;
depth: number;
}
export function observeDescendantProcessTree(
table: readonly ProcessObservation[],
rootPid: number,
) {
const byPid = new Map(table.map((candidate) => [candidate.pid, candidate]));
const byParent = new Map<number, ProcessObservation[]>();
for (const candidate of table) {
const children = byParent.get(candidate.parentPid) ?? [];
children.push(candidate);
byParent.set(candidate.parentPid, children);
}
const observed = new Map<number, ObservedProcessTreeMember>();
const pending = [{ pid: rootPid, depth: 0 }];
while (pending.length > 0) {
const next = pending.shift()!;
if (observed.has(next.pid)) continue;
const candidate = byPid.get(next.pid);
if (!candidate) continue;
observed.set(next.pid, { process: candidate, depth: next.depth });
for (const child of byParent.get(next.pid) ?? []) {
pending.push({ pid: child.pid, depth: next.depth + 1 });
}
}
const groupsById = new Map<number, ObservedProcessGroup>();
for (const { process: candidate, depth } of observed.values()) {
const group = groupsById.get(candidate.processGroupId) ?? {
processGroupId: candidate.processGroupId,
depth,
members: [],
};
group.depth = Math.max(group.depth, depth);
group.members.push({ pid: candidate.pid, started: candidate.started });
groupsById.set(candidate.processGroupId, group);
}
return {
members: [...observed.values()].sort(
(left, right) => left.depth - right.depth,
),
groups: [...groupsById.values()].sort(
(left, right) => right.depth - left.depth,
),
};
}
export function safeProcessGroupTerminationOrder(input: {
rootProcessGroupId: number;
currentProcessGroupId: number | null;
groups: readonly ObservedProcessGroup[];
}) {
if (input.currentProcessGroupId === null) return [];
const safe = new Map<number, number>();
for (const group of input.groups) {
if (
group.processGroupId <= 1 ||
group.processGroupId === input.currentProcessGroupId
) {
continue;
}
safe.set(
group.processGroupId,
Math.max(safe.get(group.processGroupId) ?? -1, group.depth),
);
}
if (
input.rootProcessGroupId > 1 &&
input.rootProcessGroupId !== input.currentProcessGroupId &&
input.groups.some(
(group) => group.processGroupId === input.rootProcessGroupId,
)
) {
safe.set(input.rootProcessGroupId, -1);
}
return [...safe.entries()]
.sort((left, right) => right[1] - left[1])
.map(([processGroupId]) => processGroupId);
}
export function revalidateObservedProcessGroups(
groups: readonly ObservedProcessGroup[],
table: readonly ProcessObservation[],
) {
const byPid = new Map(table.map((candidate) => [candidate.pid, candidate]));
return groups.filter((group) =>
group.members.some((member) => {
const candidate = byPid.get(member.pid);
return (
candidate?.processGroupId === group.processGroupId &&
candidate.started === member.started
);
}),
);
}
/**
* Refresh groups only after their original members have been revalidated.
* A process may replace itself or fork a final cleanup helper after SIGTERM.
* Retain that continuously live group until one poll observes it as empty.
*/
export function refreshContinuouslyLiveProcessGroups(
groups: readonly ObservedProcessGroup[],
table: readonly ProcessObservation[],
) {
const membersByGroup = new Map<number, ProcessObservation[]>();
for (const candidate of table) {
const members = membersByGroup.get(candidate.processGroupId) ?? [];
members.push(candidate);
membersByGroup.set(candidate.processGroupId, members);
}
return groups.flatMap((group) => {
const members = membersByGroup.get(group.processGroupId);
return members
? [
{
...group,
members: members.map((candidate) => ({
pid: candidate.pid,
started: candidate.started,
})),
},
]
: [];
});
}
const diagnosticProcessKinds = new Set([
"bash",
"chrome",
"codex",
"google-chrome",
"node",
"paperclip-runnerd",
"playwright",
"pnpm",
"postgres",
"sh",
"tsx",
]);
export function diagnosticProcessKind(command: string, platform: NodeJS.Platform = process.platform) {
const name = path.basename(command);
// Linux comm is limited to 15 bytes. This is a diagnostic role only;
// preservation still requires the trusted run PID/start/group and ancestry.
if (platform === "linux" && name === "paperclip-runne") return "paperclip-runnerd";
return diagnosticProcessKinds.has(name) ? name : "other";
}
export async function readProcessTable(
startInspector?: () => ChildProcess,
onFailure?: (reason: string) => void,
): Promise<ProcessObservation[] | null> {
if (process.platform === "win32") {
return null;
}
return await new Promise<ProcessObservation[] | null>((resolve) => {
const inspector = startInspector ? startInspector() : spawn(
"ps",
["-e", "-o", "pid=,ppid=,pgid=,lstart=,stat=,comm="],
{
env: { PATH: "/usr/bin:/bin", LANG: "C", LC_ALL: "C" },
stdio: ["ignore", "pipe", "pipe"],
// A wrapper-group shutdown must not interrupt its own bounded audit.
detached: true,
},
);
let output = "";
let settled = false;
let truncated = false;
let errorOutput = "";
const finish = (value: ProcessObservation[] | null, reason?: string) => {
if (settled) return;
settled = true;
clearTimeout(inspectionTimeout);
if (value === null && reason) onFailure?.(reason);
resolve(value);
};
const inspectionTimeout = setTimeout(() => {
inspector.kill("SIGKILL");
finish(null, "ps did not finish within 5000ms");
}, 5_000);
inspectionTimeout.unref();
inspector.stdout?.setEncoding("utf8").on("data", (chunk: string) => {
if (output.length + chunk.length > 1024 * 1024) {
truncated = true;
inspector.kill("SIGKILL");
finish(null, "ps output exceeded 1 MiB");
return;
}
output += chunk;
});
inspector.stderr?.setEncoding("utf8").on("data", (chunk: string) => {
errorOutput = (errorOutput + chunk).slice(-512);
});
inspector.once("error", (error) => finish(null, `ps could not start: ${error.message}`));
inspector.once("close", (code, signal) => {
if (code !== 0 || truncated) {
finish(null, `ps exited code=${code} signal=${signal}${errorOutput.trim() ? `: ${errorOutput.trim()}` : ""}`);
return;
}
const observations = output
.split(/\r?\n/)
.map((line) =>
/^\s*(\d+)\s+(\d+)\s+(\d+)\s+(\S+\s+\S+\s+\d+\s+\d{2}:\d{2}:\d{2}\s+\d{4})\s+(\S+)\s+(.+?)\s*$/.exec(
line,
),
)
.filter((match): match is RegExpExecArray => match !== null)
.map((match): ProcessObservation => {
// A target process can choose its own argv and process name. Emit a
// fixed category instead of target-controlled text so diagnostics
// can never turn that metadata into a secret-exfiltration channel.
const kind = diagnosticProcessKind(match[6]!);
return {
pid: Number(match[1]),
parentPid: Number(match[2]),
processGroupId: Number(match[3]),
started: match[4]!,
kind,
state: match[5]!,
};
});
finish(observations);
});
}).catch(() => null);
}