mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:48:15 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner end-to-end harness stops the Paperclip server and its launcher through a process tree owner that reads `ps` > - The owner already waits out a briefly uncertain process group, but one failed `ps` listing ended the stop at once > - The stop then fell back to the direct child and always reported `Owned tree cleanup incomplete`, even when the launcher exited normally > - CI showed this as `Owned tree cleanup incomplete: Error: Could not inspect owned process identities` in `server-stop.test.ts` on 5 unrelated branches > - This pull request re-reads a failed listing a bounded number of times inside the same deadline, and names why `ps` failed > - The benefit is that one transient listing failure no longer fails a server stop, and a persistent failure is diagnosable ## Linked Issues or Issue Description **What happened?** `tests/runner-e2e/server-stop.test.ts > lets launcher cancellation join wrapper shutdown without signaling the server twice` (job "Runner support suite", project `runner-e2e-support`) fails intermittently with `Error: Owned tree cleanup incomplete: Error: Could not inspect owned process identities`, thrown from `tests/runner-e2e/process-tree-owner.ts:265`. It failed 6 times on 5 unrelated branches in 48 hours. **Steps to reproduce** 1. The failing inspection is `readProcessTable()` returning `null` (non-zero `ps` exit, `ps` timeout, or spawn error). The CI logs do not say which, because `ps` stderr was discarded. 2. Locally on macOS, 40 plain runs pass (the Linux `ps` failure does not occur on this host). 3. Inject the fault: make `readProcessTable()` return `null` for 15% of calls. 26 of 30 runs then fail with the exact CI error. **Paperclip version or commit** `master` at the base of this branch (CI on unrelated pull requests, 2026-11-10 to 2026-11-12). **Expected behavior** A failed listing grants no authority to signal anything. The stop reads the table again within its deadline, as it already does for an uncertain group, and it still falls back to the direct child when the listing keeps failing. **Actual behavior** `settleUncertainIdentity` only retried the uncertain-identity error. The first failed listing went straight to `incompleteTreeFallback`, which always throws. ## What Changed - `process-tree-owner.ts`: `settleUncertainIdentity` also re-reads after `Could not inspect owned process identities`, up to 3 attempts per step. It does not start a new read when less than one poll interval (100 ms) is left before the step deadline. No signal is sent between attempts. A persistent failure keeps the old fallback, so the direct child still gets its grace period. - `process-tree.ts`: `readProcessTable` takes an optional `onFailure(reason)` callback and keeps `ps` stderr. The owner adds the reason (exit code, signal, timeout, or stderr tail) to the inspection error. - `process-tree-owner.test.ts`: four regression tests. A listing that fails once during a stop no longer fails it. A listing that always fails still falls back to the direct child after 3 reads. No read starts at the deadline. A failed listing reports its reason. ## Verification - Red/green: both new tests fail on master. The first fails with the exact CI message `Owned tree cleanup incomplete: Error: Could not inspect owned process identities`. Both pass with this change. - Loop of the flaky test, 30 runs each, with a temporary local fault injection (`readProcessTable` returns `null` for 15% of calls, not committed): - Before: 4 pass, 26 fail (24 with `Owned tree cleanup incomplete: Error: Could not inspect owned process identities`). - After: 27 pass, 3 fail. In a 20-run recheck, the only failures (2) came from the test's own preliminary `await owner.observe()` at line 166, outside the stop path. The 15% injected rate is far above the real rate. - Without injection: 40 of 40 pass on master locally. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/process-tree-owner.test.ts tests/runner-e2e/server-stop.test.ts`: 41 tests pass. - `tsc --noEmit -p tests/runner-e2e/tsconfig.json`: no errors in the changed files. ## Risks - Low risk. This is test harness code. Each step can now wait up to 2 extra 100 ms polls when `ps` fails. No new read starts within 100 ms of the deadline. As before this change, one `ps` read that is already running can still take up to its 5 s timeout. The exact cause of the Linux `ps` failure is not proven yet. The new error text will name it if it happens again. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
248 lines
7.9 KiB
TypeScript
248 lines
7.9 KiB
TypeScript
import { spawn, type ChildProcess } from "node:child_process";
|
|
import path from "node:path";
|
|
|
|
export interface ProcessObservation {
|
|
pid: number;
|
|
parentPid: number;
|
|
processGroupId: number;
|
|
started: string;
|
|
kind: string;
|
|
state?: string;
|
|
}
|
|
|
|
export interface ObservedProcessGroup {
|
|
processGroupId: number;
|
|
depth: number;
|
|
members: Array<Pick<ProcessObservation, "pid" | "started">>;
|
|
}
|
|
|
|
export interface ObservedProcessTreeMember {
|
|
process: ProcessObservation;
|
|
depth: number;
|
|
}
|
|
|
|
export function observeDescendantProcessTree(
|
|
table: readonly ProcessObservation[],
|
|
rootPid: number,
|
|
) {
|
|
const byPid = new Map(table.map((candidate) => [candidate.pid, candidate]));
|
|
const byParent = new Map<number, ProcessObservation[]>();
|
|
for (const candidate of table) {
|
|
const children = byParent.get(candidate.parentPid) ?? [];
|
|
children.push(candidate);
|
|
byParent.set(candidate.parentPid, children);
|
|
}
|
|
const observed = new Map<number, ObservedProcessTreeMember>();
|
|
const pending = [{ pid: rootPid, depth: 0 }];
|
|
while (pending.length > 0) {
|
|
const next = pending.shift()!;
|
|
if (observed.has(next.pid)) continue;
|
|
const candidate = byPid.get(next.pid);
|
|
if (!candidate) continue;
|
|
observed.set(next.pid, { process: candidate, depth: next.depth });
|
|
for (const child of byParent.get(next.pid) ?? []) {
|
|
pending.push({ pid: child.pid, depth: next.depth + 1 });
|
|
}
|
|
}
|
|
const groupsById = new Map<number, ObservedProcessGroup>();
|
|
for (const { process: candidate, depth } of observed.values()) {
|
|
const group = groupsById.get(candidate.processGroupId) ?? {
|
|
processGroupId: candidate.processGroupId,
|
|
depth,
|
|
members: [],
|
|
};
|
|
group.depth = Math.max(group.depth, depth);
|
|
group.members.push({ pid: candidate.pid, started: candidate.started });
|
|
groupsById.set(candidate.processGroupId, group);
|
|
}
|
|
return {
|
|
members: [...observed.values()].sort(
|
|
(left, right) => left.depth - right.depth,
|
|
),
|
|
groups: [...groupsById.values()].sort(
|
|
(left, right) => right.depth - left.depth,
|
|
),
|
|
};
|
|
}
|
|
|
|
export function safeProcessGroupTerminationOrder(input: {
|
|
rootProcessGroupId: number;
|
|
currentProcessGroupId: number | null;
|
|
groups: readonly ObservedProcessGroup[];
|
|
}) {
|
|
if (input.currentProcessGroupId === null) return [];
|
|
const safe = new Map<number, number>();
|
|
for (const group of input.groups) {
|
|
if (
|
|
group.processGroupId <= 1 ||
|
|
group.processGroupId === input.currentProcessGroupId
|
|
) {
|
|
continue;
|
|
}
|
|
safe.set(
|
|
group.processGroupId,
|
|
Math.max(safe.get(group.processGroupId) ?? -1, group.depth),
|
|
);
|
|
}
|
|
if (
|
|
input.rootProcessGroupId > 1 &&
|
|
input.rootProcessGroupId !== input.currentProcessGroupId &&
|
|
input.groups.some(
|
|
(group) => group.processGroupId === input.rootProcessGroupId,
|
|
)
|
|
) {
|
|
safe.set(input.rootProcessGroupId, -1);
|
|
}
|
|
return [...safe.entries()]
|
|
.sort((left, right) => right[1] - left[1])
|
|
.map(([processGroupId]) => processGroupId);
|
|
}
|
|
|
|
export function revalidateObservedProcessGroups(
|
|
groups: readonly ObservedProcessGroup[],
|
|
table: readonly ProcessObservation[],
|
|
) {
|
|
const byPid = new Map(table.map((candidate) => [candidate.pid, candidate]));
|
|
return groups.filter((group) =>
|
|
group.members.some((member) => {
|
|
const candidate = byPid.get(member.pid);
|
|
return (
|
|
candidate?.processGroupId === group.processGroupId &&
|
|
candidate.started === member.started
|
|
);
|
|
}),
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Refresh groups only after their original members have been revalidated.
|
|
* A process may replace itself or fork a final cleanup helper after SIGTERM.
|
|
* Retain that continuously live group until one poll observes it as empty.
|
|
*/
|
|
export function refreshContinuouslyLiveProcessGroups(
|
|
groups: readonly ObservedProcessGroup[],
|
|
table: readonly ProcessObservation[],
|
|
) {
|
|
const membersByGroup = new Map<number, ProcessObservation[]>();
|
|
for (const candidate of table) {
|
|
const members = membersByGroup.get(candidate.processGroupId) ?? [];
|
|
members.push(candidate);
|
|
membersByGroup.set(candidate.processGroupId, members);
|
|
}
|
|
return groups.flatMap((group) => {
|
|
const members = membersByGroup.get(group.processGroupId);
|
|
return members
|
|
? [
|
|
{
|
|
...group,
|
|
members: members.map((candidate) => ({
|
|
pid: candidate.pid,
|
|
started: candidate.started,
|
|
})),
|
|
},
|
|
]
|
|
: [];
|
|
});
|
|
}
|
|
|
|
const diagnosticProcessKinds = new Set([
|
|
"bash",
|
|
"chrome",
|
|
"codex",
|
|
"google-chrome",
|
|
"node",
|
|
"paperclip-runnerd",
|
|
"playwright",
|
|
"pnpm",
|
|
"postgres",
|
|
"sh",
|
|
"tsx",
|
|
]);
|
|
|
|
export function diagnosticProcessKind(command: string, platform: NodeJS.Platform = process.platform) {
|
|
const name = path.basename(command);
|
|
// Linux comm is limited to 15 bytes. This is a diagnostic role only;
|
|
// preservation still requires the trusted run PID/start/group and ancestry.
|
|
if (platform === "linux" && name === "paperclip-runne") return "paperclip-runnerd";
|
|
return diagnosticProcessKinds.has(name) ? name : "other";
|
|
}
|
|
|
|
export async function readProcessTable(
|
|
startInspector?: () => ChildProcess,
|
|
onFailure?: (reason: string) => void,
|
|
): Promise<ProcessObservation[] | null> {
|
|
if (process.platform === "win32") {
|
|
return null;
|
|
}
|
|
return await new Promise<ProcessObservation[] | null>((resolve) => {
|
|
const inspector = startInspector ? startInspector() : spawn(
|
|
"ps",
|
|
["-e", "-o", "pid=,ppid=,pgid=,lstart=,stat=,comm="],
|
|
{
|
|
env: { PATH: "/usr/bin:/bin", LANG: "C", LC_ALL: "C" },
|
|
stdio: ["ignore", "pipe", "pipe"],
|
|
// A wrapper-group shutdown must not interrupt its own bounded audit.
|
|
detached: true,
|
|
},
|
|
);
|
|
let output = "";
|
|
let settled = false;
|
|
let truncated = false;
|
|
let errorOutput = "";
|
|
const finish = (value: ProcessObservation[] | null, reason?: string) => {
|
|
if (settled) return;
|
|
settled = true;
|
|
clearTimeout(inspectionTimeout);
|
|
if (value === null && reason) onFailure?.(reason);
|
|
resolve(value);
|
|
};
|
|
const inspectionTimeout = setTimeout(() => {
|
|
inspector.kill("SIGKILL");
|
|
finish(null, "ps did not finish within 5000ms");
|
|
}, 5_000);
|
|
inspectionTimeout.unref();
|
|
inspector.stdout?.setEncoding("utf8").on("data", (chunk: string) => {
|
|
if (output.length + chunk.length > 1024 * 1024) {
|
|
truncated = true;
|
|
inspector.kill("SIGKILL");
|
|
finish(null, "ps output exceeded 1 MiB");
|
|
return;
|
|
}
|
|
output += chunk;
|
|
});
|
|
inspector.stderr?.setEncoding("utf8").on("data", (chunk: string) => {
|
|
errorOutput = (errorOutput + chunk).slice(-512);
|
|
});
|
|
inspector.once("error", (error) => finish(null, `ps could not start: ${error.message}`));
|
|
inspector.once("close", (code, signal) => {
|
|
if (code !== 0 || truncated) {
|
|
finish(null, `ps exited code=${code} signal=${signal}${errorOutput.trim() ? `: ${errorOutput.trim()}` : ""}`);
|
|
return;
|
|
}
|
|
const observations = output
|
|
.split(/\r?\n/)
|
|
.map((line) =>
|
|
/^\s*(\d+)\s+(\d+)\s+(\d+)\s+(\S+\s+\S+\s+\d+\s+\d{2}:\d{2}:\d{2}\s+\d{4})\s+(\S+)\s+(.+?)\s*$/.exec(
|
|
line,
|
|
),
|
|
)
|
|
.filter((match): match is RegExpExecArray => match !== null)
|
|
.map((match): ProcessObservation => {
|
|
// A target process can choose its own argv and process name. Emit a
|
|
// fixed category instead of target-controlled text so diagnostics
|
|
// can never turn that metadata into a secret-exfiltration channel.
|
|
const kind = diagnosticProcessKind(match[6]!);
|
|
return {
|
|
pid: Number(match[1]),
|
|
parentPid: Number(match[2]),
|
|
processGroupId: Number(match[3]),
|
|
started: match[4]!,
|
|
kind,
|
|
state: match[5]!,
|
|
};
|
|
});
|
|
finish(observations);
|
|
});
|
|
}).catch(() => null);
|
|
}
|