test(chaos): treat a crash-restart EPERM as a warning once the server pid is dead (#20232)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness crashes the local Paperclip server with
SIGKILL (K1, K13 storms) and grades how tasks survive
> - `crashForRestart` sends SIGKILL to every process group the server
owns
> - On macOS, `killpg` can report EPERM for a group whose members are
still exiting
> - When that happens the fault is recorded as failed, and the wave
exits 2, even though the server process itself is dead
> - This pull request tolerates EPERM in the crash path once the server
pid is confirmed dead, and logs it as a warning
> - The benefit is that a wave grades the product, not a cleanup race in
the harness

## Linked Issues or Issue Description

**What happened?**
A local Codex K13 storm at concurrency 6 (wave w-20261109T084023-s9403,
112 tasks) exited 2 because faults f04 and f09 failed with `kill EPERM`.
The server pids of both faults had died, and the tasks recovered
normally.

**Steps to reproduce**
1. Run a long K13 storm on macOS with high concurrency.
2. Occasionally the SIGKILL of the server's process groups races with
members that are exiting.

**Expected behavior**
If the server process is dead, the crash completes and the remaining
cleanup continues; the EPERM is recorded as a warning.

**Actual behavior**
The fault fails with `kill EPERM` and the wave reports an error.

## What Changed

- `crashForRestart` catches EPERM from the group SIGKILL. It then sends
SIGKILL to the server pid directly if needed and waits for it to exit
within the forced timeout.
- If the server exited, it re-observes the tree and sends SIGKILL to
each remaining group on its own, so one group's EPERM does not leave the
others to drain gracefully. It logs a warning naming the server pid and
any group that still refused SIGKILL, then continues with the normal
post-kill cleanup.
- If the server is still alive, or the error is not EPERM, the error is
rethrown as before.
- Four tests: EPERM with a dead server finishes with a warning; a
detached descendant is still killed without a graceful drain when the
first group kill reports EPERM; EPERM with a surviving server fails; a
non-EPERM error fails. The failure tests stop their owners in `finally`.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/server-stop.test.ts`: the dead-server test fails before
the change (`kill EPERM`), the descendant test fails against the first
revision (the descendant drains on SIGTERM), and all 22 pass after it.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: clean.

## Risks

- A group that is still running and returns EPERM is no longer reported
in the crash step itself. The cleanup that follows still checks every
observed group, and it fails if any group survives.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

tests/runner-e2e/server-stop.test.ts

@@ -430,3 +430,100 @@ it.skipIf(process.platform === "win32")("still fails safely when a group stays u
     expect(groupSignals).toEqual([]);
   } finally { owner.stopObserving(); if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL"); }
 });
+
+function eperm() {
+  return Object.assign(new Error("kill EPERM"), { code: "EPERM" });
+}
+function crashStopper(signalError: () => Error, refusedGroup: () => number | undefined = () => undefined) {
+  const logs: string[] = [];
+  let thrown = false;
+  const stop = createRunnerE2EServerStopper({
+    gracefulTimeoutMs: 2_000, forcedTimeoutMs: 1_000,
+    hasSpawnError: () => false, markExpectedStop: () => {}, log: line => logs.push(line),
+    createOwner: candidate => {
+      const owner = createProcessTreeOwner(candidate);
+      return { ...owner, signal: async (signal, selected) => {
+        if (signal === "SIGKILL" && !thrown) {
+          thrown = true;
+          await owner.observe();
+          const refused = refusedGroup();
+          if (refused !== undefined) {
+            await owner.signal("SIGKILL", new Set(owner.liveGroups().map(group => group.processGroupId).filter(pid => pid !== refused)));
+          }
+          throw signalError();
+        }
+        return await owner.signal(signal, selected);
+      } };
+    },
+  });
+  return { stop, logs };
+}
+
+it.skipIf(process.platform === "win32")("finishes a crash restart with a warning when the group kill reports EPERM but the server is dead", async () => {
+  const { entry } = await fixture();
+  const child = start(entry);
+  await once(child, "message");
+  const { stop, logs } = crashStopper(eperm);
+  await stop.watch(child);
+  try {
+    await stop.crashForRestart(child);
+    expect(child.signalCode).toBe("SIGKILL");
+    expect(logs.join("")).toMatch(new RegExp(`Warning: .*EPERM.*pid=${child.pid}`));
+  } finally { await stop.stopAll(); }
+});
+
+it.skipIf(process.platform === "win32")("fails a crash restart when the group kill reports EPERM and the server survives", async () => {
+  const { entry } = await fixture();
+  const child = start(entry);
+  await once(child, "message");
+  const { stop } = crashStopper(eperm);
+  await stop.watch(child);
+  const kill = child.kill.bind(child);
+  child.kill = () => true;
+  try {
+    await expect(stop.crashForRestart(child)).rejects.toThrow(/EPERM/);
+    expect([child.exitCode, child.signalCode]).toEqual([null, null]);
+  } finally {
+    child.kill = kill;
+    await stop.stopAll().catch(() => {});
+  }
+});
+
+it.skipIf(process.platform === "win32")("fails a crash restart on a group kill error other than EPERM", async () => {
+  const { entry } = await fixture();
+  const child = start(entry);
+  await once(child, "message");
+  const { stop } = crashStopper(() => Object.assign(new Error("kill EINVAL"), { code: "EINVAL" }));
+  await stop.watch(child);
+  try {
+    await expect(stop.crashForRestart(child)).rejects.toThrow(/EINVAL/);
+  } finally { await stop.stopAll().catch(() => {}); }
+});
+
+it.skipIf(process.platform === "win32")("still SIGKILLs a detached descendant without a graceful drain when its group refuses the bulk kill after the server's", async () => {
+  const { root } = await fixture();
+  const marker = path.join(root, "graceful-drain.txt"), descendantEntry = path.join(root, "descendant.cjs");
+  await writeFile(descendantEntry, `
+    process.on('SIGTERM', () => require('node:fs').writeFileSync(${JSON.stringify(marker)}, 'descendant drained'));
+    process.send('ready'); setInterval(() => {}, 1000);
+  `);
+  const { entry } = await fixture(`
+    const child = require('node:child_process').fork(${JSON.stringify(descendantEntry)}, [], { detached: true, stdio: ['ignore', 'ignore', 'ignore', 'ipc'] });
+    child.once('message', () => process.send({ ready: true, descendantPid: child.pid }));
+    setInterval(() => {}, 1000);
+  `);
+  const controller = start(entry), controllerReady = once(controller, "message");
+  let descendantGroup: number | undefined;
+  const { stop, logs } = crashStopper(eperm, () => descendantGroup);
+  await stop.watch(controller);
+  try {
+    const [ready] = await controllerReady;
+    descendantGroup = ready.descendantPid;
+    await stop.crashForRestart(controller);
+    expect(controller.signalCode).toBe("SIGKILL");
+    await expect(readFile(marker)).rejects.toMatchObject({ code: "ENOENT" });
+    const table = await readProcessTable();
+    expect(table!.some(row => row.pid === ready.descendantPid && !row.state?.startsWith("Z"))).toBe(false);
+    expect(logs.join("")).toMatch(/groups still refusing SIGKILL: none/);
+  } finally { await stop.stopAll(); }
+});

tests/runner-e2e/server-stop.ts

@@ -124,7 +124,26 @@ export function createRunnerE2EServerStopper(options: {
       if (state.finalRequested || state.restartPromise) return Promise.reject(new Error("Restart already requested or final cleanup active"));
       state.restartPromise = (async () => {
         state.gracefulDeadline = Date.now() + options.forcedTimeoutMs;
-        await state.owner.signal("SIGKILL");
+        try {
+          await state.owner.signal("SIGKILL");
+        } catch (error) {
+          if ((error as NodeJS.ErrnoException).code !== "EPERM") throw error;
+          if (!exited(child)) child.kill("SIGKILL");
+          while (!exited(child) && Date.now() < state.gracefulDeadline) await wait(25);
+          if (!exited(child)) throw error;
+          const refused: number[] = [];
+          await settleUncertainIdentity(() => state.owner.observe(), () => state.gracefulDeadline!);
+          for (const group of state.owner.liveGroups()) {
+            try {
+              await state.owner.signal("SIGKILL", new Set([group.processGroupId]));
+            } catch (groupError) {
+              if ((groupError as NodeJS.ErrnoException).code !== "EPERM") throw groupError;
+              refused.push(group.processGroupId);
+            }
+          }
+          options.log(`Warning: SIGKILL of the Paperclip process groups reported EPERM after server pid=${child.pid} exited; `
+            + `groups still refusing SIGKILL: ${refused.join(",") || "none"}; continuing cleanup\n`);
+        }
         await stopOnce(child, state, "SIGKILL");
       })();
       return state.restartPromise;