From ca34aca77750a143dfc637c96769d64f9b14cbdb Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 17 Nov 2026 04:31:45 -0600 Subject: [PATCH] fix(native): report a Runner caught mid-exit as not running (#21987) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm agent session keeps its Runner alive in a sandbox between turns. The server closes an idle session after 5 minutes. A monitor loop polls the Runner process with `REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT` until the process exits. > - The check reads `/proc//cmdline` and exits 48 (`process_runner_id_missing`) when the command line does not hold `--runner-id`. > - A Linux process that is exiting releases its memory before it becomes a zombie. For that short time its command line reads as empty, but the process still answers `kill -0` and its stat is not yet `Z`. > - So a check that lands in that window reports exit 48, and the close fails although the Runner is exiting as asked. > - This pull request makes the check wait up to 1 second for a process of the saved generation whose command line is empty. If the process becomes a zombie or goes away, the check exits 45 (`process_not_running`). > - The benefit is that an idle warm close no longer fails, and is no longer logged as a failure, when it catches the Runner while it exits. ## Linked Issues or Issue Description Refs #21853 **What happened?** Restart-chaos rerun 11 A1 (master 22d3f9a03c, wave w-20261117T084017-s7106) logged `idle warm native session close failed` with `runner_remote_process_identity_unavailable [process_verification/process_runner_id_missing]` for run `6871fda4` (t011, Codex, Daytona), close attempt 1, at 09:09:33Z. **Evidence** - The run succeeded at 09:04:34Z. The idle close ran 5 minutes later. The server logged `transport_close_requested` at 09:09:33Z, then the close failure in the same second. - The Runner marker holds pid 714, with a saved process generation (`linux::`). So the generation check passed, and the process at pid 714 was still the Runner. - The Runner's log-rotation watcher (`while kill -0 "$pid"; do sleep 15; …`) started its last `sleep 15` at 09:09:21Z, so pid 714 was alive then. The sandbox capture at 09:09:35Z shows no pid 714. So the Runner exited between 09:09:21Z and 09:09:35Z, during the close. - Exit 48 needs a live pid whose generation matches and whose command line lacks `--runner-id`. The only way for the same process to lose its command line is the exit window, after the kernel releases its memory and before it becomes a zombie. - The next evidence line shows the sandbox Runner `suspended`, `settled`, identity matching, and no Runner process. `warmCloseFailureProvesOwnerGone` accepted this proof (#21853) and released the owner on attempt 1 (`warm native session's Runner or sandbox is gone; released its owner`). No retry was needed. `warm-shutdown.json` reports pass, 0 leaked sandboxes and the environment deleted. **Expected behavior** A check that catches the Runner while it exits reports `process_not_running`, so the monitor confirms the exit and the idle close completes cleanly. **Why the close failed spuriously** The monitor loop in `native-session-executor.ts` accepts only exit 0 (alive, poll again) or exit 45 (exit confirmed). Any other exit code throws `RemoteRunnerRecoveryError`, which fails the close. ## What Changed - `server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts`: in `REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT`, when the command line lacks the runner id, is empty, and the saved process generation matched, the check polls `/proc//stat` and `kill -0` every 50 ms for up to 1 second. It exits 45 when the process becomes `Z` or `X`, or is gone. Otherwise it exits 48, as before. - `server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`: two regression tests. A process of the saved generation with an empty command line and state `R` becomes a zombie, or is reaped, while the check runs: the check exits 45. The existing cases are unchanged: a live process with an empty command line (state `S`) still exits 48, and without a saved generation a zombie still exits 48. The helper timeout goes from 2 to 5 seconds, because the unchanged empty-command-line case now waits 1 second. ## Verification - `cd server && npx vitest run src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts src/services/native-runtime/sandbox-runner-port-holders.test.ts`: 18 passed. - Red on master: with the master script, both new tests fail with `expected 48 to be 45`. - `pnpm verify` selected the full server Vitest suite, because many server modules import this file. I stopped that local run so that it did not load the machine while a chaos wave ran. Required CI supplies the broad coverage.- No chaos wave was run, and no Daytona sandbox was touched. Rerun 11 holds the slot. ## Risks - Low risk. The wait runs only when the command line is empty and the saved generation matches, so it applies only to the Runner's own process while it exits. A live process that keeps an empty command line still exits 48, after 1 extra second. - A check without a saved generation does not change, because there a pid can be reused. - The script changes no persisted state. Markers, state files and checkpoints saved by the previous release are read the same way. - The wait is in the recovery and close checks only. It adds no work before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits) for the log analysis, code and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ...remote-runner-recovery-diagnostics.test.ts | 30 ++++++++++++++++++- .../remote-runner-recovery-diagnostics.ts | 22 ++++++++++++++ 2 files changed, 51 insertions(+), 1 deletion(-) diff --git a/server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts b/server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts index d872a071ff2..f7bb79bb987 100644 --- a/server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts +++ b/server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts @@ -11,7 +11,7 @@ import { function shellExit(script: string, args: string[]) { try { - execFileSync("sh", ["-c", script, "recovery-test", ...args], { stdio: "pipe", timeout: 2_000 }); + execFileSync("sh", ["-c", script, "recovery-test", ...args], { stdio: "pipe", timeout: 5_000 }); return 0; } catch (error) { return (error as { status: number }).status; @@ -77,6 +77,34 @@ describe("remote recovery diagnostic shell receipts", () => { } finally { await rm(root, { recursive: true, force: true }); } }); + it.each([ + { exit: 'fs.writeFileSync(stat, real.call(fs, stat, "utf8").replace(" R ", " Z "))', name: "becomes a zombie" }, + { exit: "fs.rmSync(stat)", name: "is reaped" }, + ])("reports a Runner caught mid-exit, with its command line already released, as not running once it $name", async ({ exit }) => { + const root = await mkdtemp(join(tmpdir(), "remote-probe-exiting-")); + const marker = join(root, "marker"); + const stat = join(root, "stat"); + const hook = join(root, "exit-on-second-stat-read.cjs"); + try { + await writeFile(marker, `nonce\n${process.pid}\n2026-11-17T08:51:21.931Z\nrunner\n`); + await writeFile(join(root, "cmdline"), ""); + await writeFile(stat, `${process.pid} (MainThread) ${["R", ...Array(18).fill("0"), 100].join(" ")}\n`); + await writeFile(join(root, "boot_id"), "abcd-1234\n"); + await writeFile(hook, [ + 'const fs = require("node:fs");', + `const stat = ${JSON.stringify(stat)};`, + "const real = fs.readFileSync;", + "let reads = 0;", + `fs.readFileSync = function (path, ...rest) { if (path === stat && ++reads === 2) ${exit}; return real.call(this, path, ...rest); };`, + ].join("\n")); + const script = `NODE_OPTIONS="--require=${hook}"; export NODE_OPTIONS; ` + REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT + .replaceAll('"/proc/" + pid + "/cmdline"', JSON.stringify(join(root, "cmdline"))) + .replaceAll('"/proc/" + pid + "/stat"', JSON.stringify(stat)) + .replaceAll('"/proc/sys/kernel/random/boot_id"', JSON.stringify(join(root, "boot_id"))); + expect(shellExit(script, [marker, "nonce", "runner", String(process.pid), "linux:abcd-1234:100"])).toBe(45); + } finally { await rm(root, { recursive: true, force: true }); } + }); + it("distinguishes missing, unsafe and valid marker reads without exposing their contents", async () => { const root = await mkdtemp(join(tmpdir(), "remote-recovery-read-")); const marker = join(root, "marker"); diff --git a/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts b/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts index 58d95862346..ffdee69d62d 100644 --- a/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts +++ b/server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts @@ -206,6 +206,27 @@ function verifyGeneration() { process.exit(46); } } +function awaitExitingProcess() { + const pause = new Int32Array(new SharedArrayBuffer(4)); + const until = Date.now() + 1000; + while (Date.now() < until) { + let state; + try { + const stat = fs.readFileSync("/proc/" + pid + "/stat", "utf8"); + state = stat.slice(stat.lastIndexOf(") ") + 2).trim().split(/\s+/)[0]; + } catch (error) { + if (error.code === "ENOENT") process.exit(45); + return; + } + if (state === "Z" || state === "X" || state === "x") process.exit(45); + try { process.kill(Number(pid), 0); } + catch (error) { + if (error.code === "ESRCH") process.exit(45); + return; + } + Atomics.wait(pause, 0, 0, 50); + } +} requireLiveProcess(); const commandPath = "/proc/" + pid + "/cmdline"; if (process.platform !== "linux" && !expectedGeneration && !fs.existsSync(commandPath)) process.exit(0); @@ -219,6 +240,7 @@ catch (error) { if (expectedGeneration) verifyGeneration(); if (!args.includes("--runner-id") || !args.includes(runnerId)) { requireLiveProcess(); + if (expectedGeneration && args.join("") === "") awaitExitingProcess(); process.exit(48); } ' "$pid" "$expected_runner_id"` + ' "${5:-}"',