fix(native): report a Runner caught mid-exit as not running (#21987)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes an idle session after 5 minutes. A monitor loop
polls the Runner process with `REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT`
until the process exits.
> - The check reads `/proc/<pid>/cmdline` and exits 48
(`process_runner_id_missing`) when the command line does not hold
`--runner-id`.
> - A Linux process that is exiting releases its memory before it
becomes a zombie. For that short time its command line reads as empty,
but the process still answers `kill -0` and its stat is not yet `Z`.
> - So a check that lands in that window reports exit 48, and the close
fails although the Runner is exiting as asked.
> - This pull request makes the check wait up to 1 second for a process
of the saved generation whose command line is empty. If the process
becomes a zombie or goes away, the check exits 45
(`process_not_running`).
> - The benefit is that an idle warm close no longer fails, and is no
longer logged as a failure, when it catches the Runner while it exits.

## Linked Issues or Issue Description

Refs #21853

**What happened?**
Restart-chaos rerun 11 A1 (master 22d3f9a03c, wave
w-20261117T084017-s7106) logged `idle warm native session close failed`
with `runner_remote_process_identity_unavailable
[process_verification/process_runner_id_missing]` for run `6871fda4`
(t011, Codex, Daytona), close attempt 1, at 09:09:33Z.

**Evidence**
- The run succeeded at 09:04:34Z. The idle close ran 5 minutes later.
The server logged `transport_close_requested` at 09:09:33Z, then the
close failure in the same second.
- The Runner marker holds pid 714, with a saved process generation
(`linux:<boot>:<ticks>`). So the generation check passed, and the
process at pid 714 was still the Runner.
- The Runner's log-rotation watcher (`while kill -0 "$pid"; do sleep 15;
…`) started its last `sleep 15` at 09:09:21Z, so pid 714 was alive then.
The sandbox capture at 09:09:35Z shows no pid 714. So the Runner exited
between 09:09:21Z and 09:09:35Z, during the close.
- Exit 48 needs a live pid whose generation matches and whose command
line lacks `--runner-id`. The only way for the same process to lose its
command line is the exit window, after the kernel releases its memory
and before it becomes a zombie.
- The next evidence line shows the sandbox Runner `suspended`,
`settled`, identity matching, and no Runner process.
`warmCloseFailureProvesOwnerGone` accepted this proof (#21853) and
released the owner on attempt 1 (`warm native session's Runner or
sandbox is gone; released its owner`). No retry was needed.
`warm-shutdown.json` reports pass, 0 leaked sandboxes and the
environment deleted.

**Expected behavior**
A check that catches the Runner while it exits reports
`process_not_running`, so the monitor confirms the exit and the idle
close completes cleanly.

**Why the close failed spuriously**
The monitor loop in `native-session-executor.ts` accepts only exit 0
(alive, poll again) or exit 45 (exit confirmed). Any other exit code
throws `RemoteRunnerRecoveryError`, which fails the close.

## What Changed

-
`server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts`:
in `REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT`, when the command line lacks
the runner id, is empty, and the saved process generation matched, the
check polls `/proc/<pid>/stat` and `kill -0` every 50 ms for up to 1
second. It exits 45 when the process becomes `Z` or `X`, or is gone.
Otherwise it exits 48, as before.
-
`server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`:
two regression tests. A process of the saved generation with an empty
command line and state `R` becomes a zombie, or is reaped, while the
check runs: the check exits 45. The existing cases are unchanged: a live
process with an empty command line (state `S`) still exits 48, and
without a saved generation a zombie still exits 48. The helper timeout
goes from 2 to 5 seconds, because the unchanged empty-command-line case
now waits 1 second.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/sandbox-runner-port-holders.test.ts`: 18
passed.
- Red on master: with the master script, both new tests fail with
`expected 48 to be 45`.
- `pnpm verify` selected the full server Vitest suite, because many
server modules import this file. I stopped that local run so that it did
not load the machine while a chaos wave ran. Required CI supplies the
broad coverage.- No chaos wave was run, and no Daytona sandbox was
touched. Rerun 11 holds the slot.

## Risks

- Low risk. The wait runs only when the command line is empty and the
saved generation matches, so it applies only to the Runner's own process
while it exits. A live process that keeps an empty command line still
exits 48, after 1 extra second.
- A check without a saved generation does not change, because there a
pid can be reused.
- The script changes no persisted state. Markers, state files and
checkpoints saved by the previous release are read the same way.
- The wait is in the recovery and close checks only. It adds no work
before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) for the log analysis, code and
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-17 04:31:45 -06:00
1 parent 55228527cf
commit ca34aca777
2 files changed
+51 -1

No files matched your search

@@ -11,7 +11,7 @@ import {
function shellExit(script: string, args: string[]) {
try {
execFileSync("sh", ["-c", script, "recovery-test", ...args], { stdio: "pipe", timeout: 2_000 });
execFileSync("sh", ["-c", script, "recovery-test", ...args], { stdio: "pipe", timeout: 5_000 });
return 0;
} catch (error) {
return (error as { status: number }).status;
@@ -77,6 +77,34 @@ describe("remote recovery diagnostic shell receipts", () => {
} finally { await rm(root, { recursive: true, force: true }); }
});
it.each([
{ exit: 'fs.writeFileSync(stat, real.call(fs, stat, "utf8").replace(" R ", " Z "))', name: "becomes a zombie" },
{ exit: "fs.rmSync(stat)", name: "is reaped" },
])("reports a Runner caught mid-exit, with its command line already released, as not running once it $name", async ({ exit }) => {
const root = await mkdtemp(join(tmpdir(), "remote-probe-exiting-"));
const marker = join(root, "marker");
const stat = join(root, "stat");
const hook = join(root, "exit-on-second-stat-read.cjs");
try {
await writeFile(marker, `nonce\n${process.pid}\n2026-11-17T08:51:21.931Z\nrunner\n`);
await writeFile(join(root, "cmdline"), "");
await writeFile(stat, `${process.pid} (MainThread) ${["R", ...Array(18).fill("0"), 100].join(" ")}\n`);
await writeFile(join(root, "boot_id"), "abcd-1234\n");
await writeFile(hook, [
'const fs = require("node:fs");',
`const stat = ${JSON.stringify(stat)};`,
"const real = fs.readFileSync;",
"let reads = 0;",
`fs.readFileSync = function (path, ...rest) { if (path === stat && ++reads === 2) ${exit}; return real.call(this, path, ...rest); };`,
].join("\n"));
const script = `NODE_OPTIONS="--require=${hook}"; export NODE_OPTIONS; ` + REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT
.replaceAll('"/proc/" + pid + "/cmdline"', JSON.stringify(join(root, "cmdline")))
.replaceAll('"/proc/" + pid + "/stat"', JSON.stringify(stat))
.replaceAll('"/proc/sys/kernel/random/boot_id"', JSON.stringify(join(root, "boot_id")));
expect(shellExit(script, [marker, "nonce", "runner", String(process.pid), "linux:abcd-1234:100"])).toBe(45);
} finally { await rm(root, { recursive: true, force: true }); }
});
it("distinguishes missing, unsafe and valid marker reads without exposing their contents", async () => {
const root = await mkdtemp(join(tmpdir(), "remote-recovery-read-"));
const marker = join(root, "marker");
@@ -206,6 +206,27 @@ function verifyGeneration() {
process.exit(46);
}
}
function awaitExitingProcess() {
const pause = new Int32Array(new SharedArrayBuffer(4));
const until = Date.now() + 1000;
while (Date.now() < until) {
let state;
try {
const stat = fs.readFileSync("/proc/" + pid + "/stat", "utf8");
state = stat.slice(stat.lastIndexOf(") ") + 2).trim().split(/\s+/)[0];
} catch (error) {
if (error.code === "ENOENT") process.exit(45);
return;
}
if (state === "Z" || state === "X" || state === "x") process.exit(45);
try { process.kill(Number(pid), 0); }
catch (error) {
if (error.code === "ESRCH") process.exit(45);
return;
}
Atomics.wait(pause, 0, 0, 50);
}
}
requireLiveProcess();
const commandPath = "/proc/" + pid + "/cmdline";
if (process.platform !== "linux" && !expectedGeneration && !fs.existsSync(commandPath)) process.exit(0);
@@ -219,6 +240,7 @@ catch (error) {
if (expectedGeneration) verifyGeneration();
if (!args.includes("--runner-id") || !args.includes(runnerId)) {
requireLiveProcess();
if (expectedGeneration && args.join("") === "") awaitExitingProcess();
process.exit(48);
}
' "$pid" "$expected_runner_id"` + ' "${5:-}"',