Files
PaperClipAI/server
DottaandClaude Opus 5.5 ca34aca777 fix(native): report a Runner caught mid-exit as not running (#21987)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes an idle session after 5 minutes. A monitor loop
polls the Runner process with `REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT`
until the process exits.
> - The check reads `/proc/<pid>/cmdline` and exits 48
(`process_runner_id_missing`) when the command line does not hold
`--runner-id`.
> - A Linux process that is exiting releases its memory before it
becomes a zombie. For that short time its command line reads as empty,
but the process still answers `kill -0` and its stat is not yet `Z`.
> - So a check that lands in that window reports exit 48, and the close
fails although the Runner is exiting as asked.
> - This pull request makes the check wait up to 1 second for a process
of the saved generation whose command line is empty. If the process
becomes a zombie or goes away, the check exits 45
(`process_not_running`).
> - The benefit is that an idle warm close no longer fails, and is no
longer logged as a failure, when it catches the Runner while it exits.

## Linked Issues or Issue Description

Refs #21853

**What happened?**
Restart-chaos rerun 11 A1 (master 22d3f9a03c, wave
w-20261117T084017-s7106) logged `idle warm native session close failed`
with `runner_remote_process_identity_unavailable
[process_verification/process_runner_id_missing]` for run `6871fda4`
(t011, Codex, Daytona), close attempt 1, at 09:09:33Z.

**Evidence**
- The run succeeded at 09:04:34Z. The idle close ran 5 minutes later.
The server logged `transport_close_requested` at 09:09:33Z, then the
close failure in the same second.
- The Runner marker holds pid 714, with a saved process generation
(`linux:<boot>:<ticks>`). So the generation check passed, and the
process at pid 714 was still the Runner.
- The Runner's log-rotation watcher (`while kill -0 "$pid"; do sleep 15;
…`) started its last `sleep 15` at 09:09:21Z, so pid 714 was alive then.
The sandbox capture at 09:09:35Z shows no pid 714. So the Runner exited
between 09:09:21Z and 09:09:35Z, during the close.
- Exit 48 needs a live pid whose generation matches and whose command
line lacks `--runner-id`. The only way for the same process to lose its
command line is the exit window, after the kernel releases its memory
and before it becomes a zombie.
- The next evidence line shows the sandbox Runner `suspended`,
`settled`, identity matching, and no Runner process.
`warmCloseFailureProvesOwnerGone` accepted this proof (#21853) and
released the owner on attempt 1 (`warm native session's Runner or
sandbox is gone; released its owner`). No retry was needed.
`warm-shutdown.json` reports pass, 0 leaked sandboxes and the
environment deleted.

**Expected behavior**
A check that catches the Runner while it exits reports
`process_not_running`, so the monitor confirms the exit and the idle
close completes cleanly.

**Why the close failed spuriously**
The monitor loop in `native-session-executor.ts` accepts only exit 0
(alive, poll again) or exit 45 (exit confirmed). Any other exit code
throws `RemoteRunnerRecoveryError`, which fails the close.

## What Changed

-
`server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts`:
in `REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT`, when the command line lacks
the runner id, is empty, and the saved process generation matched, the
check polls `/proc/<pid>/stat` and `kill -0` every 50 ms for up to 1
second. It exits 45 when the process becomes `Z` or `X`, or is gone.
Otherwise it exits 48, as before.
-
`server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`:
two regression tests. A process of the saved generation with an empty
command line and state `R` becomes a zombie, or is reaped, while the
check runs: the check exits 45. The existing cases are unchanged: a live
process with an empty command line (state `S`) still exits 48, and
without a saved generation a zombie still exits 48. The helper timeout
goes from 2 to 5 seconds, because the unchanged empty-command-line case
now waits 1 second.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/sandbox-runner-port-holders.test.ts`: 18
passed.
- Red on master: with the master script, both new tests fail with
`expected 48 to be 45`.
- `pnpm verify` selected the full server Vitest suite, because many
server modules import this file. I stopped that local run so that it did
not load the machine while a chaos wave ran. Required CI supplies the
broad coverage.- No chaos wave was run, and no Daytona sandbox was
touched. Rerun 11 holds the slot.

## Risks

- Low risk. The wait runs only when the command line is empty and the
saved generation matches, so it applies only to the Runner's own process
while it exits. A live process that keeps an empty command line still
exits 48, after 1 extra second.
- A check without a saved generation does not change, because there a
pid can be reused.
- The script changes no persisted state. Markers, state files and
checkpoints saved by the previous release are read the same way.
- The wait is in the recovery and close checks only. It adds no work
before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) for the log analysis, code and
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 04:31:45 -06:00
..