Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 3374ca54f5 fix(runner): record a runner exit mid-turn as runner loss, and stop the lineage sampler panicking (#20259)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner (`runnerd`) runs each provider turn, and the
server records why a run stopped and what the person can do next
> - When `runnerd` exits during a provider turn, the server saved the
run as `native_provider_terminal_failed`: "The provider session is
permanently unusable", with automatic retries stopped
> - The real cause was the runner process, and the transport already
names it (`native_runner_process_exited`) at the start of the
closed-stream message
> - Separately, the process-lineage sampler added in #19727 panicked on
Linux debug builds when a provider tool started its own session, and
that panic text became the run's visible failure
> - This pull request records a runner exit mid-turn as runner loss, and
reads session IDs without the assertion that panicked
> - The benefit is that a person sees the true cause of a stopped run,
and recovery treats a lost runner like every other runner exit

## Linked Issues or Issue Description

Refs #19727

**What happened**

A chat eval killed only the owned `runnerd` process (SIGKILL) while a
provider tool was running. The run failed with `errorCode:
native_provider_terminal_failed` and the next action "The provider
session is permanently unusable". The error text was `Notification
transport failed closed: native_runner_process_exited: runnerd exited
unexpectedly: paperclip-runnerd panic: panicked at
rustix-1.1.4/src/pid.rs:61:9: assertion failed: raw > 0`. The panic came
from the lineage sampler thread before the kill. It called
`rustix::process::getsid`, which asserts a nonzero session. A kernel
thread, or a process whose session leader is outside the PID namespace,
reports session 0.

**Expected behavior**

A runner process exit is recorded as `native_runner_process_exited`, the
same cause the server already uses for a runner exit outside a turn.
Recovery then applies its normal runner-loss rules (checkpoint evidence,
bounded attempts, cleanup quarantine). The lineage sampler does not
panic, and its diagnostics do not replace the real failure text.

**Steps to reproduce**

1. Start a native OpenCode run on Linux with a debug `runnerd`.
2. Let the agent run a shell tool that starts its own session (for
example a long `node -e` command).
3. Send SIGKILL to the `runnerd` PID only.
4. Read `GET /api/heartbeat-runs/<runId>`: `errorCode` is
`native_provider_terminal_failed` and the error names the rustix panic.

## What Changed

- `nativeSessionFailureSourceCode` returns
`native_runner_process_exited` for a `NativeProviderTerminalFailure`
with code `notification_transport_failed` when its message starts with
`Notification transport failed closed: native_runner_process_exited: `.
Only the runner transport writes that prefix. Other closed-stream causes
stay `native_provider_terminal_failed`.
- `paperclip-process-table` has a new `session(pid)` function. Linux
reads the session field of `/proc/<pid>/stat`. macOS calls `getsid`. A
session of 0 or an error gives `None`.
- `process_lineage` reads sessions through
`paperclip_process_table::session` instead of rustix `getsid`.
- Regression tests: the runner-loss classification (attempts 1 and 3,
and three negative shapes), the stat session parser with a kernel-thread
line, and a live read of this process's session.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "runner
process exit that closed|preserves stable provider and runner|model
rejection from a provider terminal|operator-owned through terminal"` (8
passed)
- `cd packages/paperclip-runner/runner && cargo test -p
paperclip-process-table` (macOS: 3 passed; the Linux-only parser tests
run in CI)
- `cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --lib process_lineage` (6 passed)
- End to end: the `recovery.*` chat eval cells that kill `runnerd` are
run against this change on a separate eval branch.

## Risks

- Behavior shift: a runner exit during a turn now follows runner-loss
recovery instead of stopping as a permanent provider failure. Recovery
still decides from durable evidence. Ambiguous state stays terminal, and
an unverified cleanup stays quarantined.
- Previous-release story: no persisted shape changes. Runs saved earlier
with `native_provider_terminal_failed` keep that code. Only new failures
are classified with the new rule.
- The session read on Linux now uses `/proc` instead of a syscall. It
gives the same value, and it reports 0 (now `None`) in the same cases.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 06:38:24 -06:00
DottaandClaude Opus 5.5 b5b972ceb1 fix(runner): stop provider tool processes that leave the owned process group (#19727)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The local runner (`paperclip-runnerd`) supervises each provider
process, such as the ACPX sidecar for Claude, in its own process group
> - Run teardown stops that process group, and nothing else
> - Claude Code starts each Bash tool command in a new session
(`setsid`). A command such as `nohup ./bench.sh & echo $!` leaves a job
in that session, and the wrapper exits at once. The job is then a child
of init and outside the owned group
> - In a restart chaos storm, a run failed after a restart resume. Its
tool job kept running beside the task's next run, in the same workspace,
for more than four minutes
> - This pull request makes the supervisor record every process that its
provider starts, including processes that leave the group, and stop them
all before teardown reports success
> - The benefit is that a run's tools cannot outlive the run and cause
duplicate side effects in the next run

## Linked Issues or Issue Description

**What happened?**
A restart chaos storm ran a local ACPX Claude task. After a restart
resume, the agent ran `nohup ./bench.sh <nonce> > log 2>&1 & echo
"started pid $!"` and then ended its turn without a result. The run
failed with `native_finalization_missing`. The runner tore down the
sidecar group, and the guardian killed the provider group. `bench.sh`
was in the session that Claude Code made for the tool (pgid and sid =
the exited wrapper, ppid = 1). It was never signalled. It kept forking
`sleep 1` while the task's next run started and finished in the same
workspace.

**Steps to reproduce**
1. Run a local ACPX Claude task whose agent runs a Bash tool command
such as `nohup ./bench.sh x > log 2>&1 & echo $!`.
2. End the run (any outcome, including failure after a restart resume).
3. Run `ps -axo pid,ppid,pgid,command | grep bench.sh`. The job is still
running with ppid 1.

**Agent adapter(s) involved**
Claude Code (local ACPX). The gap is in the shared supervisor, so any
provider whose tools leave the group is affected.

**Root cause**
`SupervisedProcess`
(`packages/paperclip-runner/runner/crates/runner-core/src/process_supervisor.rs`)
spawns with `process_group(0)` and stops only that group in
`terminate_group`, `terminate_group_immediately`, `wait` and `Drop`. Its
own comment said that a caller that must exclude escaped or re-parented
descendants needs a separate lifetime fence. No caller had one. The ACPX
sidecar transport (`acpx_sidecar_transport.rs`, `shutdown` and `poison`)
and the ACPX provider guardian (`installation-integrity.ts`, which kills
only its own group) therefore left every `setsid` tool job behind. The
failure path after the resume was not special: the same gap applies to
every teardown.

**Expected behavior**
When a run ends for any reason, every process that its provider or tools
started stops, and teardown proves that they are gone. Identity comes
from PID, group, session and start time, never from a command name.

## What Changed

- Add the `paperclip-process-table` crate. It reads PID, parent, process
group and start time for every process (libproc on macOS, `/proc` on
Linux). `runner-core` forbids unsafe code, so the two checked libproc
calls live in this small crate.
- Add `process_lineage.rs` to `runner-core`. A sampler thread (every 50
ms) records each descendant of the supervised process by PID and start
time. It also records each group and session that a descendant leads,
with the leader's start time. A process belongs to the lineage when its
parent belongs, or when it is in a recorded group or session and started
no earlier than its leader. Groups and sessions that the supervised
process only inherited (runnerd's session) are never recorded. A
recorded ID whose PID now names a different process is dropped.
- `SupervisedProcess` samples the lineage just before it signals, and
after the leader is reaped it stops every live member: TERM, then KILL
after the shutdown grace. It fails with an error until it proves that no
member is alive (grace plus 5 s). This applies to `terminate_group`,
`terminate_group_immediately`, `wait` and `Drop`.
- If the lineage sampler cannot start, the supervisor stops the process
it just started and reports the failure, so a provider never runs
without tool cleanup.
- Table reads and state updates happen under one lock, so an older read
cannot erase members that a newer read recorded.
- The process table reports processes that exist but cannot be read. A
recorded member that cannot be read stays unresolved, and teardown fails
until its exit is proven.
- The Runner workspace now has two members.
`scripts/runner-release-crate-cache.mjs` and
`scripts/runner-debug-crate-cache.mjs` cache and mark current the
process-table lib unit with the Runner crate (otherwise Cargo rebuilds
the Runner crate after every restore). The debug contract accepts only
this library-only member and still refuses any other workspace change.
The workflow cache path lists and the debug clean step are updated to
match.
- `terminate_group_immediately` now accepts `EPERM` from the group
signal when the leader has already exited. macOS returns `EPERM` for a
group whose only member is the unreaped leader. The lineage pass then
proves that no other member remains.

## Verification

- `cargo test -p paperclip-runner-core --lib -- process_lineage tool_job
immediate_group_fence` (macOS, local).
- New real-process tests in `process_supervisor.rs`: a supervised shell
starts a wrapper with `setsid` (Perl `POSIX::setsid`), the wrapper runs
`nohup <loop> &`, writes the job PID and exits. The test asserts that
the job is outside the owned group and re-parented, then calls
`terminate_group`, `terminate_group_immediately` or drops the process,
and asserts that the job is gone when the call returns.
- Red: with the tracker disabled, all three fail with `tool job <pid>
outlived its provider`.
  - Green: all three pass.
- Unit tests in `process_lineage.rs` cover a re-parented job, a job that
left its group but stayed in the recorded session, a session made after
the first sighting, PID reuse, and no claim through the supervisor's
inherited session.
- `node --test scripts/__tests__/runner-release-crate-cache.test.mjs
scripts/__tests__/runner-debug-crate-cache.test.mjs` (new case: the
process-table library is cached with the crate, and any other member
still needs review).
- `cargo test -p paperclip-process-table` (includes an exited process
reported as gone), and `cargo test -p paperclip-runner-core --lib` (570
passed).
`acpx_provider_backend::tests::unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend`
fails sometimes under full-suite load. It fails at the same rate with
the tracker disabled, so it is a flake that already existed.
- Sampler cost: one table read takes about 0.4 ms for 1,200 processes on
macOS (about 0.8% of one core for each supervised process).
- The restart-chaos replay (`--replay w-20261107T023921-s7106/t67`) did
not run. The local session was not permitted to load provider
credentials. It is not proven against a live Claude run.

## Risks

- Teardown now stops all the processes that a provider started, also
intentional background jobs. This is the intended contract: nothing
outlives the run.
- A wrapper that lives less than one sample (50 ms) and orphans a job
before the sampler sees it is not recorded. Claude Code's wrapper
sources a shell snapshot and lives for seconds. On Linux, a
child-subreaper would close this gap. That is not in this change.
- When `runnerd` itself is killed with SIGKILL, this code does not run.
The provider guardian still kills its own group, but escaped tool
sessions are not stopped in that case.
- Teardown can return a new error when it cannot prove the lineage
stopped. Callers already handle `terminate_group` errors as provider
shutdown failures.
- The server does not yet gate the next run on the tool lineage of the
prior run directly. It relies on the runner finishing its teardown
before the runner exits.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 08:24:41 -06:00