mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
DottaandClaude Opus 5.5
fd2e7482c0
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A local native run uses one runner daemon (`paperclip-runnerd`).
runnerd starts the provider (for example OpenCode), and the provider
starts tool processes
> - When runnerd dies in the middle of a tool call, the provider and its
tools are re-parented. The only record of that process tree was in
runnerd's memory
> - Close then cannot prove that those processes stopped. It quarantines
the session, and the next attempt stops with
`native_session_cleanup_quarantined`: "Automatic recovery stopped /
operator recovery required"
> - Restarts and runner loss are product failures. Interrupted work must
recover by itself, and it must never get two writers in one workspace
> - This pull request makes runnerd journal its process lineage to disk,
and makes close stop and prove gone every journalled process after
runnerd is lost
> - The benefit is that a killed runnerd leads to an automatic, safe
retry within the normal attempt budget, instead of an operator stop
## Linked Issues or Issue Description
Refs #20259 (runner loss classification and the lineage sampler panic).
Refs #20260 (automatic continuation of an inert cleanup quarantine).
**What happened?**
A chat eval killed only the owned local `runnerd` process (SIGKILL)
during an OpenCode tool call
(`chat-features.runner-opencode.local.recovery.retry-exhausted`,
`recovery.explain-stall` and `recovery.source-action`). Attempt 1
recorded `native_runner_process_exited` with "Retry scheduled 1/3".
Attempt 2 stopped with `native_session_cleanup_quarantined`. The task
was blocked with no owner, and the next action was operator recovery.
**Expected behavior**
After runnerd is lost, the product stops the provider and tool processes
that runnerd started, proves them gone, and continues the run within its
retry budget. When it cannot prove that, it keeps the fail-closed
quarantine.
**Steps to reproduce**
1. Start a local native OpenCode (or Codex) run.
2. Let the agent run a tool that leaves the provider's session (for
example `nohup job &`).
3. Send SIGKILL to the `runnerd` PID only.
4. Close rejects with `runner did not durably suspend before
checkpoint`, and the next attempt fails with
`native_session_cleanup_quarantined`.
**Paperclip version or commit**
master at 4c15eea770
## What Changed
- `process_lineage.rs`: runnerd journals each supervised process's
lineage to `<runner state>/process-lineage/lineage-<runnerd
pid>-<n>.json`. The journal holds the root, every member, and every
group and session a member leads, each with its start time. The first
record is written when the provider is spawned, before any tool can run.
Later records are written by the existing 50 ms sampler only when the
record changes. A teardown that proves the lineage gone removes the
journal.
- `process_lineage/journal.rs` (new module): the on-disk format, the
write rules and the reaper `reap_journals`. The reaper refuses when
runnerd never journalled (no writer marker) or when runnerd is still
alive. It restores each journal into the same `LineageState` rules,
sends TERM and then KILL only to processes whose PID and start time
still match, and reports `verified` only when none remain. A journal
that it cannot read refuses the proof.
- Unsampled escapes: each record stores `sampled_through`, the newest
process start that its read listed. The value is refreshed at most once
a second while the membership is unchanged. The reaper refuses
(`unattributed_orphan`) when a same-user orphan (parent PID 1) that is
in no journal started after that point. Such a process can be a tool
that started after the last read, so the reaper neither signals it nor
ignores it.
- A failed journal write removes the writer marker. Removal needs no
free space. After that, no reap can verify from a record that may be
missing a member.
- `paperclip-runnerd`: durable mode enables the journal before any
provider starts. A new `reap-lineage --state-dir --runner-pid` mode
prints the report and exits 0 only when verified.
- `runnerd-codex-transport.ts`: close samples whether runnerd exited
unsuspended before close sent any signal. In that case, for a runnerd
that this transport spawned on this host, it runs the reaper from the
same executable. A verified report ends the session as a crash stop with
its durable record intact. Recovery then continues the run. Any other
result keeps the existing `NativeSessionCloseUnrecoverableError`
quarantine. Launcher-owned (remote) and adopted runners do not use this
path.
- `runnerd-codex-transport.ts`: close does not queue a durable
`runner.suspend` for a runnerd that it already knows is lost. Before,
that suspend stayed pending in the saved control-plane state. A later
runnerd on the same root then failed at startup with `terminal command
result acknowledgement changed its durable identity`. The shared
suspension barrier is unchanged for remote and adopted runners.
- `runnerd-codex-transport.ts`: a runnerd that exits during close (for
example because its provider snapshot is unreadable) is not a loss. It
keeps the quarantine.
- `runnerd-codex-transport.ts`: `#runnerHasExited` treated a
signal-killed runnerd as alive, because `exitCode` is `null` and only
`signalCode` is set. Close then spent its whole deadline trying to
suspend a dead runner. It now checks `signalCode` too.
- `fake-codex-app-server`: `--spawn-escaped-tool-job <pidfile>` starts a
tool job in its own session at turn start.
- `doc/execution-semantics.md`: describes the journal, the reaper and
the fallback.
## Verification
- Rust, `cargo test -p paperclip-runner-core --lib process_lineage` (14
pass) and `process_supervisor` (20 pass). New cases:
- `reaping_refuses_an_orphan_born_after_the_last_sample`: a tool starts
after the last read, starts a new session and exits, and leaves a
non-leader orphan job. The reap refuses and does not signal the job. It
fails without the guard.
- `a_failed_journal_write_withdraws_the_writer_marker` and
`an_unchanged_record_is_rewritten_only_to_advance_sampled_through`.
- `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job`: a
real provider and a real `setsid` tool job, with a dead writer. The reap
stops both, proves them gone and retires the journal and marker.
- `reaping_refuses_without_a_marker_or_while_the_writer_lives` and
`reaping_refuses_an_unreadable_journal`.
- `a_journalled_record_restores_the_same_lineage`: after a round trip,
the restored record still finds a re-parented job and its later child,
and ignores an unrelated process.
- TypeScript, live with the real debug runnerd: `stops a lost runnerd's
provider and escaped tool job from its lineage journal`.
- `journal=true`: SIGKILL runnerd mid-turn. The escaped tool job
outlives runnerd. Close resolves, the job is gone, and the report is
`verified`. Before the fix, close rejected with `runner did not durably
suspend before checkpoint` and the job stayed alive.
- `journal=false` (an older runnerd with no journal): close still
rejects, and the job is not signalled.
- `runner-lineage-reap.test.ts`: a report from an older runnerd, a
malformed report, or an inconsistent `verified` report never counts as
proof.
- Turn-start cost: the journal adds one single-process read and one
small file write when runnerd spawns its provider (about 1.5 ms in a
macOS debug build), once for each runnerd. Nothing is added for each
turn.
- Paid runner E2E,
`chat-features.runner-opencode.local.recovery.retry-exhausted`, on the
eval branch with this fix (runs 42637596571 and 42644400434). Before the
fix (run 42615758956), attempt 2 stopped with
`native_session_cleanup_quarantined`. Now each of the three runnerd
kills records `native_runner_process_exited`. Attempts 1 and 2 are
`retryable_failure`, and attempt 3 is `terminal_failure` with cause
`native_session_retry_exhausted` (attempt 3 of 3). The reaper ran after
the first kill and verified. The cell still fails one eval setup check.
The check waits for an *active* exhausted recovery action, but
`execution-recovery-resolution.ts` settles that action as resolved and
blocks the task (`preserve_without_replay_v1`). That is a separate
grader decision, outside this PR.
## Risks
- Medium. The change only affects a close where runnerd already exited
without suspending and without a signal from close. Every other close is
unchanged.
- The reaper signals processes by PID. Each signal re-checks the PID's
start time first, and a group or session is used only while its leader
PID still names the recorded process (the same rules as the existing
in-process teardown).
- A crash stop leaves the durable root unsuspended. The coordinator
already scheduled a retry for `native_runner_process_exited`. That retry
now runs instead of failing on the quarantine. The paid E2E shows that
attempts 2 and 3 start on the same run.
- The escape guard can refuse when an unrelated same-user process
becomes an orphan in the window of about one second after the last read.
The fallback is today's quarantine. An orphan of a subreaper other than
init is not seen by the guard.
- The journal adds at most one small write each second for each runnerd
while processes start on the host. This happens on the background
sampler, not before turn start.
- Previous-release story: no database or saved-session schema change.
The journal is a new file in the runner state directory. A runnerd from
an older release writes no journal and has no `reap-lineage` mode. Its
loss keeps today's quarantine, and the existing board Retry and
stopped-session proofs still apply.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>