mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
Devin FoleyandPaperclip
4f8dd0e25a
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute through Paperclip Runner. The server-side
transport supervises the runnerd process and restarts it when it exits
unexpectedly
> - That recovery keeps launching replacements until one restores the
durable PRP session, within a grace window
> - Each successful restore starts a fresh grace window. So a runner
that restores and then exits again a few seconds later loops with no
limit
> - In a real run, a deterministic runnerd crash looped this way 62
times in 15 minutes, with no error recorded, until a server restart
ended it
> - This pull request fails the transport after 3 restored runners each
exit within a minute of their restore. The error carries the runner's
last diagnostic
> - The benefit is that a crashing runner fails the run in under a
minute, with the reason recorded, instead of looping silently
## Linked Issues or Issue Description
**What happened?**
After runnerd exited unexpectedly, `#recoverRunnerProcess` launched a
replacement. The replacement authenticated ("runner process restored its
durable PRP session") and then exited about 5 seconds later with the
same diagnostic. `#watchRunner` then started a new recovery with a new
420-second grace window. This repeated every ~14 seconds for 15 minutes,
until the server shut down. The run produced no output and recorded no
failure during that time.
**Expected behavior**
A runner that keeps exiting right after each restore fails the transport
quickly. The failure names the runner's own diagnostic.
**Steps to reproduce**
1. Start a transport with `runnerReconnectGraceMs` set.
2. Kill runnerd. Recovery launches a replacement, which restores the
session.
3. Kill each restored replacement within a few seconds. On `master`,
recovery relaunches every time; the new test shows no failure after 3
restored runners exit.
**Paperclip version or commit**
`master` at 491dfa1c6d.
## What Changed
- `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`:
- Records the time of each successful restore.
- Both runner-exit paths in `#watchRunner` (exit and completion
rejection) now go through `#recoverOrFailRunnerProcess`. It counts an
exit that comes within `RUNNER_RESTORE_STABLE_MS` (60 s) of the last
restore. A runner that stays up longer than that resets the count.
- On the third such exit (`MAX_UNSTABLE_RUNNER_RESTORES`), it fails the
transport with `native_runner_process_exited: runnerd exited 3 times
within 60000ms of being restored: <last diagnostic>`, and does not
launch again.
- The counting rule is a pure function, `unstableRunnerRestoreCount`,
exposed through `runnerdRecoveryInternals` for tests.
- `runnerd-codex-transport.test.ts`:
- An integration test with the real runnerd: it kills each restored
runner and expects the failure after 3 restores, with exactly 4 launches
and no further relaunch.
- A unit test for the counting rule, including the reset after a stable
runner.
## Verification
- `npx vitest run src/live/runnerd-codex-transport.test.ts -t "keeps
exiting right after|stable window|replacement whose staging"`: 3 pass (2
new, and the existing replacement-restore test).
- Red check: with `master`'s transport and the same tests, the loop test
fails. The transport never reports the failure and keeps relaunching.
- Full `npx vitest run src/live/runnerd-codex-transport.test.ts`: 296
pass.
- `npx tsc -p tsconfig.json --noEmit` in `packages/paperclip-runner`
(after `pnpm run ensure:eval-build-deps`): clean.
## Risks
- A runner that crashes repeatedly right after restoring now fails the
run after 3 restores instead of retrying indefinitely. The server's own
recovery then handles the failed run as it does any other
`native_runner_process_exited` failure.
- A healthy runner that restores and stays up longer than 60 seconds is
unaffected, and a later exit starts a new count.
- A companion pull request, #20607, removes the runnerd crash that
caused the observed loop.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use
(shell, file editing, vitest and cargo runs, read-only log analysis).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>