mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute in a remote Runner that the server controls over
the durable PRP channel
> - After a server restart, the next server reattaches to the live
Runner and sends recovery barriers (`runner.drain`, `session.snapshot`)
> - These barriers timed out while the Runner stayed connected, and each
same-run retry failed the same way
> - After three attempts the run stopped with
`native_session_retry_exhausted` and needed a manual Retry
> - This pull request finds the two causes and fixes both in the
controller and the transport
> - The benefit is that restart recovery finishes on its own, without a
manual Retry
## Linked Issues or Issue Description
**What happened?**
After a server restart, a run reattached to its live remote Runner. The
Runner authenticated. Then a recovery command got no recorded answer for
30 s or more, although the Runner stayed connected. Errors on staging:
- `PRP command session.snapshot timed out (runner connected; pending
ahead 0; last runner event seq 869 55s ago)`
- `PRP command runner.drain timed out (runner connected; pending ahead 1
[semantic_tool.result] oldest 170s ...)`
- `runnerd exited while waiting for runner.drain`
Each reattach failed the same way. After three attempts the run ended
with "Workspace repair required" (`native_session_retry_exhausted`).
Root causes, proven with the real Runner and the fake Codex harness:
1. The Runner answered in about 40 ms. At reconnect it also sends the
event backlog that it collected while the server was down. The
controller handles inbound frames of a connection in order, and it
commits each event (accounting write and checkpoint flush) before the
next frame. So the answer waited behind the whole backlog. With 400
events at 100 ms each, the answer was handled after 44 s, and the 30 s
barrier had already failed. The `semantic_tool.result` case is the same:
its answer waited behind the backlog, and the next command is sent only
after that answer is handled. The Runner code was not the cause.
2. When a Runner did not answer during close, the close path queued
`runner.drain` and `runner.suspend`, kept the Runner alive for
reattachment, and saved them in the journal. The next attempt delivered
that old `runner.suspend` to the reattached Runner, and the Runner
suspended and exited.
**Expected behavior**
A reattach to a live Runner completes recovery. A Runner answer that is
queued behind earlier frames is not a timeout. A reattach never delivers
a terminal command from an earlier attempt.
**Steps to reproduce**
1. Start a Codex run in a remote Runner with a turn that streams output.
2. Restart the server. Let the Runner collect a few hundred events while
it is disconnected.
3. Let the next server reattach while event commits are slow, as on a
loaded database.
4. See `session.snapshot timed out (runner connected; pending ahead 0)`.
5. Alternatively, let a close path leave `runner.drain` and
`runner.suspend` pending, then reattach. See `runnerd exited while
waiting for runner.drain`.
**Paperclip version or commit**
`master` at 3fb35fcf13.
**Deployment mode**
Hosted (staging), remote Daytona sandbox Runner.
**Agent adapter(s) involved**
Codex (native Runner, `paperclip-runnerd`).
## What Changed
- `DurablePrpControlPlane` records each Runner `command_result` frame
when it arrives, before the ordered frame chain handles it.
`queuedCommandResultProgressAt(commandId)` returns the latest time that
the chain made progress while that answer is queued.
- `waitForRunnerCommand` keeps waiting past its deadline while the
answer to the command, or to a pending command ahead of it, is queued
and the chain advanced in the last 30 s. When the chain stops for 30 s,
the barrier fails as before. A wait that starts after its deadline still
rejects a late ACK. Frame order does not change.
- When a barrier completes after such an extension, it credits the time
past the deadline (plus the 5 s post-stop grace) to the admission clock
(`RunnerStallLedger.credit`). Later barriers that share a cold admission
deadline (Pi, OpenCode, ACPX) keep their time.
`#waitForProviderIdentityWork` checks the ready state once before it
rejects an expired deadline.
- `DurablePrpControlPlane.withdrawUndeliveredTerminalCommands()` runs
only when the transport adopts a live Runner. The controller sends only
the first pending command, so no command after it reached the Runner,
and a live, serving Runner cannot have executed a suspend. The method
removes every pending `runner.suspend` and `runner.shutdown` from that
pending suffix. It keeps every other pending command, for example a
`semantic_tool.result` that a still-running tool callback queued after
the close, and renumbers the kept commands so the Runner's command
sequence stays contiguous. Pending drains stay, because they have no
effect.
- A generated command id now skips an id that a renumbered command
already holds.
- `doc/execution-semantics.md` describes both rules.
### Changed existing test
`runnerd-codex-transport-remote-retention.test.ts` asserted that the
reattaching controller delivers the failed close's `runner.suspend`, so
the adopted Runner suspends and exits, and a later attempt relaunches it
(#20923). On staging that path failed the attempt with `runnerd exited
while waiting for runner.drain` (run 3e28c2b0) and did not recover. This
PR requires that a reattach does not inherit that suspend. The test now
asserts that the adopted Runner stays alive, answers `thread/read`,
keeps a `ready` journal, gets no suspend, and no duplicate Runner is
launched. The retention part of the test (the close keeps the frozen
Runner alive) is unchanged.
## Verification
- Red before the fix, green after (real `paperclip-runnerd` and fake
Codex):
- `... past an event backlog ahead of its recovery snapshot result`
failed with the staging error `session.snapshot timed out (runner
connected; pending ahead 0; ...)`. Debug timing showed the Runner
answered in 40 ms and the controller handled the answer 44 s later.
- `... past a runner.suspend left by an earlier attempt's close path`
failed with the staging error `runnerd exited while waiting for
runner.drain`.
- `... past a runner.suspend with a tool result queued after it` failed
the same way with tail-only withdrawal (the Greptile case), and passes
now. The tool result is delivered once and all sequence numbers stay
contiguous.
- Unit tests for `waitForRunnerCommand`: it waits while a queued answer
advances, it times out when progress stops, and a later barrier that
shares the admission deadline still completes after an extended one.
- `... past a tool result its predecessor queued after the restart`
guards delivery of a `semantic_tool.result` queued between controller
generations.
- The existing `shares Pi's absolute cold deadline across recovery
barriers and rejects a late ACK` passes unchanged.
- `cd packages/paperclip-runner && pnpm exec vitest run
src/live/runnerd-codex-transport.test.ts
src/live/runnerd-codex-transport-remote-retention.test.ts
src/live/runnerd-codex-transport-reattach-carried-commands.test.ts`
passes (320 tests).
- Related suites pass: adoption, carried suspend, stale controller
journal, stall, deadlines,
`src/backends/harness-driver-backend.test.ts`, and all
`src/control-plane` tests (330 tests).
- `tsc --noEmit -p tsconfig.test.json` passes in
`packages/paperclip-runner`.
- `pnpm verify` asked for broader coverage (exit 2). Required CI
supplies it.
## Risks
- A barrier can now wait longer than its deadline when the Runner has
answered and the controller still commits a large backlog. The wait ends
30 s after progress stops. This trades a slower recovery for no failed
run.
- The withdrawal applies only to a live adopted Runner. If such a Runner
had somehow journaled a withdrawn suspend, the next command gets a
`controller_sequence_mismatch` rejection, and the Runner executes
nothing.
- Renumbering changes the `controllerSeq` of pending commands that never
reached the Runner. Their command ids do not change.
- Not in this PR: a fallback that stops a connected Runner that never
answers and resumes it from its journal. With these fixes, the observed
staging failures no longer reach that state.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, browser for read-only staging API reads).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>