Files
PaperClipAI/packages
DottaandClaude Opus 5.5 d9b977baf2 fix(runner): answer a live Runner's commands while it replays its backlog after a reconnect (#21442)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox is driven by a Runner process. The
server talks to it over PRP and saves the controller state in a
checkpoint.
> - After a release swap, the incoming server reconnects to Runners that
kept working while no server was attached. Each Runner first replays the
events it built up.
> - The server answered commands one at a time and only after every
event queued ahead of them. It wrote one checkpoint and one
acknowledgement per event. A recovery read sent at reconnect therefore
timed out at 30 s behind the replay, although the Runner was alive and
had answered.
> - A tool call that the previous server started and never answered
blocked every later command, and a failed close was replayed forever.
> - This pull request lets read and barrier answers pass the replay,
extends a wait while the replay still commits, acknowledges a replayed
burst once per batch, delivers every pending command at attach, settles
an orphaned tool call at attach, and retries a close whose drain timed
out.
> - The benefit is that a live Runner is reached after a late reconnect
instead of being declared unanswered and then replaced, which lost its
turn.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

Authentication took under 1 s. The identity was recorded 4 to 7 minutes
later, because it is written only after the Runner answers `run.attach`,
`runner.drain` and the thread read. Those answers waited behind the
replayed event backlog. The failure codes on the first attempt were:

- `PRP command session.snapshot timed out` and
`codex_history_read_failed: thread/turns/list read failed` (all 10
`ingress` runs in the sample)
- `PRP command runner.drain timed out (runner connected; pending ahead 2
[semantic_tool.result] oldest 1023s)` on the retry

A separate class, `cleanup_quarantined` (23 failures in 3 hours, no swap
involved), showed "prior session cleanup remains incomplete" on every
retry, because a close that failed once kept replaying the same
rejection.

A live Daytona capture showed the stale `semantic_tool.result`: Codex
waited for the result of an exec tool call that the stopped server
owned. It was not an unacknowledged frame.

**Expected behavior**

A live Runner answers recovery commands within its deadline after a late
reconnect. A tool call left by an earlier controller is settled once
(with its durable receipt, or as interrupted with an unknown outcome,
never executed again). A close that failed only because its journal
drain timed out reaches its real outcome on the next close.

**Steps to reproduce**

1. Start a control plane with a commit callback that takes 50 ms per
event.
2. Connect a Runner that replays 1,000 events, then send
`session.snapshot`.
3. On master the snapshot times out at its 30 s deadline. The new test
`answers a command issued during a 1,000-event replay within its
deadline` does this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

All changes are controller-side TypeScript under
`packages/paperclip-runner/src`.

- `durable-prp-control-plane.ts`: a `command_result` for
`session.snapshot`, `session.goal.get`, `runner.drain` or
`semantic_tool.result` that reports completed or rejected is applied
when it arrives, not after the events queued ahead of it. Failed
results, lifecycle, attachment and turn commands keep their place.
- `durable-prp-control-plane.ts`: the welcome carries every consecutive
pending command (up to 32, within the existing byte bound). A lifecycle
command or `run.attach` that is not first is never batched.
- `durable-prp-control-plane.ts`: an ordinary event inside a replayed
burst is committed at once but checkpointed and acknowledged once per
batch (when the queue drains, when a critical event arrives, or after 64
events). An ACK is still sent only after the checkpoint that contains
its cursor. The owner fence in `flushCheckpoint` is unchanged. Critical
events keep their own checkpoint and ACK.
- `runnerd-codex-transport.ts`: `waitForRunnerCommand` also extends a
deadline while the controller keeps committing the Runner's events. A
Runner with no committed event for 30 s still times out at the original
deadline. A Runner that keeps emitting but never answers times out after
at most 5 more minutes.
- `runnerd-codex-transport.ts`: recovery of a semantic tool input that
an earlier generation dispatched and never answered now also runs when
the controller attaches, not only after startup completes.
`paperclip_finish` and `paperclip_block` still wait for startup.
- `runnerd-codex-transport.ts`: a close that failed only because the
controller journal drain timed out is no longer cached. The next close
drains again with a fresh bound and finishes as settled or as a typed
`NativeSessionCloseUnrecoverableError`.

## Verification

Red on master, green on this branch:

- `durable-prp-control-plane.test.ts`
- `answers a command issued during a 1,000-event replay within its
deadline`
- `keeps a failed command result behind the events the Runner sent
before it`
- `delivers every command pending at attach so an earlier generation's
result cannot hold the recovery probe`
  - `never delivers a lifecycle command ahead of its turn at attach`
- `acknowledges a replayed burst once per batch, never ahead of its
durable checkpoint`
- `still acknowledges a lifecycle event inside a burst before the next
frame`
- `answers a snapshot from a Runner whose provider stays wedged until
its whole replay is acknowledged` (models the Daytona capture: the Codex
Runner stops reading provider output while it holds unacknowledged
events)
- `runnerd-codex-transport-reattach-carried-commands.test.ts`
- `keeps waiting past the deadline while the controller is still
committing the Runner's replayed events`
- `times out a Runner that keeps replaying but never answers once the
replay extension is spent`
- `does not extend a deadline for a Runner whose replay has stopped
advancing`
- `runnerd-codex-transport-orphaned-tool-at-attach.test.ts`: `settles an
earlier generation's in-flight tool call at attach even when that attach
never finishes startup`
- `runnerd-detach-durability.test.ts`: new case `close_timeout` in
`joins an output flush before controller retirement permits a successor
writer`

Measured answer latency for a snapshot sent during a replay (50 ms
commit callback): 100 / 1,000 / 10,000 events took 13.95 s / more than
900 s / did not finish on master, and 26 ms / 26 ms / 170 ms with this
branch.

Local runs on this branch: `tsc --noEmit` is clean for
`packages/paperclip-runner` and `server`.
`durable-prp-control-plane.test.ts`: 227 of 227 pass.
`runnerd-codex-transport-reattach-carried-commands`, `-adoption`,
`-orphaned-tool-at-attach` and `runnerd-detach-durability`: all pass.
`runnerd-codex-transport-settlement` and `-deadlines`: the run on this
shared machine (load 50 to 96) had 8 tests fail. A rerun of those two
files alone (load 40 to 72) still had 7 failures: one timing assertion
(`expected 7419 to be less than 3500`) and six settlement cases that
ended with `native_cleanup_maintenance_unproven` after timeouts. The set
of failing cases changed between the two runs. The CI Runner support
suite ran both files on this head and passed, so I rely on CI for them.

Runner content ID: this PR changes only controller-side TypeScript under
`packages/paperclip-runner/src`. There is no Rust, `protocol/` or
`generated/` change. The Runner content ID changes because its hash
covers that source, but existing Runner images work with this
controller. The Daytona cells below ran this controller against an image
built for an earlier content ID.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR and
three others (the reattach-first PR (`fix/reattach-live-runs-first`),
the adopt-live-Runner PR (`fix/adopt-live-runner-on-retry`), and #21362,
which is merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed with
`runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. That chain did not recur. All 6 runs that crossed swap 2 were
adopted on the same Runner process, with no retry and no 900 s timeout.
- Caveat: G5c ran before the reattach-phase timing fix in the
reattach-first PR (`fix/reattach-live-runs-first`). The command fixes in
this PR were present in G5c except the test-only fix for the
wedged-provider test.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection): 0
failed runs at either swap; 11 of 11 survivors at both. Swap 2 was slow
(see the reattach-first PR (`fix/reattach-live-runs-first`)).

## Risks

- Not proven: the Runner-side stall. A Codex Runner stops reading
provider output while no controller is attached (bounded 256-line
channel, provider stdout pipe full). This PR makes the controller
consume the replay quickly, so the stall clears in about 2 to 6 s once
the controller drives it. The durable fix is in the Runner (keep
draining provider output to the durable outbox while disconnected, save
once per ACK batch, run commands ahead of ACKs). That needs a new Runner
image and is not in this PR.
- Not proven: why a connected Runner with an empty outbox sometimes did
not answer for more than 30 s. The G5 capture points to the server
stalling past its own deadline (a `session.open` answered and
acknowledged 8 minutes before the server reported it timed out), not to
the Runner.
- Not done: cleanup takeover for a same-run retry when
`remoteCleanupScope` is absent, and for a predecessor run that shares
the warm sandbox. This PR only stops the replayed close failure.
- Time to reach a wedged Codex Runner still grows with its backlog:
about backlog x per-event commit / database concurrency (40 runs x 1,000
events x 20 ms / 8 is about 100 s).
- Behavior shift: a crash during a batched burst replays at most the
unacknowledged batch (up to 64 events). The Runner still holds those
events, and commit order is unchanged.
- A Runner that keeps emitting but never answers now fails after at most
5 more minutes instead of 30 s.
- The predecessor-retirement PR (`fix/retire-predecessor-session`)
builds on the close path changed here (`retry a close whose controller
journal drain timed out`).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code. The fixes were written by Claude agents in the
same session family.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:34:31 -06:00
..