Files
PaperClipAI/packages
DottaandClaude Opus 5.5 9e1a85fb46 fix(runner): accept a new controller when the listener's controller goes silent (#22088)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On Daytona, the Paperclip server connects to a Runner that listens
inside the sandbox, through the Daytona proxy
> - A listening Runner accepts the next controller connection only after
its current connection closes
> - When the network path broke without a close, the Runner kept a dead
socket and never accepted the new server after a release swap
> - This pull request makes a listening Runner treat a controller that
sends nothing for 90 seconds as gone, and listen again
> - The benefit is that live runs survive a network outage that ends
near a release swap, instead of losing their session and restarting
fresh

## Linked Issues or Issue Description

**What happened?**

In a restart-chaos wave (201 tasks, 6 graceful release swaps), the host
could not resolve Daytona hosts for about 12 minutes. Swap 4 happened at
the end of that outage. The new server tried to reattach to 16 live
runs. It reached none of them: all 16 were given up after 66 to 75
seconds (`reattach phase ended`, `authenticated: 0`, `givenUp: 16`).
Each run then failed its three same-run resumes with
`runner_ingress_unavailable` and continued in a fresh run. Six tasks
showed a brief Retry banner, and the fresh runs repeated work.

A sandbox snapshot taken 11 seconds after the swap shows the cause. The
Runner process was alive. Its listening socket had 24 connections in the
accept backlog and about 25 `CLOSE-WAIT` connections with an unread
upgrade request. The Runner held one established connection from the
Daytona proxy, with empty queues. Its saved state showed
`reconnectCount: 0`, and no run event had arrived since the outage
started. The Runner was waiting on a dead connection and did not call
`accept` again.

**Expected behavior**

The server already pings the Runner every 20 seconds and drops a silent
Runner after 90 seconds (#19407). The Runner side must do the same: a
listening Runner that receives nothing from its controller for 90
seconds closes that connection and accepts the next one.

**Steps to reproduce**

1. Start a Runner in listener mode and connect a controller.
2. Break the path between the controller and the Runner without a TCP
close (for example, a proxy that keeps the sandbox-side socket open).
3. Connect a second controller. See that the Runner never accepts it.

**Paperclip version or commit**

master `ca34aca777` (wave) and `f5f60cfcb2`.

## What Changed

- `AuthenticatedTransport` records when it last read any frame (text,
ping, or pong).
- On a listener-mode transport, a control read returns a transport error
when no frame arrived within `DurableRunnerConfig::controller_silence`
(default 90 s). The Runner then takes its existing disconnect path: it
records a diagnostic, closes the socket, and accepts the next controller
within its reconnect grace. A dial-mode transport is unchanged.
- The silence rule applies only to steady-state control reads
(`receive_json` and `receive_queued_json`). The welcome read during
connection setup keeps its own deadline and timeout message, and the
silence window restarts once the welcome validates, so a slow welcome is
never reported as controller silence (which, after `auth_response`,
would fail a bootstrap closed).
- `DurableRunnerConfig::validate` rejects a `controller_silence` outside
100 ms to one hour, so a zero window cannot cause a tight reconnect
loop.
- `DURABLE_TRANSPORT.md` states the rule.

## Verification

- New test
`listener_accepts_a_new_controller_after_the_current_one_goes_silent`
runs the real durable Runner loop in listener mode. An authenticated
controller goes silent. A replacement controller authenticates with the
lease and sends `runner.shutdown`. The Runner executes it once and exits
cleanly. Red: with the silence branch disabled, the replacement is never
accepted and the test fails. Green with the fix.
- New test `listener_controller_pings_keep_an_idle_connection`: a
controller that sends only WebSocket pings (every 400 ms, 1 s window,
2.5 s run) never triggers the silence error. Red when `last_inbound` is
refreshed only on text frames.
- New test
`listener_welcome_timeout_is_not_reported_as_controller_silence`: with a
100 ms window and a lease-authenticated controller that never sends
welcome, the failure is the welcome/auth deadline, not the liveness
error, and stays retryable (`bootstrap_maybe_consumed` false). Red when
the welcome read goes through the silence-converting reader.
- New test `controller_silence_must_be_a_plausible_window` covers the
validate bounds.
- `cargo test -p paperclip-runner-core --lib -- durable::` passes (133
tests) on `af3c9b3649`, after merging `origin/master`; all runner-core
test targets compile.
- `cargo clippy` reports no new findings in the changed files. The one
clippy error is in `runtime_notifications.rs` on master.
- `pnpm verify` asks for broad CI coverage because Runner core changed.
CI supplies it.

## Risks

- A controller that does not ping (a server older than #19407) and stays
idle for 90 seconds now gets a reconnect. The server reconnect loop
handles this within its grace.
- A server event loop blocked for more than 90 seconds can cause one
reconnect. This is the same tradeoff the server side already makes.
- No change to dial mode, to provider work, or to the work done before
`turn.start`.
- Previous-release story: no persisted run, session or lease state
changes. `controller_silence` is an in-memory config field that
`paperclip-runnerd` sets to the 90 s default, so a Runner resumed from
state saved by the previous release behaves the same apart from the new
liveness rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 00:22:26 -06:00
..