mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 23:08:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Daytona, the Paperclip server connects to a Runner that listens inside the sandbox, through the Daytona proxy > - A listening Runner accepts the next controller connection only after its current connection closes > - When the network path broke without a close, the Runner kept a dead socket and never accepted the new server after a release swap > - This pull request makes a listening Runner treat a controller that sends nothing for 90 seconds as gone, and listen again > - The benefit is that live runs survive a network outage that ends near a release swap, instead of losing their session and restarting fresh ## Linked Issues or Issue Description **What happened?** In a restart-chaos wave (201 tasks, 6 graceful release swaps), the host could not resolve Daytona hosts for about 12 minutes. Swap 4 happened at the end of that outage. The new server tried to reattach to 16 live runs. It reached none of them: all 16 were given up after 66 to 75 seconds (`reattach phase ended`, `authenticated: 0`, `givenUp: 16`). Each run then failed its three same-run resumes with `runner_ingress_unavailable` and continued in a fresh run. Six tasks showed a brief Retry banner, and the fresh runs repeated work. A sandbox snapshot taken 11 seconds after the swap shows the cause. The Runner process was alive. Its listening socket had 24 connections in the accept backlog and about 25 `CLOSE-WAIT` connections with an unread upgrade request. The Runner held one established connection from the Daytona proxy, with empty queues. Its saved state showed `reconnectCount: 0`, and no run event had arrived since the outage started. The Runner was waiting on a dead connection and did not call `accept` again. **Expected behavior** The server already pings the Runner every 20 seconds and drops a silent Runner after 90 seconds (#19407). The Runner side must do the same: a listening Runner that receives nothing from its controller for 90 seconds closes that connection and accepts the next one. **Steps to reproduce** 1. Start a Runner in listener mode and connect a controller. 2. Break the path between the controller and the Runner without a TCP close (for example, a proxy that keeps the sandbox-side socket open). 3. Connect a second controller. See that the Runner never accepts it. **Paperclip version or commit** master `ca34aca777` (wave) and `f5f60cfcb2`. ## What Changed - `AuthenticatedTransport` records when it last read any frame (text, ping, or pong). - On a listener-mode transport, a control read returns a transport error when no frame arrived within `DurableRunnerConfig::controller_silence` (default 90 s). The Runner then takes its existing disconnect path: it records a diagnostic, closes the socket, and accepts the next controller within its reconnect grace. A dial-mode transport is unchanged. - The silence rule applies only to steady-state control reads (`receive_json` and `receive_queued_json`). The welcome read during connection setup keeps its own deadline and timeout message, and the silence window restarts once the welcome validates, so a slow welcome is never reported as controller silence (which, after `auth_response`, would fail a bootstrap closed). - `DurableRunnerConfig::validate` rejects a `controller_silence` outside 100 ms to one hour, so a zero window cannot cause a tight reconnect loop. - `DURABLE_TRANSPORT.md` states the rule. ## Verification - New test `listener_accepts_a_new_controller_after_the_current_one_goes_silent` runs the real durable Runner loop in listener mode. An authenticated controller goes silent. A replacement controller authenticates with the lease and sends `runner.shutdown`. The Runner executes it once and exits cleanly. Red: with the silence branch disabled, the replacement is never accepted and the test fails. Green with the fix. - New test `listener_controller_pings_keep_an_idle_connection`: a controller that sends only WebSocket pings (every 400 ms, 1 s window, 2.5 s run) never triggers the silence error. Red when `last_inbound` is refreshed only on text frames. - New test `listener_welcome_timeout_is_not_reported_as_controller_silence`: with a 100 ms window and a lease-authenticated controller that never sends welcome, the failure is the welcome/auth deadline, not the liveness error, and stays retryable (`bootstrap_maybe_consumed` false). Red when the welcome read goes through the silence-converting reader. - New test `controller_silence_must_be_a_plausible_window` covers the validate bounds. - `cargo test -p paperclip-runner-core --lib -- durable::` passes (133 tests) on `af3c9b3649`, after merging `origin/master`; all runner-core test targets compile. - `cargo clippy` reports no new findings in the changed files. The one clippy error is in `runtime_notifications.rs` on master. - `pnpm verify` asks for broad CI coverage because Runner core changed. CI supplies it. ## Risks - A controller that does not ping (a server older than #19407) and stays idle for 90 seconds now gets a reconnect. The server reconnect loop handles this within its grace. - A server event loop blocked for more than 90 seconds can cause one reconnect. This is the same tradeoff the server side already makes. - No change to dial mode, to provider work, or to the work done before `turn.start`. - Previous-release story: no persisted run, session or lease state changes. `controller_silence` is an in-memory config field that `paperclip-runnerd` sets to the 90 s default, so a Runner resumed from state saved by the previous release behaves the same apart from the new liveness rule. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>