From 8299f7c38956db97e6a4a538770bd2ecbae13cda Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Sun, 15 Nov 2026 14:17:52 -0800 Subject: [PATCH] perf(runner): check a listening Runner's backlog every 10 ms, not every reconnect delay (#21821) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote `paperclip_runner` agent runs `paperclip-runnerd` in a sandbox or VM. A provider ingress proxies Core's WebSocket to a Runner that listens (`--listen-address`). > - In listen mode, the Runner checks for an incoming connection only once per `--reconnect-delay-ms` (default 250 ms). It waits that delay again after each disconnect, and it backs off after a failed handshake. > - That delay exists to pace redials in dial mode. A listener never dials. Core's connection waits in the kernel backlog until the next `accept`. > - Every listen-mode connect pays part of that wait. A warm run attachment pays it in full: the Runner disconnects at the authority rotation, and Core's new-authority connect then waits up to 250 ms or more. > - This pull request makes a listening Runner check for a connection every 10 ms. Dial mode keeps the configured reconnect delay. > - The benefit is about 0.4 s less on each warm run attachment and about 0.1 s less on each cold start, for every listen-mode Runner. ## Linked Issues or Issue Description **What happened?** A warm follow-up turn on a listen-mode remote Runner reached its first turn later than a cold turn on the same agent. In traces, a cold start makes one Core→Runner connect of 134–178 ms. Each warm attachment makes an old-authority reconnect (72–233 ms) and then a new-authority connect of 411–504 ms. This was 6 of 6 warm runs. **Expected behavior** A connect to a listening Runner completes in about one network round trip plus the WebSocket handshake. It does not wait for a reconnect timer. **Steps to reproduce** 1. Start `paperclip-runnerd` in listen mode with the default `--reconnect-delay-ms`. 2. Let it reach the disconnected wait (startup, a park claim, or a `run.attach` authority rotation). 3. Connect to the listener. The upgrade is not answered until the next 250 ms check. **Root cause** `durable/runner.rs` schedules the next `accept` at `now + config.reconnect_delay` when no connection is pending, after a disconnect, and as the base of the failure backoff. The Launcher-style hosts do not pass `--reconnect-delay-ms`, so the 250 ms default applies. The tests use a 1 ms delay, so they did not show the wait. ## What Changed - `durable/transport.rs`: add `RunnerTransportEndpoint::retry_delay()`. A listener returns `min(configured, 10 ms)`. A dial endpoint returns the configured delay without change. - `durable/runner.rs`: the four places that waited `config.reconnect_delay` (no pending connection, failure backoff base, two disconnect paths) now use `endpoint.retry_delay(config.reconnect_delay)`. - `DURABLE_TRANSPORT.md`: document the listen-mode connection check interval. - Add a unit test for the listener and dial delays. - Add `tests/listener_accept.rs`. It starts `paperclip-runnerd` in listen mode with the default reconnect delay. One connection drops before authentication, as a superseded route's redial does. The test then checks that the next upgrade is answered in under 120 ms. Before this change, that upgrade waited 188–313 ms. ## Verification - `cargo test -p paperclip-runner-core --lib` passes locally. - `cargo clippy -p paperclip-runner-core --all-targets` reports no new findings in the changed files. - `cargo fmt --all -- --check` passes. - New tests: `durable::transport::tests::listener_checks_its_backlog_sooner_than_a_dial_reconnects` and `tests/listener_accept.rs`. - `tests/listener_accept.rs` fails on the previous `runner.rs` and passes with this change (3 runs each). - Existing listener and warm-attachment tests pass (`listener_*`, `warm_run_attachment_*`, `warm_external_attachment_*`). - I did not measure the change on a live host. The effect is visible in the next deployment as a shorter `runner.transport.registration` on warm attachments. ## Risks - Low risk. Dial mode does not change. - A disconnected listener now wakes every 10 ms instead of every 250 ms. Each wake is one non-blocking `accept` and a non-blocking host-control read. The loop already slept in 25 ms steps. - Failed handshakes on a listener now back off from 10 ms (10, 20, 40 … 320 ms) instead of from 250 ms. Core's own reconnect loop still paces its redials, and the ingress requires credentials, so this does not create a hot loop. - The change needs a Runner release before hosts pick it up. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, trace queries). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip Co-authored-by: Claude Opus 5.5 --- .../runner/DURABLE_TRANSPORT.md | 6 + .../crates/runner-core/src/durable/runner.rs | 13 +- .../runner-core/src/durable/transport.rs | 29 ++++ .../runner-core/tests/listener_accept.rs | 138 ++++++++++++++++++ 4 files changed, 180 insertions(+), 6 deletions(-) create mode 100644 packages/paperclip-runner/runner/crates/runner-core/tests/listener_accept.rs diff --git a/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md b/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md index 3b61ad38db9..24d651c0d37 100644 --- a/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md +++ b/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md @@ -8,6 +8,12 @@ agent or recovery of its persisted run. On a fresh launch or restart, - With `--listen-address 0.0.0.0 --listen-path /api/runner/v1/connect/`, the Runner listens for Core. `--listen-port` defaults to `43127`. +A dialing Runner waits `--reconnect-delay-ms` (default 250) between +connection attempts and backs off after a failed one. A listening Runner +checks for an incoming connection every 10 ms (or sooner, when that delay is +smaller). It never redials, so a connection does not wait out the reconnect +delay after a disconnect or an authority rotation. + Choose one form. Both require `PAPERCLIP_RUNNER_BOOTSTRAP_TICKET`. Recovery uses the same entry point. The Runner rejects `--source-reconcile`. diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/durable/runner.rs b/packages/paperclip-runner/runner/crates/runner-core/src/durable/runner.rs index ae1d3bb7aa4..e9c2eafc585 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/durable/runner.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/durable/runner.rs @@ -730,7 +730,8 @@ pub fn run_durable_runner_with_host_control( let (mut transport, welcome) = match connection { Ok(Some(connection)) => connection, Ok(None) => { - next_connection_attempt = Instant::now() + config.reconnect_delay; + next_connection_attempt = + Instant::now() + endpoint.retry_delay(config.reconnect_delay); continue; } Err(error) => { @@ -749,7 +750,7 @@ pub fn run_durable_runner_with_host_control( } next_connection_attempt = Instant::now() + reconnect_delay( - config.reconnect_delay, + endpoint.retry_delay(config.reconnect_delay), connect_deadline.saturating_duration_since(Instant::now()), &mut reconnect_attempt, ); @@ -1055,8 +1056,8 @@ pub fn run_durable_runner_with_host_control( .map(|credential| credential.expires_at_unix_ms), ); next_connection_attempt = Instant::now() - + config - .reconnect_delay + + endpoint + .retry_delay(config.reconnect_delay) .min(reconnect_deadline.saturating_duration_since(Instant::now())); continue; } @@ -1419,8 +1420,8 @@ pub fn run_durable_runner_with_host_control( .map(|credential| credential.expires_at_unix_ms), ); next_connection_attempt = Instant::now() - + config - .reconnect_delay + + endpoint + .retry_delay(config.reconnect_delay) .min(reconnect_deadline.saturating_duration_since(Instant::now())); } } diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/durable/transport.rs b/packages/paperclip-runner/runner/crates/runner-core/src/durable/transport.rs index 8b245375882..578419e8bb4 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/durable/transport.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/durable/transport.rs @@ -32,6 +32,7 @@ const QUEUED_CONTROL_READ_TIMEOUT: Duration = Duration::from_millis(1); const CONNECT_ATTEMPT_TIMEOUT: Duration = Duration::from_secs(3); const CONNECT_TOTAL_TIMEOUT: Duration = Duration::from_secs(10); const MAX_CA_BUNDLE_BYTES: u64 = 4 * 1024 * 1024; +const LISTENER_ACCEPT_INTERVAL: Duration = Duration::from_millis(10); type HmacSha256 = Hmac; @@ -252,6 +253,13 @@ impl RunnerTransportEndpoint { Ok(()) } + pub(crate) fn retry_delay(&self, configured: Duration) -> Duration { + match self { + Self::Dial(_) => configured, + Self::Listen { .. } => configured.min(LISTENER_ACCEPT_INTERVAL), + } + } + fn open( &self, max_frame_bytes: usize, @@ -2172,6 +2180,27 @@ mod tests { } } + #[test] + fn listener_checks_its_backlog_sooner_than_a_dial_reconnects() { + let configured = Duration::from_millis(250); + let listener = RunnerTransportEndpoint::Listen { + listener: TcpListener::bind("0.0.0.0:0").unwrap(), + path: "/api/runner/v1/connect/run_1".to_owned(), + }; + assert_eq!(listener.retry_delay(configured), LISTENER_ACCEPT_INTERVAL); + assert!(LISTENER_ACCEPT_INTERVAL <= Duration::from_millis(25)); + assert_eq!( + listener.retry_delay(Duration::from_millis(1)), + Duration::from_millis(1) + ); + let dial = RunnerTransportEndpoint::new( + "ws://127.0.0.1:43127/api/runner/v1/connect/run_1", + "run_1", + ) + .unwrap(); + assert_eq!(dial.retry_delay(configured), configured); + } + #[test] fn listener_path_is_exact_and_unambiguous() { validate_listener_path("/api/runner/v1/connect/run_1").unwrap(); diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/listener_accept.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/listener_accept.rs new file mode 100644 index 00000000000..5af4fc24bf8 --- /dev/null +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/listener_accept.rs @@ -0,0 +1,138 @@ +#![cfg(unix)] + +use std::fs; +use std::io::{Read, Write}; +use std::net::{TcpListener, TcpStream}; +use std::os::unix::fs::PermissionsExt; +use std::path::PathBuf; +use std::process::{Child, Command, Stdio}; +use std::thread; +use std::time::{Duration, Instant}; + +struct Runner { + process: Child, + directory: PathBuf, +} + +impl Drop for Runner { + fn drop(&mut self) { + let _ = self.process.kill(); + let _ = self.process.wait(); + let _ = fs::remove_dir_all(&self.directory); + } +} + +const RUN_ID: &str = "run_accept"; + +fn upgrade(stream: &mut TcpStream) -> Duration { + let started = Instant::now(); + stream + .write_all( + format!( + "GET /api/runner/v1/connect/{RUN_ID} HTTP/1.1\r\nHost: 127.0.0.1\r\n\ + Upgrade: websocket\r\nConnection: Upgrade\r\n\ + Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\nSec-WebSocket-Version: 13\r\n\r\n" + ) + .as_bytes(), + ) + .unwrap(); + stream + .set_read_timeout(Some(Duration::from_secs(5))) + .unwrap(); + let mut response = Vec::new(); + let mut byte = [0_u8; 1]; + while !response.ends_with(b"\r\n\r\n") { + assert_eq!( + stream.read(&mut byte).unwrap(), + 1, + "Runner closed the upgrade" + ); + response.push(byte[0]); + } + assert!( + response.starts_with(b"HTTP/1.1 101"), + "{}", + String::from_utf8_lossy(&response) + ); + started.elapsed() +} + +fn spawn_listening_runner() -> (Runner, u16, TcpStream) { + for _ in 0..10 { + let reservation = TcpListener::bind("0.0.0.0:0").unwrap(); + let port = reservation.local_addr().unwrap().port(); + drop(reservation); + let directory = + std::env::temp_dir().join(format!("runner-accept-{}", uuid::Uuid::new_v4())); + fs::create_dir(&directory).unwrap(); + fs::set_permissions(&directory, fs::Permissions::from_mode(0o700)).unwrap(); + let process = Command::new(env!("CARGO_BIN_EXE_paperclip-runnerd")) + .env( + "PAPERCLIP_RUNNER_BOOTSTRAP_TICKET", + "fixture-bootstrap-ticket", + ) + .args([ + "--listen-address", + "0.0.0.0", + "--listen-port", + &port.to_string(), + "--listen-path", + &format!("/api/runner/v1/connect/{RUN_ID}"), + "--state-dir", + directory.to_str().unwrap(), + "--runner-id", + "runner", + "--environment-lease-id", + "lease", + "--run-id", + RUN_ID, + "--session-id", + "session", + "--turn-id", + "turn", + "--item-id", + "item", + "--runner-version", + "0.0.0", + "--runner-digest", + &format!("sha256:{}", "a".repeat(64)), + "--max-runtime-ms", + "20000", + ]) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + .unwrap(); + let mut runner = Runner { process, directory }; + let deadline = Instant::now() + Duration::from_secs(5); + loop { + if let Some(status) = runner.process.try_wait().unwrap() { + match TcpListener::bind(("0.0.0.0", port)) { + Err(error) if error.kind() == std::io::ErrorKind::AddrInUse => break, + _ => panic!("Runner exited before listening: {status}"), + } + } + if let Ok(stream) = TcpStream::connect(("127.0.0.1", port)) { + return (runner, port, stream); + } + assert!(Instant::now() < deadline, "Runner did not bind its port"); + thread::sleep(Duration::from_millis(10)); + } + } + panic!("could not acquire an available Runner port"); +} + +#[test] +fn listener_answers_the_next_connection_without_waiting_out_the_reconnect_delay() { + let (_runner, port, mut stale) = spawn_listening_runner(); + upgrade(&mut stale); + drop(stale); + thread::sleep(Duration::from_millis(30)); + + let mut current = TcpStream::connect(("127.0.0.1", port)).unwrap(); + let waited = upgrade(&mut current); + assert!( + waited < Duration::from_millis(120), + "listener answered after {waited:?}" + ); +}