perf(runner): check a listening Runner's backlog every 10 ms, not every reconnect delay (#21821)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote `paperclip_runner` agent runs `paperclip-runnerd` in a
sandbox or VM. A provider ingress proxies Core's WebSocket to a Runner
that listens (`--listen-address`).
> - In listen mode, the Runner checks for an incoming connection only
once per `--reconnect-delay-ms` (default 250 ms). It waits that delay
again after each disconnect, and it backs off after a failed handshake.
> - That delay exists to pace redials in dial mode. A listener never
dials. Core's connection waits in the kernel backlog until the next
`accept`.
> - Every listen-mode connect pays part of that wait. A warm run
attachment pays it in full: the Runner disconnects at the authority
rotation, and Core's new-authority connect then waits up to 250 ms or
more.
> - This pull request makes a listening Runner check for a connection
every 10 ms. Dial mode keeps the configured reconnect delay.
> - The benefit is about 0.4 s less on each warm run attachment and
about 0.1 s less on each cold start, for every listen-mode Runner.

## Linked Issues or Issue Description

**What happened?**
A warm follow-up turn on a listen-mode remote Runner reached its first
turn later than a cold turn on the same agent. In traces, a cold start
makes one Core→Runner connect of 134–178 ms. Each warm attachment makes
an old-authority reconnect (72–233 ms) and then a new-authority connect
of 411–504 ms. This was 6 of 6 warm runs.

**Expected behavior**
A connect to a listening Runner completes in about one network round
trip plus the WebSocket handshake. It does not wait for a reconnect
timer.

**Steps to reproduce**
1. Start `paperclip-runnerd` in listen mode with the default
`--reconnect-delay-ms`.
2. Let it reach the disconnected wait (startup, a park claim, or a
`run.attach` authority rotation).
3. Connect to the listener. The upgrade is not answered until the next
250 ms check.

**Root cause**
`durable/runner.rs` schedules the next `accept` at `now +
config.reconnect_delay` when no connection is pending, after a
disconnect, and as the base of the failure backoff. The Launcher-style
hosts do not pass `--reconnect-delay-ms`, so the 250 ms default applies.
The tests use a 1 ms delay, so they did not show the wait.

## What Changed

- `durable/transport.rs`: add `RunnerTransportEndpoint::retry_delay()`.
A listener returns `min(configured, 10 ms)`. A dial endpoint returns the
configured delay without change.
- `durable/runner.rs`: the four places that waited
`config.reconnect_delay` (no pending connection, failure backoff base,
two disconnect paths) now use
`endpoint.retry_delay(config.reconnect_delay)`.
- `DURABLE_TRANSPORT.md`: document the listen-mode connection check
interval.
- Add a unit test for the listener and dial delays.
- Add `tests/listener_accept.rs`. It starts `paperclip-runnerd` in
listen mode with the default reconnect delay. One connection drops
before authentication, as a superseded route's redial does. The test
then checks that the next upgrade is answered in under 120 ms. Before
this change, that upgrade waited 188–313 ms.

## Verification

- `cargo test -p paperclip-runner-core --lib` passes locally.
- `cargo clippy -p paperclip-runner-core --all-targets` reports no new
findings in the changed files.
- `cargo fmt --all -- --check` passes.
- New tests:
`durable::transport::tests::listener_checks_its_backlog_sooner_than_a_dial_reconnects`
and `tests/listener_accept.rs`.
- `tests/listener_accept.rs` fails on the previous `runner.rs` and
passes with this change (3 runs each).
- Existing listener and warm-attachment tests pass (`listener_*`,
`warm_run_attachment_*`, `warm_external_attachment_*`).
- I did not measure the change on a live host. The effect is visible in
the next deployment as a shorter `runner.transport.registration` on warm
attachments.

## Risks

- Low risk. Dial mode does not change.
- A disconnected listener now wakes every 10 ms instead of every 250 ms.
Each wake is one non-blocking `accept` and a non-blocking host-control
read. The loop already slept in 25 ms steps.
- Failed handshakes on a listener now back off from 10 ms (10, 20, 40 …
320 ms) instead of from 250 ms. Core's own reconnect loop still paces
its redials, and the ingress requires credentials, so this does not
create a hot loop.
- The change needs a Runner release before hosts pick it up.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, trace queries).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-11-15 22:17:52 +00:00
1 parent 304e40fe9c
commit 8299f7c389
4 files changed
+180 -6

No files matched your search

@@ -8,6 +8,12 @@ agent or recovery of its persisted run. On a fresh launch or restart,
- With `--listen-address 0.0.0.0 --listen-path /api/runner/v1/connect/<run-id>`,
the Runner listens for Core. `--listen-port` defaults to `43127`.
A dialing Runner waits `--reconnect-delay-ms` (default 250) between
connection attempts and backs off after a failed one. A listening Runner
checks for an incoming connection every 10 ms (or sooner, when that delay is
smaller). It never redials, so a connection does not wait out the reconnect
delay after a disconnect or an authority rotation.
Choose one form. Both require `PAPERCLIP_RUNNER_BOOTSTRAP_TICKET`.
Recovery uses the same entry point. The Runner rejects `--source-reconcile`.
@@ -730,7 +730,8 @@ pub fn run_durable_runner_with_host_control<E: CommandExecutor>(
let (mut transport, welcome) = match connection {
Ok(Some(connection)) => connection,
Ok(None) => {
next_connection_attempt = Instant::now() + config.reconnect_delay;
next_connection_attempt =
Instant::now() + endpoint.retry_delay(config.reconnect_delay);
continue;
}
Err(error) => {
@@ -749,7 +750,7 @@ pub fn run_durable_runner_with_host_control<E: CommandExecutor>(
}
next_connection_attempt = Instant::now()
+ reconnect_delay(
config.reconnect_delay,
endpoint.retry_delay(config.reconnect_delay),
connect_deadline.saturating_duration_since(Instant::now()),
&mut reconnect_attempt,
);
@@ -1055,8 +1056,8 @@ pub fn run_durable_runner_with_host_control<E: CommandExecutor>(
.map(|credential| credential.expires_at_unix_ms),
);
next_connection_attempt = Instant::now()
+ config
.reconnect_delay
+ endpoint
.retry_delay(config.reconnect_delay)
.min(reconnect_deadline.saturating_duration_since(Instant::now()));
continue;
}
@@ -1419,8 +1420,8 @@ pub fn run_durable_runner_with_host_control<E: CommandExecutor>(
.map(|credential| credential.expires_at_unix_ms),
);
next_connection_attempt = Instant::now()
+ config
.reconnect_delay
+ endpoint
.retry_delay(config.reconnect_delay)
.min(reconnect_deadline.saturating_duration_since(Instant::now()));
}
}
@@ -32,6 +32,7 @@ const QUEUED_CONTROL_READ_TIMEOUT: Duration = Duration::from_millis(1);
const CONNECT_ATTEMPT_TIMEOUT: Duration = Duration::from_secs(3);
const CONNECT_TOTAL_TIMEOUT: Duration = Duration::from_secs(10);
const MAX_CA_BUNDLE_BYTES: u64 = 4 * 1024 * 1024;
const LISTENER_ACCEPT_INTERVAL: Duration = Duration::from_millis(10);
type HmacSha256 = Hmac<Sha256>;
@@ -252,6 +253,13 @@ impl RunnerTransportEndpoint {
Ok(())
}
pub(crate) fn retry_delay(&self, configured: Duration) -> Duration {
match self {
Self::Dial(_) => configured,
Self::Listen { .. } => configured.min(LISTENER_ACCEPT_INTERVAL),
}
}
fn open(
&self,
max_frame_bytes: usize,
@@ -2172,6 +2180,27 @@ mod tests {
}
}
#[test]
fn listener_checks_its_backlog_sooner_than_a_dial_reconnects() {
let configured = Duration::from_millis(250);
let listener = RunnerTransportEndpoint::Listen {
listener: TcpListener::bind("0.0.0.0:0").unwrap(),
path: "/api/runner/v1/connect/run_1".to_owned(),
};
assert_eq!(listener.retry_delay(configured), LISTENER_ACCEPT_INTERVAL);
assert!(LISTENER_ACCEPT_INTERVAL <= Duration::from_millis(25));
assert_eq!(
listener.retry_delay(Duration::from_millis(1)),
Duration::from_millis(1)
);
let dial = RunnerTransportEndpoint::new(
"ws://127.0.0.1:43127/api/runner/v1/connect/run_1",
"run_1",
)
.unwrap();
assert_eq!(dial.retry_delay(configured), configured);
}
#[test]
fn listener_path_is_exact_and_unambiguous() {
validate_listener_path("/api/runner/v1/connect/run_1").unwrap();
@@ -0,0 +1,138 @@
#![cfg(unix)]
use std::fs;
use std::io::{Read, Write};
use std::net::{TcpListener, TcpStream};
use std::os::unix::fs::PermissionsExt;
use std::path::PathBuf;
use std::process::{Child, Command, Stdio};
use std::thread;
use std::time::{Duration, Instant};
struct Runner {
process: Child,
directory: PathBuf,
}
impl Drop for Runner {
fn drop(&mut self) {
let _ = self.process.kill();
let _ = self.process.wait();
let _ = fs::remove_dir_all(&self.directory);
}
}
const RUN_ID: &str = "run_accept";
fn upgrade(stream: &mut TcpStream) -> Duration {
let started = Instant::now();
stream
.write_all(
format!(
"GET /api/runner/v1/connect/{RUN_ID} HTTP/1.1\r\nHost: 127.0.0.1\r\n\
Upgrade: websocket\r\nConnection: Upgrade\r\n\
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\nSec-WebSocket-Version: 13\r\n\r\n"
)
.as_bytes(),
)
.unwrap();
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut response = Vec::new();
let mut byte = [0_u8; 1];
while !response.ends_with(b"\r\n\r\n") {
assert_eq!(
stream.read(&mut byte).unwrap(),
1,
"Runner closed the upgrade"
);
response.push(byte[0]);
}
assert!(
response.starts_with(b"HTTP/1.1 101"),
"{}",
String::from_utf8_lossy(&response)
);
started.elapsed()
}
fn spawn_listening_runner() -> (Runner, u16, TcpStream) {
for _ in 0..10 {
let reservation = TcpListener::bind("0.0.0.0:0").unwrap();
let port = reservation.local_addr().unwrap().port();
drop(reservation);
let directory =
std::env::temp_dir().join(format!("runner-accept-{}", uuid::Uuid::new_v4()));
fs::create_dir(&directory).unwrap();
fs::set_permissions(&directory, fs::Permissions::from_mode(0o700)).unwrap();
let process = Command::new(env!("CARGO_BIN_EXE_paperclip-runnerd"))
.env(
"PAPERCLIP_RUNNER_BOOTSTRAP_TICKET",
"fixture-bootstrap-ticket",
)
.args([
"--listen-address",
"0.0.0.0",
"--listen-port",
&port.to_string(),
"--listen-path",
&format!("/api/runner/v1/connect/{RUN_ID}"),
"--state-dir",
directory.to_str().unwrap(),
"--runner-id",
"runner",
"--environment-lease-id",
"lease",
"--run-id",
RUN_ID,
"--session-id",
"session",
"--turn-id",
"turn",
"--item-id",
"item",
"--runner-version",
"0.0.0",
"--runner-digest",
&format!("sha256:{}", "a".repeat(64)),
"--max-runtime-ms",
"20000",
])
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn()
.unwrap();
let mut runner = Runner { process, directory };
let deadline = Instant::now() + Duration::from_secs(5);
loop {
if let Some(status) = runner.process.try_wait().unwrap() {
match TcpListener::bind(("0.0.0.0", port)) {
Err(error) if error.kind() == std::io::ErrorKind::AddrInUse => break,
_ => panic!("Runner exited before listening: {status}"),
}
}
if let Ok(stream) = TcpStream::connect(("127.0.0.1", port)) {
return (runner, port, stream);
}
assert!(Instant::now() < deadline, "Runner did not bind its port");
thread::sleep(Duration::from_millis(10));
}
}
panic!("could not acquire an available Runner port");
}
#[test]
fn listener_answers_the_next_connection_without_waiting_out_the_reconnect_delay() {
let (_runner, port, mut stale) = spawn_listening_runner();
upgrade(&mut stale);
drop(stale);
thread::sleep(Duration::from_millis(30));
let mut current = TcpStream::connect(("127.0.0.1", port)).unwrap();
let waited = upgrade(&mut current);
assert!(
waited < Duration::from_millis(120),
"listener answered after {waited:?}"
);
}