mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
perf(runner): check a listening Runner's backlog every 10 ms, not every reconnect delay (#21821)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote `paperclip_runner` agent runs `paperclip-runnerd` in a sandbox or VM. A provider ingress proxies Core's WebSocket to a Runner that listens (`--listen-address`). > - In listen mode, the Runner checks for an incoming connection only once per `--reconnect-delay-ms` (default 250 ms). It waits that delay again after each disconnect, and it backs off after a failed handshake. > - That delay exists to pace redials in dial mode. A listener never dials. Core's connection waits in the kernel backlog until the next `accept`. > - Every listen-mode connect pays part of that wait. A warm run attachment pays it in full: the Runner disconnects at the authority rotation, and Core's new-authority connect then waits up to 250 ms or more. > - This pull request makes a listening Runner check for a connection every 10 ms. Dial mode keeps the configured reconnect delay. > - The benefit is about 0.4 s less on each warm run attachment and about 0.1 s less on each cold start, for every listen-mode Runner. ## Linked Issues or Issue Description **What happened?** A warm follow-up turn on a listen-mode remote Runner reached its first turn later than a cold turn on the same agent. In traces, a cold start makes one Core→Runner connect of 134–178 ms. Each warm attachment makes an old-authority reconnect (72–233 ms) and then a new-authority connect of 411–504 ms. This was 6 of 6 warm runs. **Expected behavior** A connect to a listening Runner completes in about one network round trip plus the WebSocket handshake. It does not wait for a reconnect timer. **Steps to reproduce** 1. Start `paperclip-runnerd` in listen mode with the default `--reconnect-delay-ms`. 2. Let it reach the disconnected wait (startup, a park claim, or a `run.attach` authority rotation). 3. Connect to the listener. The upgrade is not answered until the next 250 ms check. **Root cause** `durable/runner.rs` schedules the next `accept` at `now + config.reconnect_delay` when no connection is pending, after a disconnect, and as the base of the failure backoff. The Launcher-style hosts do not pass `--reconnect-delay-ms`, so the 250 ms default applies. The tests use a 1 ms delay, so they did not show the wait. ## What Changed - `durable/transport.rs`: add `RunnerTransportEndpoint::retry_delay()`. A listener returns `min(configured, 10 ms)`. A dial endpoint returns the configured delay without change. - `durable/runner.rs`: the four places that waited `config.reconnect_delay` (no pending connection, failure backoff base, two disconnect paths) now use `endpoint.retry_delay(config.reconnect_delay)`. - `DURABLE_TRANSPORT.md`: document the listen-mode connection check interval. - Add a unit test for the listener and dial delays. - Add `tests/listener_accept.rs`. It starts `paperclip-runnerd` in listen mode with the default reconnect delay. One connection drops before authentication, as a superseded route's redial does. The test then checks that the next upgrade is answered in under 120 ms. Before this change, that upgrade waited 188–313 ms. ## Verification - `cargo test -p paperclip-runner-core --lib` passes locally. - `cargo clippy -p paperclip-runner-core --all-targets` reports no new findings in the changed files. - `cargo fmt --all -- --check` passes. - New tests: `durable::transport::tests::listener_checks_its_backlog_sooner_than_a_dial_reconnects` and `tests/listener_accept.rs`. - `tests/listener_accept.rs` fails on the previous `runner.rs` and passes with this change (3 runs each). - Existing listener and warm-attachment tests pass (`listener_*`, `warm_run_attachment_*`, `warm_external_attachment_*`). - I did not measure the change on a live host. The effect is visible in the next deployment as a shorter `runner.transport.registration` on warm attachments. ## Risks - Low risk. Dial mode does not change. - A disconnected listener now wakes every 10 ms instead of every 250 ms. Each wake is one non-blocking `accept` and a non-blocking host-control read. The loop already slept in 25 ms steps. - Failed handshakes on a listener now back off from 10 ms (10, 20, 40 … 320 ms) instead of from 250 ms. Core's own reconnect loop still paces its redials, and the ingress requires credentials, so this does not create a hot loop. - The change needs a Runner release before hosts pick it up. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, trace queries). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
4 files changed
+180
-6
No files matched your search
@@ -8,6 +8,12 @@ agent or recovery of its persisted run. On a fresh launch or restart,
|
||||
- With `--listen-address 0.0.0.0 --listen-path /api/runner/v1/connect/<run-id>`,
|
||||
the Runner listens for Core. `--listen-port` defaults to `43127`.
|
||||
|
||||
A dialing Runner waits `--reconnect-delay-ms` (default 250) between
|
||||
connection attempts and backs off after a failed one. A listening Runner
|
||||
checks for an incoming connection every 10 ms (or sooner, when that delay is
|
||||
smaller). It never redials, so a connection does not wait out the reconnect
|
||||
delay after a disconnect or an authority rotation.
|
||||
|
||||
Choose one form. Both require `PAPERCLIP_RUNNER_BOOTSTRAP_TICKET`.
|
||||
Recovery uses the same entry point. The Runner rejects `--source-reconcile`.
|
||||
|
||||
|
||||
@@ -730,7 +730,8 @@ pub fn run_durable_runner_with_host_control<E: CommandExecutor>(
|
||||
let (mut transport, welcome) = match connection {
|
||||
Ok(Some(connection)) => connection,
|
||||
Ok(None) => {
|
||||
next_connection_attempt = Instant::now() + config.reconnect_delay;
|
||||
next_connection_attempt =
|
||||
Instant::now() + endpoint.retry_delay(config.reconnect_delay);
|
||||
continue;
|
||||
}
|
||||
Err(error) => {
|
||||
@@ -749,7 +750,7 @@ pub fn run_durable_runner_with_host_control<E: CommandExecutor>(
|
||||
}
|
||||
next_connection_attempt = Instant::now()
|
||||
+ reconnect_delay(
|
||||
config.reconnect_delay,
|
||||
endpoint.retry_delay(config.reconnect_delay),
|
||||
connect_deadline.saturating_duration_since(Instant::now()),
|
||||
&mut reconnect_attempt,
|
||||
);
|
||||
@@ -1055,8 +1056,8 @@ pub fn run_durable_runner_with_host_control<E: CommandExecutor>(
|
||||
.map(|credential| credential.expires_at_unix_ms),
|
||||
);
|
||||
next_connection_attempt = Instant::now()
|
||||
+ config
|
||||
.reconnect_delay
|
||||
+ endpoint
|
||||
.retry_delay(config.reconnect_delay)
|
||||
.min(reconnect_deadline.saturating_duration_since(Instant::now()));
|
||||
continue;
|
||||
}
|
||||
@@ -1419,8 +1420,8 @@ pub fn run_durable_runner_with_host_control<E: CommandExecutor>(
|
||||
.map(|credential| credential.expires_at_unix_ms),
|
||||
);
|
||||
next_connection_attempt = Instant::now()
|
||||
+ config
|
||||
.reconnect_delay
|
||||
+ endpoint
|
||||
.retry_delay(config.reconnect_delay)
|
||||
.min(reconnect_deadline.saturating_duration_since(Instant::now()));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,6 +32,7 @@ const QUEUED_CONTROL_READ_TIMEOUT: Duration = Duration::from_millis(1);
|
||||
const CONNECT_ATTEMPT_TIMEOUT: Duration = Duration::from_secs(3);
|
||||
const CONNECT_TOTAL_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
const MAX_CA_BUNDLE_BYTES: u64 = 4 * 1024 * 1024;
|
||||
const LISTENER_ACCEPT_INTERVAL: Duration = Duration::from_millis(10);
|
||||
|
||||
type HmacSha256 = Hmac<Sha256>;
|
||||
|
||||
@@ -252,6 +253,13 @@ impl RunnerTransportEndpoint {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn retry_delay(&self, configured: Duration) -> Duration {
|
||||
match self {
|
||||
Self::Dial(_) => configured,
|
||||
Self::Listen { .. } => configured.min(LISTENER_ACCEPT_INTERVAL),
|
||||
}
|
||||
}
|
||||
|
||||
fn open(
|
||||
&self,
|
||||
max_frame_bytes: usize,
|
||||
@@ -2172,6 +2180,27 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn listener_checks_its_backlog_sooner_than_a_dial_reconnects() {
|
||||
let configured = Duration::from_millis(250);
|
||||
let listener = RunnerTransportEndpoint::Listen {
|
||||
listener: TcpListener::bind("0.0.0.0:0").unwrap(),
|
||||
path: "/api/runner/v1/connect/run_1".to_owned(),
|
||||
};
|
||||
assert_eq!(listener.retry_delay(configured), LISTENER_ACCEPT_INTERVAL);
|
||||
assert!(LISTENER_ACCEPT_INTERVAL <= Duration::from_millis(25));
|
||||
assert_eq!(
|
||||
listener.retry_delay(Duration::from_millis(1)),
|
||||
Duration::from_millis(1)
|
||||
);
|
||||
let dial = RunnerTransportEndpoint::new(
|
||||
"ws://127.0.0.1:43127/api/runner/v1/connect/run_1",
|
||||
"run_1",
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(dial.retry_delay(configured), configured);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn listener_path_is_exact_and_unambiguous() {
|
||||
validate_listener_path("/api/runner/v1/connect/run_1").unwrap();
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
#![cfg(unix)]
|
||||
|
||||
use std::fs;
|
||||
use std::io::{Read, Write};
|
||||
use std::net::{TcpListener, TcpStream};
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
use std::path::PathBuf;
|
||||
use std::process::{Child, Command, Stdio};
|
||||
use std::thread;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
struct Runner {
|
||||
process: Child,
|
||||
directory: PathBuf,
|
||||
}
|
||||
|
||||
impl Drop for Runner {
|
||||
fn drop(&mut self) {
|
||||
let _ = self.process.kill();
|
||||
let _ = self.process.wait();
|
||||
let _ = fs::remove_dir_all(&self.directory);
|
||||
}
|
||||
}
|
||||
|
||||
const RUN_ID: &str = "run_accept";
|
||||
|
||||
fn upgrade(stream: &mut TcpStream) -> Duration {
|
||||
let started = Instant::now();
|
||||
stream
|
||||
.write_all(
|
||||
format!(
|
||||
"GET /api/runner/v1/connect/{RUN_ID} HTTP/1.1\r\nHost: 127.0.0.1\r\n\
|
||||
Upgrade: websocket\r\nConnection: Upgrade\r\n\
|
||||
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\nSec-WebSocket-Version: 13\r\n\r\n"
|
||||
)
|
||||
.as_bytes(),
|
||||
)
|
||||
.unwrap();
|
||||
stream
|
||||
.set_read_timeout(Some(Duration::from_secs(5)))
|
||||
.unwrap();
|
||||
let mut response = Vec::new();
|
||||
let mut byte = [0_u8; 1];
|
||||
while !response.ends_with(b"\r\n\r\n") {
|
||||
assert_eq!(
|
||||
stream.read(&mut byte).unwrap(),
|
||||
1,
|
||||
"Runner closed the upgrade"
|
||||
);
|
||||
response.push(byte[0]);
|
||||
}
|
||||
assert!(
|
||||
response.starts_with(b"HTTP/1.1 101"),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&response)
|
||||
);
|
||||
started.elapsed()
|
||||
}
|
||||
|
||||
fn spawn_listening_runner() -> (Runner, u16, TcpStream) {
|
||||
for _ in 0..10 {
|
||||
let reservation = TcpListener::bind("0.0.0.0:0").unwrap();
|
||||
let port = reservation.local_addr().unwrap().port();
|
||||
drop(reservation);
|
||||
let directory =
|
||||
std::env::temp_dir().join(format!("runner-accept-{}", uuid::Uuid::new_v4()));
|
||||
fs::create_dir(&directory).unwrap();
|
||||
fs::set_permissions(&directory, fs::Permissions::from_mode(0o700)).unwrap();
|
||||
let process = Command::new(env!("CARGO_BIN_EXE_paperclip-runnerd"))
|
||||
.env(
|
||||
"PAPERCLIP_RUNNER_BOOTSTRAP_TICKET",
|
||||
"fixture-bootstrap-ticket",
|
||||
)
|
||||
.args([
|
||||
"--listen-address",
|
||||
"0.0.0.0",
|
||||
"--listen-port",
|
||||
&port.to_string(),
|
||||
"--listen-path",
|
||||
&format!("/api/runner/v1/connect/{RUN_ID}"),
|
||||
"--state-dir",
|
||||
directory.to_str().unwrap(),
|
||||
"--runner-id",
|
||||
"runner",
|
||||
"--environment-lease-id",
|
||||
"lease",
|
||||
"--run-id",
|
||||
RUN_ID,
|
||||
"--session-id",
|
||||
"session",
|
||||
"--turn-id",
|
||||
"turn",
|
||||
"--item-id",
|
||||
"item",
|
||||
"--runner-version",
|
||||
"0.0.0",
|
||||
"--runner-digest",
|
||||
&format!("sha256:{}", "a".repeat(64)),
|
||||
"--max-runtime-ms",
|
||||
"20000",
|
||||
])
|
||||
.stdout(Stdio::null())
|
||||
.stderr(Stdio::null())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let mut runner = Runner { process, directory };
|
||||
let deadline = Instant::now() + Duration::from_secs(5);
|
||||
loop {
|
||||
if let Some(status) = runner.process.try_wait().unwrap() {
|
||||
match TcpListener::bind(("0.0.0.0", port)) {
|
||||
Err(error) if error.kind() == std::io::ErrorKind::AddrInUse => break,
|
||||
_ => panic!("Runner exited before listening: {status}"),
|
||||
}
|
||||
}
|
||||
if let Ok(stream) = TcpStream::connect(("127.0.0.1", port)) {
|
||||
return (runner, port, stream);
|
||||
}
|
||||
assert!(Instant::now() < deadline, "Runner did not bind its port");
|
||||
thread::sleep(Duration::from_millis(10));
|
||||
}
|
||||
}
|
||||
panic!("could not acquire an available Runner port");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn listener_answers_the_next_connection_without_waiting_out_the_reconnect_delay() {
|
||||
let (_runner, port, mut stale) = spawn_listening_runner();
|
||||
upgrade(&mut stale);
|
||||
drop(stale);
|
||||
thread::sleep(Duration::from_millis(30));
|
||||
|
||||
let mut current = TcpStream::connect(("127.0.0.1", port)).unwrap();
|
||||
let waited = upgrade(&mut current);
|
||||
assert!(
|
||||
waited < Duration::from_millis(120),
|
||||
"listener answered after {waited:?}"
|
||||
);
|
||||
}
|
||||
Reference in new issue
Block a user