mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 16:29:42 +01:00
fix(runner): accept a new controller when the listener's controller goes silent (#22088)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Daytona, the Paperclip server connects to a Runner that listens inside the sandbox, through the Daytona proxy > - A listening Runner accepts the next controller connection only after its current connection closes > - When the network path broke without a close, the Runner kept a dead socket and never accepted the new server after a release swap > - This pull request makes a listening Runner treat a controller that sends nothing for 90 seconds as gone, and listen again > - The benefit is that live runs survive a network outage that ends near a release swap, instead of losing their session and restarting fresh ## Linked Issues or Issue Description **What happened?** In a restart-chaos wave (201 tasks, 6 graceful release swaps), the host could not resolve Daytona hosts for about 12 minutes. Swap 4 happened at the end of that outage. The new server tried to reattach to 16 live runs. It reached none of them: all 16 were given up after 66 to 75 seconds (`reattach phase ended`, `authenticated: 0`, `givenUp: 16`). Each run then failed its three same-run resumes with `runner_ingress_unavailable` and continued in a fresh run. Six tasks showed a brief Retry banner, and the fresh runs repeated work. A sandbox snapshot taken 11 seconds after the swap shows the cause. The Runner process was alive. Its listening socket had 24 connections in the accept backlog and about 25 `CLOSE-WAIT` connections with an unread upgrade request. The Runner held one established connection from the Daytona proxy, with empty queues. Its saved state showed `reconnectCount: 0`, and no run event had arrived since the outage started. The Runner was waiting on a dead connection and did not call `accept` again. **Expected behavior** The server already pings the Runner every 20 seconds and drops a silent Runner after 90 seconds (#19407). The Runner side must do the same: a listening Runner that receives nothing from its controller for 90 seconds closes that connection and accepts the next one. **Steps to reproduce** 1. Start a Runner in listener mode and connect a controller. 2. Break the path between the controller and the Runner without a TCP close (for example, a proxy that keeps the sandbox-side socket open). 3. Connect a second controller. See that the Runner never accepts it. **Paperclip version or commit** master `ca34aca777` (wave) and `f5f60cfcb2`. ## What Changed - `AuthenticatedTransport` records when it last read any frame (text, ping, or pong). - On a listener-mode transport, a control read returns a transport error when no frame arrived within `DurableRunnerConfig::controller_silence` (default 90 s). The Runner then takes its existing disconnect path: it records a diagnostic, closes the socket, and accepts the next controller within its reconnect grace. A dial-mode transport is unchanged. - The silence rule applies only to steady-state control reads (`receive_json` and `receive_queued_json`). The welcome read during connection setup keeps its own deadline and timeout message, and the silence window restarts once the welcome validates, so a slow welcome is never reported as controller silence (which, after `auth_response`, would fail a bootstrap closed). - `DurableRunnerConfig::validate` rejects a `controller_silence` outside 100 ms to one hour, so a zero window cannot cause a tight reconnect loop. - `DURABLE_TRANSPORT.md` states the rule. ## Verification - New test `listener_accepts_a_new_controller_after_the_current_one_goes_silent` runs the real durable Runner loop in listener mode. An authenticated controller goes silent. A replacement controller authenticates with the lease and sends `runner.shutdown`. The Runner executes it once and exits cleanly. Red: with the silence branch disabled, the replacement is never accepted and the test fails. Green with the fix. - New test `listener_controller_pings_keep_an_idle_connection`: a controller that sends only WebSocket pings (every 400 ms, 1 s window, 2.5 s run) never triggers the silence error. Red when `last_inbound` is refreshed only on text frames. - New test `listener_welcome_timeout_is_not_reported_as_controller_silence`: with a 100 ms window and a lease-authenticated controller that never sends welcome, the failure is the welcome/auth deadline, not the liveness error, and stays retryable (`bootstrap_maybe_consumed` false). Red when the welcome read goes through the silence-converting reader. - New test `controller_silence_must_be_a_plausible_window` covers the validate bounds. - `cargo test -p paperclip-runner-core --lib -- durable::` passes (133 tests) on `af3c9b3649`, after merging `origin/master`; all runner-core test targets compile. - `cargo clippy` reports no new findings in the changed files. The one clippy error is in `runtime_notifications.rs` on master. - `pnpm verify` asks for broad CI coverage because Runner core changed. CI supplies it. ## Risks - A controller that does not ping (a server older than #19407) and stays idle for 90 seconds now gets a reconnect. The server reconnect loop handles this within its grace. - A server event loop blocked for more than 90 seconds can cause one reconnect. This is the same tradeoff the server side already makes. - No change to dial mode, to provider work, or to the work done before `turn.start`. - Previous-release story: no persisted run, session or lease state changes. `controller_silence` is an in-memory config field that `paperclip-runnerd` sets to the 90 s default, so a Runner resumed from state saved by the previous release behaves the same apart from the new liveness rule. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
42858fe4e5
commit
9e1a85fb46
20 files changed
+374
-7
No files matched your search
@@ -14,6 +14,12 @@ checks for an incoming connection every 10 ms (or sooner, when that delay is
|
||||
smaller). It never redials, so a connection does not wait out the reconnect
|
||||
delay after a disconnect or an authority rotation.
|
||||
|
||||
A listening Runner accepts one controller at a time. Core pings the Runner
|
||||
every 20 seconds. When the current controller sends no frame, ping included,
|
||||
for 90 seconds, the Runner closes that connection and accepts the next one
|
||||
within its reconnect grace. A path that broke without a close therefore cannot
|
||||
block a replacement controller.
|
||||
|
||||
In both modes, Runner enables `TCP_NODELAY` before the WebSocket handshake
|
||||
when the connected peer is on loopback.
|
||||
|
||||
|
||||
@@ -274,6 +274,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
)),
|
||||
max_runtime: Duration::ZERO,
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
};
|
||||
run_durable_runner(
|
||||
config,
|
||||
|
||||
@@ -3626,6 +3626,7 @@ mod tests {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(60),
|
||||
turn_scoped: false,
|
||||
controller_silence: crate::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -527,6 +527,7 @@ fn run_durable(args: &[String]) -> Result<(), LocalRunnerError> {
|
||||
reconnect_grace: optional_u64(args, "--reconnect-grace-ms")?.map(Duration::from_millis),
|
||||
max_runtime: duration("--max-runtime-ms", 0)?,
|
||||
turn_scoped: turn_scoped(args)?,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
};
|
||||
if host_control.is_some() && !config.connect_url.starts_with("listen://") {
|
||||
return Err(LocalRunnerError::invalid(
|
||||
|
||||
@@ -29,6 +29,7 @@ pub const PROTOCOL: &str = "paperclip.runner";
|
||||
pub const PROTOCOL_MIN_VERSION: u64 = 1;
|
||||
pub const PROTOCOL_VERSION: u64 = 3;
|
||||
pub const BOOTSTRAP_TICKET_ENV: &str = "PAPERCLIP_RUNNER_BOOTSTRAP_TICKET";
|
||||
pub const DEFAULT_CONTROLLER_SILENCE: Duration = Duration::from_secs(90);
|
||||
const MAX_OUTBOX_BYTES: usize = 512 * 1024 * 1024;
|
||||
const MAX_FRAME_BYTES: usize = 16 * 1024 * 1024;
|
||||
|
||||
@@ -259,6 +260,7 @@ pub struct DurableRunnerConfig {
|
||||
/// per_turn`): the process exits with its run, so nothing reuses a provider
|
||||
/// generation prepared after its turn settles.
|
||||
pub turn_scoped: bool,
|
||||
pub controller_silence: Duration,
|
||||
}
|
||||
|
||||
impl DurableRunnerConfig {
|
||||
@@ -314,6 +316,13 @@ impl DurableRunnerConfig {
|
||||
"reconnect delay must be between one millisecond and 60 seconds",
|
||||
));
|
||||
}
|
||||
if self.controller_silence < Duration::from_millis(100)
|
||||
|| self.controller_silence > Duration::from_secs(3600)
|
||||
{
|
||||
return Err(DurableRunnerError::invalid(
|
||||
"controller silence must be between 100 milliseconds and one hour",
|
||||
));
|
||||
}
|
||||
if self.reconnect_grace.is_some_and(|grace| grace.is_zero()) {
|
||||
return Err(DurableRunnerError::invalid(
|
||||
"reconnect grace must be non-zero when configured",
|
||||
@@ -440,6 +449,7 @@ mod tests {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(60),
|
||||
turn_scoped: false,
|
||||
controller_silence: crate::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -3133,6 +3133,7 @@ mod tests {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(1),
|
||||
turn_scoped: false,
|
||||
controller_silence: crate::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2882,6 +2882,7 @@ mod tests {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(1),
|
||||
turn_scoped: false,
|
||||
controller_silence: crate::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+296
@@ -0,0 +1,296 @@
|
||||
#[test]
|
||||
fn listener_accepts_a_new_controller_after_the_current_one_goes_silent() {
|
||||
struct ShutdownOnly(Arc<std::sync::Mutex<Vec<String>>>);
|
||||
|
||||
impl super::super::CommandExecutor for ShutdownOnly {
|
||||
fn execute(
|
||||
&mut self,
|
||||
command: &Command,
|
||||
) -> Result<super::super::CommandExecution, DurableRunnerError> {
|
||||
self.0.lock().unwrap().push(command.command_type.clone());
|
||||
Ok(super::super::CommandExecution::result(
|
||||
json!({"status": "completed"}),
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
let reservation = TcpListener::bind("127.0.0.1:0").unwrap();
|
||||
let address = reservation.local_addr().unwrap();
|
||||
drop(reservation);
|
||||
let path = "/api/runner/v1/connect/run_1";
|
||||
let mut config = config(address.port());
|
||||
config.connect_url = format!("listen://0.0.0.0:{}{path}", address.port());
|
||||
config.max_runtime = Duration::from_secs(10);
|
||||
config.controller_silence = Duration::from_millis(500);
|
||||
let directory = std::env::temp_dir().join(format!(
|
||||
"paperclip-runner-listener-silent-controller-{}",
|
||||
std::process::id()
|
||||
));
|
||||
let _ = std::fs::remove_dir_all(&directory);
|
||||
config.state_dir = directory.clone();
|
||||
let state = test_state(&config);
|
||||
let expires = current_unix_ms().unwrap() + 60_000;
|
||||
let server_config = config.clone();
|
||||
let server_state = state.clone();
|
||||
let (outcome, received) = mpsc::channel();
|
||||
let server = thread::spawn(move || {
|
||||
let open = |deadline: Instant| loop {
|
||||
match TcpStream::connect(address) {
|
||||
Ok(stream) => break stream,
|
||||
Err(_) if Instant::now() < deadline => thread::sleep(Duration::from_millis(10)),
|
||||
Err(error) => panic!("Runner never listened: {error}"),
|
||||
}
|
||||
};
|
||||
let stream = open(Instant::now() + Duration::from_secs(5));
|
||||
let (mut silent, _) = client(format!("ws://{address}{path}"), stream).unwrap();
|
||||
let mut silent_secure = server_authenticate(
|
||||
&mut silent,
|
||||
&server_config,
|
||||
&server_state,
|
||||
ServerCredential {
|
||||
token: "bootstrap-secret",
|
||||
kind: "bootstrap",
|
||||
lease_id: None,
|
||||
expires_at_unix_ms: expires,
|
||||
revocation_epoch: 0,
|
||||
},
|
||||
);
|
||||
send_secure(
|
||||
&mut silent,
|
||||
&mut silent_secure,
|
||||
&server_config,
|
||||
&welcome(&server_state, "connection_1", Some("lease-secret"), expires, 0, vec![]),
|
||||
);
|
||||
let replacement = open(Instant::now() + Duration::from_secs(5));
|
||||
replacement
|
||||
.set_read_timeout(Some(Duration::from_secs(3)))
|
||||
.unwrap();
|
||||
let (mut socket, _) = match client(format!("ws://{address}{path}"), replacement) {
|
||||
Ok(connected) => connected,
|
||||
Err(error) => {
|
||||
outcome.send(format!("replacement was not accepted: {error}")).unwrap();
|
||||
return;
|
||||
}
|
||||
};
|
||||
let mut secure = server_authenticate(
|
||||
&mut socket,
|
||||
&server_config,
|
||||
&server_state,
|
||||
ServerCredential {
|
||||
token: "lease-secret",
|
||||
kind: "lease",
|
||||
lease_id: Some("lease_1"),
|
||||
expires_at_unix_ms: expires,
|
||||
revocation_epoch: 1,
|
||||
},
|
||||
);
|
||||
send_secure(
|
||||
&mut socket,
|
||||
&mut secure,
|
||||
&server_config,
|
||||
&welcome(&server_state, "connection_2", None, expires, 0, vec![]),
|
||||
);
|
||||
send_secure(
|
||||
&mut socket,
|
||||
&mut secure,
|
||||
&server_config,
|
||||
&control(
|
||||
&server_state,
|
||||
"connection_2",
|
||||
"command",
|
||||
json!({
|
||||
"schema": "paperclip.prp.command.v1",
|
||||
"commandId": "command_1",
|
||||
"controllerSeq": 1,
|
||||
"type": "runner.shutdown",
|
||||
"issuedAt": "2026-11-17T19:52:17.000Z",
|
||||
"payload": {},
|
||||
}),
|
||||
),
|
||||
);
|
||||
let result = receive_secure(&mut socket, &mut secure, &server_config);
|
||||
assert_eq!(result["kind"], "command_result");
|
||||
send_secure(
|
||||
&mut socket,
|
||||
&mut secure,
|
||||
&server_config,
|
||||
&control(
|
||||
&server_state,
|
||||
"connection_2",
|
||||
"command_result_ack",
|
||||
json!({
|
||||
"commandId": "command_1",
|
||||
"commandType": "runner.shutdown",
|
||||
"controllerSeq": 1,
|
||||
"status": "completed",
|
||||
}),
|
||||
),
|
||||
);
|
||||
outcome.send("accepted".to_owned()).unwrap();
|
||||
drop(silent);
|
||||
});
|
||||
|
||||
let log = Arc::new(std::sync::Mutex::new(Vec::new()));
|
||||
let run = super::super::run_durable_runner(
|
||||
config,
|
||||
BootstrapTicket::new("bootstrap-secret".to_owned()).unwrap(),
|
||||
ShutdownOnly(log.clone()),
|
||||
);
|
||||
let outcome = received.recv_timeout(Duration::from_secs(10)).unwrap();
|
||||
server.join().unwrap();
|
||||
assert_eq!(outcome, "accepted");
|
||||
run.unwrap();
|
||||
assert_eq!(*log.lock().unwrap(), ["runner.shutdown"]);
|
||||
std::fs::remove_dir_all(directory).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn listener_controller_pings_keep_an_idle_connection() {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
|
||||
listener.set_nonblocking(true).unwrap();
|
||||
let address = listener.local_addr().unwrap();
|
||||
let path = "/api/runner/v1/connect/run_1".to_owned();
|
||||
let endpoint = RunnerTransportEndpoint::Listen {
|
||||
listener,
|
||||
path: path.clone(),
|
||||
};
|
||||
let mut config = config(address.port());
|
||||
config.controller_silence = Duration::from_millis(1_000);
|
||||
let state = test_state(&config);
|
||||
let ticket = BootstrapTicket::new("bootstrap-secret".to_owned()).unwrap();
|
||||
let (ready, connected) = mpsc::channel();
|
||||
let (stop, stopped) = mpsc::channel::<()>();
|
||||
let peer_config = config.clone();
|
||||
let peer_state = state.clone();
|
||||
let peer = thread::spawn(move || {
|
||||
let stream = TcpStream::connect(address).unwrap();
|
||||
ready.send(()).unwrap();
|
||||
let (mut socket, _) = client(format!("ws://{address}{path}"), stream).unwrap();
|
||||
let expires = current_unix_ms().unwrap() + 60_000;
|
||||
let mut secure = server_authenticate(
|
||||
&mut socket,
|
||||
&peer_config,
|
||||
&peer_state,
|
||||
ServerCredential {
|
||||
token: "bootstrap-secret",
|
||||
kind: "bootstrap",
|
||||
lease_id: None,
|
||||
expires_at_unix_ms: expires,
|
||||
revocation_epoch: 0,
|
||||
},
|
||||
);
|
||||
send_secure(
|
||||
&mut socket,
|
||||
&mut secure,
|
||||
&peer_config,
|
||||
&welcome(&peer_state, "connection_1", Some("lease-secret"), expires, 0, vec![]),
|
||||
);
|
||||
while stopped.recv_timeout(Duration::from_millis(400)).is_err() {
|
||||
socket.send(Message::Ping(Vec::new().into())).unwrap();
|
||||
}
|
||||
});
|
||||
connected.recv().unwrap();
|
||||
let (mut transport, _) = AuthenticatedTransport::connect(
|
||||
&endpoint,
|
||||
&config,
|
||||
&state,
|
||||
Some(&ticket),
|
||||
None,
|
||||
Instant::now() + config.max_runtime,
|
||||
)
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
let pinged_until = Instant::now() + Duration::from_millis(2_500);
|
||||
while Instant::now() < pinged_until {
|
||||
assert!(transport.receive_json().unwrap().is_none());
|
||||
}
|
||||
stop.send(()).unwrap();
|
||||
peer.join().unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn listener_welcome_timeout_is_not_reported_as_controller_silence() {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
|
||||
listener.set_nonblocking(true).unwrap();
|
||||
let address = listener.local_addr().unwrap();
|
||||
let path = "/api/runner/v1/connect/run_1".to_owned();
|
||||
let endpoint = RunnerTransportEndpoint::Listen {
|
||||
listener,
|
||||
path: path.clone(),
|
||||
};
|
||||
let mut config = config(address.port());
|
||||
config.controller_silence = Duration::from_millis(100);
|
||||
let state = test_state(&config);
|
||||
let expires = current_unix_ms().unwrap() + 60_000;
|
||||
let lease = LeaseCredential {
|
||||
lease_id: "lease_1".to_owned(),
|
||||
expires_at_unix_ms: expires,
|
||||
revocation_epoch: 1,
|
||||
token: super::super::Secret::new("lease-secret".to_owned()),
|
||||
renewal_requested: false,
|
||||
};
|
||||
let (ready, connected) = mpsc::channel();
|
||||
let (done, finished) = mpsc::channel::<()>();
|
||||
let peer_config = config.clone();
|
||||
let peer_state = state.clone();
|
||||
let peer = thread::spawn(move || {
|
||||
let stream = TcpStream::connect(address).unwrap();
|
||||
ready.send(()).unwrap();
|
||||
let (mut socket, _) = client(format!("ws://{address}{path}"), stream).unwrap();
|
||||
server_authenticate(
|
||||
&mut socket,
|
||||
&peer_config,
|
||||
&peer_state,
|
||||
ServerCredential {
|
||||
token: "lease-secret",
|
||||
kind: "lease",
|
||||
lease_id: Some("lease_1"),
|
||||
expires_at_unix_ms: expires,
|
||||
revocation_epoch: 1,
|
||||
},
|
||||
);
|
||||
finished.recv_timeout(Duration::from_secs(10)).unwrap();
|
||||
});
|
||||
connected.recv().unwrap();
|
||||
let failure = match AuthenticatedTransport::connect(
|
||||
&endpoint,
|
||||
&config,
|
||||
&state,
|
||||
None,
|
||||
Some(&lease),
|
||||
Instant::now() + Duration::from_millis(800),
|
||||
) {
|
||||
Err(failure) => failure,
|
||||
Ok(_) => panic!("connection without welcome unexpectedly authenticated"),
|
||||
};
|
||||
done.send(()).unwrap();
|
||||
peer.join().unwrap();
|
||||
let message = failure.to_string();
|
||||
assert!(!message.contains("liveness window"), "{message}");
|
||||
assert!(
|
||||
message.contains("welcome timed out") || message.contains("message deadline elapsed"),
|
||||
"{message}"
|
||||
);
|
||||
assert!(!failure.bootstrap_maybe_consumed);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn controller_silence_must_be_a_plausible_window() {
|
||||
let mut config = config(43127);
|
||||
for silence in [
|
||||
Duration::ZERO,
|
||||
Duration::from_millis(99),
|
||||
Duration::from_secs(3601),
|
||||
] {
|
||||
config.controller_silence = silence;
|
||||
assert!(config.validate().is_err(), "{silence:?}");
|
||||
}
|
||||
for silence in [
|
||||
Duration::from_millis(100),
|
||||
crate::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
Duration::from_secs(3600),
|
||||
] {
|
||||
config.controller_silence = silence;
|
||||
config.validate().unwrap();
|
||||
}
|
||||
}
|
||||
@@ -943,6 +943,8 @@ pub(crate) struct AuthenticatedTransport {
|
||||
socket: RunnerSocket,
|
||||
secure_channel: SecureChannel,
|
||||
max_frame_bytes: usize,
|
||||
last_inbound: Instant,
|
||||
controller_silence: Duration,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
@@ -1121,6 +1123,8 @@ impl AuthenticatedTransport {
|
||||
socket,
|
||||
secure_channel,
|
||||
max_frame_bytes: config.max_frame_bytes,
|
||||
last_inbound: Instant::now(),
|
||||
controller_silence: config.controller_silence,
|
||||
};
|
||||
// Ticket validation and lease persistence happen before the server
|
||||
// emits welcome. Give that durable admission the connection setup
|
||||
@@ -1147,6 +1151,7 @@ impl AuthenticatedTransport {
|
||||
expected_lease,
|
||||
challenge.selected_version,
|
||||
)?;
|
||||
transport.last_inbound = Instant::now();
|
||||
// Authentication can wait longer for control-plane validation, but
|
||||
// the steady-state runner loop must return to provider polling
|
||||
// promptly when no control message is available.
|
||||
@@ -1176,12 +1181,17 @@ impl AuthenticatedTransport {
|
||||
}
|
||||
|
||||
pub(crate) fn receive_json(&mut self) -> Result<Option<Value>, DurableRunnerError> {
|
||||
self.receive_json_until(None)
|
||||
match self.receive_json_until(None)? {
|
||||
None if self.listener_controller_silent() => Err(DurableRunnerError::invalid(
|
||||
"controller sent no frame within the liveness window; listening for a new connection",
|
||||
)),
|
||||
message => Ok(message),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn receive_queued_json(&mut self) -> Result<Option<Value>, DurableRunnerError> {
|
||||
self.set_runtime_read_timeout(QUEUED_CONTROL_READ_TIMEOUT)?;
|
||||
let message = self.receive_json_until(None);
|
||||
let message = self.receive_json();
|
||||
self.set_runtime_read_timeout(RUNTIME_READ_TIMEOUT)?;
|
||||
message
|
||||
}
|
||||
@@ -1196,13 +1206,22 @@ impl AuthenticatedTransport {
|
||||
&mut self,
|
||||
deadline: Option<Instant>,
|
||||
) -> Result<Option<Value>, DurableRunnerError> {
|
||||
let frame =
|
||||
match receive_plain_optional_until(&mut self.socket, self.max_frame_bytes, deadline)? {
|
||||
Some(frame) => frame,
|
||||
None => return Ok(None),
|
||||
};
|
||||
let frame = match receive_plain_optional_until_observed(
|
||||
&mut self.socket,
|
||||
self.max_frame_bytes,
|
||||
deadline,
|
||||
Some(&mut self.last_inbound),
|
||||
)? {
|
||||
Some(frame) => frame,
|
||||
None => return Ok(None),
|
||||
};
|
||||
self.secure_channel.decrypt(&frame, false).map(Some)
|
||||
}
|
||||
|
||||
fn listener_controller_silent(&self) -> bool {
|
||||
matches!(self.socket, RunnerSocket::Listen(_))
|
||||
&& self.last_inbound.elapsed() >= self.controller_silence
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Deserialize)]
|
||||
@@ -1629,6 +1648,15 @@ fn receive_plain_optional_until(
|
||||
socket: &mut (impl WebSocketMessages + ?Sized),
|
||||
max_frame_bytes: usize,
|
||||
deadline: Option<Instant>,
|
||||
) -> Result<Option<Value>, DurableRunnerError> {
|
||||
receive_plain_optional_until_observed(socket, max_frame_bytes, deadline, None)
|
||||
}
|
||||
|
||||
fn receive_plain_optional_until_observed(
|
||||
socket: &mut (impl WebSocketMessages + ?Sized),
|
||||
max_frame_bytes: usize,
|
||||
deadline: Option<Instant>,
|
||||
mut last_inbound: Option<&mut Instant>,
|
||||
) -> Result<Option<Value>, DurableRunnerError> {
|
||||
loop {
|
||||
if deadline.is_some_and(|deadline| Instant::now() >= deadline) {
|
||||
@@ -1642,6 +1670,9 @@ fn receive_plain_optional_until(
|
||||
"WebSocket authentication message deadline elapsed",
|
||||
));
|
||||
}
|
||||
if let (Ok(_), Some(last_inbound)) = (&message, last_inbound.as_deref_mut()) {
|
||||
*last_inbound = Instant::now();
|
||||
}
|
||||
match message {
|
||||
Ok(Message::Text(text)) => {
|
||||
if text.len() > max_frame_bytes {
|
||||
@@ -1782,6 +1813,7 @@ mod tests {
|
||||
include!("tests/acpx_park_relaunch.rs");
|
||||
include!("tests/ack_release_poll.rs");
|
||||
include!("tests/indeterminate_terminal_ack.rs");
|
||||
include!("tests/listener_silent_controller.rs");
|
||||
include!("tests/loopback_nodelay.rs");
|
||||
|
||||
use std::net::{IpAddr, Ipv4Addr, TcpListener};
|
||||
@@ -1818,6 +1850,7 @@ mod tests {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(1),
|
||||
turn_scoped: false,
|
||||
controller_silence: crate::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1085,6 +1085,7 @@ mod tests {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(60),
|
||||
turn_scoped: false,
|
||||
controller_silence: crate::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
}
|
||||
}
|
||||
fn command(kind: &str, payload: Value) -> Command {
|
||||
|
||||
@@ -2618,6 +2618,7 @@ mod tests {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(60),
|
||||
turn_scoped: false,
|
||||
controller_silence: crate::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -413,6 +413,7 @@ mod tests {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::ZERO,
|
||||
turn_scoped: false,
|
||||
controller_silence: crate::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -269,6 +269,7 @@ mod tests {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::ZERO,
|
||||
turn_scoped: false,
|
||||
controller_silence: crate::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -130,6 +130,7 @@ impl Harness {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(30),
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
};
|
||||
let descriptor = json!({
|
||||
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
|
||||
|
||||
@@ -86,6 +86,7 @@ fn fixture(
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(30),
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
};
|
||||
let descriptor = json!({
|
||||
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
|
||||
|
||||
@@ -81,6 +81,7 @@ fn checkpoints_the_sidecar_and_rebinds_consecutive_warm_runs_before_accepting_wo
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(30),
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
};
|
||||
let mut descriptor = json!({
|
||||
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
|
||||
@@ -241,6 +242,7 @@ fn reopens_the_conversation_for_a_turn_after_a_suspension_stop() {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(30),
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
};
|
||||
let descriptor = json!({
|
||||
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
|
||||
@@ -359,6 +361,7 @@ fn only_a_restored_session_with_an_active_turn_polls_as_live() {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(30),
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
};
|
||||
let descriptor = json!({
|
||||
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
|
||||
@@ -444,6 +447,7 @@ fn restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart() {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(30),
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
};
|
||||
let descriptor = json!({
|
||||
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
|
||||
@@ -587,6 +591,7 @@ fn restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile() {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(30),
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
};
|
||||
let descriptor = json!({
|
||||
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
|
||||
@@ -714,6 +719,7 @@ fn continues_a_settled_conversation_in_the_next_run_after_a_runner_upgrade() {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(30),
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
};
|
||||
let mut descriptor = json!({
|
||||
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
|
||||
@@ -845,6 +851,7 @@ fn a_lost_cancellation_reply_still_settles_the_turn_and_suspends_the_runner() {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(60),
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
};
|
||||
let descriptor = json!({
|
||||
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
|
||||
|
||||
@@ -781,6 +781,7 @@ fn durable_config(directory: &Path) -> DurableRunnerConfig {
|
||||
reconnect_grace: None,
|
||||
max_runtime: std::time::Duration::from_secs(5),
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -33,6 +33,7 @@ fn config(state_dir: PathBuf) -> DurableRunnerConfig {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(1),
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -153,6 +153,7 @@ fn config(state_dir: &Path) -> DurableRunnerConfig {
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(60),
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -57,6 +57,7 @@ fn real_cli_uses_relative_state_from_another_directory_and_never_process_run_ide
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::ZERO,
|
||||
turn_scoped: false,
|
||||
controller_silence: paperclip_runner_core::durable::DEFAULT_CONTROLLER_SILENCE,
|
||||
};
|
||||
let store = DurableStateStore::new(&path).unwrap();
|
||||
let (mut state, _) = store.load_or_create(&config).unwrap();
|
||||
|
||||
Reference in new issue
Block a user