From 254744f77c32414946368e2c19ae61be7e2d3e19 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 13 Nov 2026 16:45:58 -0800 Subject: [PATCH] fix(runner): bound the ACPX receive buffer by bytes, not 512 events (#21524) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner talks to ACPX providers through a sidecar process. The sidecar writes events and command replies as JSON lines on stdout. > - While the transport waits for a command reply, it buffers every event that arrives before the reply. It failed when 512 events were buffered. > - The sidecar allows up to 4 MiB of unflushed stdout. A backlog of small events passes 512 events long before any memory concern, for example during a long turn with several subagents while the runner relays events slowly. > - When the limit hits, the transport is poisoned and the turn fails with `acpx_provider_transport_failed`, which is not recoverable. The task is then blocked. > - This pull request bounds the receive buffer by frame bytes, consistent with the sidecar's own output bound, and keeps a count backstop for floods of tiny frames. > - The benefit is that a normal event backlog no longer kills the turn, and the buffer stays bounded in memory. ## Linked Issues or Issue Description **What happened?** A routine run started several subagents, which made about 190 GitHub tool calls in about 10 minutes. The runner relays provider events at about one per controller round trip, so the sidecar output fell several minutes behind. A semantic tool result was then delivered through `tool.resolve`. While it waited for the reply, the transport read more than 512 backlogged events, failed with `ACPX sidecar exceeded the buffered event limit`, and poisoned the transport. The turn failed with `acpx_provider_transport_failed`, and recovery blocked the task. **Expected behavior** A backlog that the sidecar itself accepted does not fail a request. Only a backlog that is too large for memory fails. **Steps to reproduce** 1. Let a sidecar write more than 512 small events before it writes the reply to a pending command. 2. The command fails with `ACPX sidecar exceeded the buffered event limit`, and the transport is poisoned. **Relevant logs or output** ``` ACPX sidecar exceeded the buffered event limit turn.failed code=acpx_provider_transport_failed recoverable=false ``` No related public PR exists. I searched open and closed PRs for this limit. The slow event relay that builds the backlog is a separate problem. ## What Changed - `acpx_sidecar_transport.rs`: - Each buffered event keeps the byte length of its frame. The transport keeps a running byte total. - The buffer fails above 16 MiB (`MAX_BUFFERED_EVENT_BYTES`, four times the sidecar's 4 MiB unflushed output bound `SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES`), or above 65,536 events (one event per 256 bytes of the budget) as a backstop for tiny frames. - `poll_event` and `defer_polled_event` keep the byte total correct. A poisoned transport resets it. - `fake-acpx-sidecar.rs`: new `backlog`, `large-backlog` and `count-flood` modes. `flood` now writes 18 MiB of 64 KiB events instead of 513 small events. - `tests/acpx_sidecar_transport.rs`: new tests for a backlog of 2,048 small events, for freeing the byte budget as events drain, and for a flood of tiny events. The existing flood test still checks the error. ## Verification - `cargo test --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test acpx_sidecar_transport` (in `packages/paperclip-runner`): 23 passed. - `buffers_a_backlog_of_small_events_before_a_response` fails before this change (2,048 events are above the old 512 limit). `frees_the_byte_budget_as_buffered_events_drain` needs correct accounting: two backlogs of 12 MiB pass only when drained events free their bytes. - `cargo fmt --all -- --check` and `cargo check --locked --workspace` pass. - `cargo test --release --locked -p paperclip-runner-core --no-fail-fast`: 1,128 passed, 4 failed. The failures are `codex_provider::tests::fenced_receipt_output_keeps_prefix_when_stdout_never_closes` and `fenced_receipt_output_reports_capacity_without_dropping_known_frames` ("failed to signal owned process group: Operation not permitted"), and `stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement` and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay exit timing). None of them uses the ACPX sidecar transport. They come from the macOS sandbox where I ran the suite. CI covers them on Linux. - `cargo test --release -p paperclip-runner-core --lib acpx`: 99 passed. `--test acpx_provider_resolutions --test acpx_sidecar_stopped_time`: passed. - I did not run a live provider. After the deploy, a long turn with many tool events should not fail with `buffered event limit`. ## Risks - Low risk. The change is local to the transport receive buffer. - The memory bound is now 16 MiB of frames instead of 512 events. Parsed events use more memory than their frames, so the worst case is a few times 16 MiB in the runner process. - The sidecar's own 4 MiB unflushed output bound does not change. A burst that is longer than that still fails in the sidecar with `output_buffer_limit_exceeded`. The slow relay that builds the backlog is a separate fix. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .../runner-core/src/acpx_sidecar_transport.rs | 55 ++++++++++++++----- .../runner-core/src/bin/fake-acpx-sidecar.rs | 40 +++++++++++++- .../tests/acpx_sidecar_transport.rs | 51 ++++++++++++++++- 3 files changed, 128 insertions(+), 18 deletions(-) diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs index 042c52bdb06..350cece37b6 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs @@ -18,7 +18,9 @@ use crate::provider_failure::validated_provider_failure; use crate::stable_identity::{is_stable_id, DURABLE_STABLE_ID_CHARS, SHORT_STABLE_ID_CHARS}; pub const ACPX_SIDECAR_MAX_FRAME_BYTES: usize = 1024 * 1024; -const MAX_BUFFERED_EVENTS: usize = 512; +const SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES: usize = 4 * ACPX_SIDECAR_MAX_FRAME_BYTES; +const MAX_BUFFERED_EVENT_BYTES: usize = 4 * SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES; +const MAX_BUFFERED_EVENTS: usize = MAX_BUFFERED_EVENT_BYTES / 256; const MAX_EVENT_POLL_TIMEOUT: Duration = Duration::from_secs(120); const MAX_JSON_SAFE_INTEGER: u64 = 9_007_199_254_740_991; /// A response wait blocks in slices of at most this long. @@ -128,7 +130,9 @@ pub struct AcpxSidecarTransport { request_deadline: Option, next_request_id: u64, last_event_sequence: u64, - buffered_events: VecDeque, + buffered_events: VecDeque<(AcpxSidecarEvent, usize)>, + buffered_event_bytes: usize, + last_polled_event_bytes: usize, stderr_tail: BoundedLogBuffer, stderr_categories: BTreeSet<&'static str>, poisoned: bool, @@ -439,6 +443,8 @@ impl AcpxSidecarTransport { next_request_id: 1, last_event_sequence: 0, buffered_events: VecDeque::new(), + buffered_event_bytes: 0, + last_polled_event_bytes: 0, stderr_tail: BoundedLogBuffer::new(32, 8 * 1024), stderr_categories: BTreeSet::new(), poisoned: false, @@ -464,7 +470,7 @@ impl AcpxSidecarTransport { } pub(crate) fn buffered_events(&self) -> impl ExactSizeIterator { - self.buffered_events.iter() + self.buffered_events.iter().map(|(event, _)| event) } pub fn request( @@ -550,22 +556,33 @@ impl AcpxSidecarTransport { "ACPX sidecar event poll timeout must not exceed 120 s", )); } - if let Some(event) = self.buffered_events.pop_front() { + self.last_polled_event_bytes = 0; + if let Some((event, bytes)) = self.buffered_events.pop_front() { + self.buffered_event_bytes -= bytes; + self.last_polled_event_bytes = bytes; return Ok(Some(event)); } if timeout.is_zero() { return Ok(None); } - let result = self.poll_event_inner(timeout); - if result.is_err() { - self.poison(); + match self.poll_event_inner(timeout) { + Ok(Some((event, bytes))) => { + self.last_polled_event_bytes = bytes; + Ok(Some(event)) + } + Ok(None) => Ok(None), + Err(error) => { + self.poison(); + Err(error) + } } - result } pub(crate) fn defer_polled_event(&mut self, event: AcpxSidecarEvent) { // poll_event already validated its sequence and freed its FIFO slot. - self.buffered_events.push_front(event); + let bytes = std::mem::take(&mut self.last_polled_event_bytes); + self.buffered_event_bytes += bytes; + self.buffered_events.push_front((event, bytes)); } pub fn shutdown(&mut self) -> Result<(), LocalRunnerError> { @@ -767,7 +784,7 @@ impl AcpxSidecarTransport { continue; }; match parse_frame(&line)? { - ParsedFrame::Event(event) => self.buffer_event(event)?, + ParsedFrame::Event(event) => self.buffer_event(event, line.len())?, ParsedFrame::Response(response) => { if self.abandoned_observation == Some(response.id) { self.abandoned_observation = None; @@ -811,7 +828,7 @@ impl AcpxSidecarTransport { fn poll_event_inner( &mut self, timeout: Duration, - ) -> Result, LocalRunnerError> { + ) -> Result, LocalRunnerError> { let deadline = Instant::now() + timeout; loop { let Some(line) = self.receive_stdout_line( @@ -824,7 +841,7 @@ impl AcpxSidecarTransport { match parse_frame(&line)? { ParsedFrame::Event(event) => { self.validate_event_sequence(event.sequence)?; - return Ok(Some(event)); + return Ok(Some((event, line.len()))); } ParsedFrame::Response(response) if self.abandoned_observation == Some(response.id) => @@ -882,14 +899,21 @@ impl AcpxSidecarTransport { } } - fn buffer_event(&mut self, event: AcpxSidecarEvent) -> Result<(), LocalRunnerError> { - if self.buffered_events.len() >= MAX_BUFFERED_EVENTS { + fn buffer_event( + &mut self, + event: AcpxSidecarEvent, + bytes: usize, + ) -> Result<(), LocalRunnerError> { + if self.buffered_events.len() >= MAX_BUFFERED_EVENTS + || self.buffered_event_bytes.saturating_add(bytes) > MAX_BUFFERED_EVENT_BYTES + { return Err(LocalRunnerError::invalid( "ACPX sidecar exceeded the buffered event limit", )); } self.validate_event_sequence(event.sequence)?; - self.buffered_events.push_back(event); + self.buffered_event_bytes += bytes; + self.buffered_events.push_back((event, bytes)); Ok(()) } @@ -994,6 +1018,7 @@ impl AcpxSidecarTransport { } self.poisoned = true; self.buffered_events.clear(); + self.buffered_event_bytes = 0; let _ = self.process.terminate_group(); } } diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs index 188b32c7e22..f9b9e78563b 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs @@ -682,8 +682,30 @@ fn run() -> Result<(), Box> { write_json(&mut stdout, &success(id, command, &request))?; } "flood" => { - for sequence in 1..=513 { - write_event(&mut stdout, sequence)?; + for _ in 0..288 { + write_padded_event(&mut stdout, next_sequence, 64 * 1024)?; + next_sequence += 1; + } + write_json(&mut stdout, &success(id, command, &request))?; + } + "count-flood" => { + for _ in 0..=65_536 { + write_event(&mut stdout, next_sequence)?; + next_sequence += 1; + } + write_json(&mut stdout, &success(id, command, &request))?; + } + "backlog" => { + for _ in 0..2_048 { + write_event(&mut stdout, next_sequence)?; + next_sequence += 1; + } + write_json(&mut stdout, &success(id, command, &request))?; + } + "large-backlog" => { + for _ in 0..192 { + write_padded_event(&mut stdout, next_sequence, 64 * 1024)?; + next_sequence += 1; } write_json(&mut stdout, &success(id, command, &request))?; } @@ -1490,6 +1512,20 @@ fn write_event(output: &mut impl Write, sequence: u64) -> io::Result<()> { ) } +fn write_padded_event(output: &mut impl Write, sequence: u64, padding: usize) -> io::Result<()> { + write_json( + output, + &json!({ + "protocolVersion": GENERATED_ACPX_SIDECAR_PROTOCOL_VERSION, + "sequence": sequence, + "eventType": "runtime.diagnostic", + "runId": null, + "turnId": null, + "payload": { "code": "fake_event", "message": "x".repeat(padding) }, + }), + ) +} + fn write_json(output: &mut impl Write, value: &Value) -> io::Result<()> { serde_json::to_writer(&mut *output, value)?; output.write_all(b"\n")?; diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_sidecar_transport.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_sidecar_transport.rs index 9feaae2dedb..17a4558de23 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_sidecar_transport.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_sidecar_transport.rs @@ -91,13 +91,62 @@ fn a_poisoned_transport_discards_events_buffered_before_failure() { #[test] fn bounds_events_while_waiting_for_a_response() { - let mut transport = transport("flood", Duration::from_secs(2)); + let mut transport = transport("flood", Duration::from_secs(10)); let error = transport .request(GeneratedAcpxSidecarCommand::Initialize, json!({})) .expect_err("event flood must fail"); assert!(error.to_string().contains("buffered event limit")); } +#[test] +fn bounds_tiny_events_by_count_while_waiting_for_a_response() { + let mut transport = transport("count-flood", Duration::from_secs(20)); + let error = transport + .request(GeneratedAcpxSidecarCommand::Initialize, json!({})) + .expect_err("a flood of tiny events must fail"); + assert!(error.to_string().contains("buffered event limit")); +} + +#[test] +fn buffers_a_backlog_of_small_events_before_a_response() { + let mut transport = transport("backlog", Duration::from_secs(10)); + transport + .request(GeneratedAcpxSidecarCommand::Initialize, json!({})) + .expect("a backlog within the byte budget should not fail the request"); + for expected in 1..=2_048 { + let event = transport + .poll_event(Duration::ZERO) + .expect("buffered event should parse") + .expect("buffered event should exist"); + assert_eq!(event.sequence, expected); + } + assert!(transport + .poll_event(Duration::ZERO) + .expect("drained buffer should not fail") + .is_none()); + transport.shutdown().expect("fake sidecar should stop"); +} + +#[test] +fn frees_the_byte_budget_as_buffered_events_drain() { + let mut transport = transport("large-backlog", Duration::from_secs(10)); + let mut expected = 1; + for _ in 0..2 { + transport + .request(GeneratedAcpxSidecarCommand::SessionRead, json!({})) + .expect("a drained buffer should admit the next backlog"); + for _ in 0..192 { + let event = transport + .poll_event(Duration::ZERO) + .expect("buffered event should parse") + .expect("buffered event should exist"); + assert_eq!(event.sequence, expected); + expected += 1; + } + } + transport.shutdown().expect("fake sidecar should stop"); +} + #[test] fn rejects_an_oversized_stdout_frame() { let mut transport = transport("oversized", Duration::from_secs(1));