mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(runner): bound the ACPX receive buffer by bytes, not 512 events (#21524)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner talks to ACPX providers through a sidecar process.
The sidecar writes events and command replies as JSON lines on stdout.
> - While the transport waits for a command reply, it buffers every
event that arrives before the reply. It failed when 512 events were
buffered.
> - The sidecar allows up to 4 MiB of unflushed stdout. A backlog of
small events passes 512 events long before any memory concern, for
example during a long turn with several subagents while the runner
relays events slowly.
> - When the limit hits, the transport is poisoned and the turn fails
with `acpx_provider_transport_failed`, which is not recoverable. The
task is then blocked.
> - This pull request bounds the receive buffer by frame bytes,
consistent with the sidecar's own output bound, and keeps a count
backstop for floods of tiny frames.
> - The benefit is that a normal event backlog no longer kills the turn,
and the buffer stays bounded in memory.
## Linked Issues or Issue Description
**What happened?**
A routine run started several subagents, which made about 190 GitHub
tool calls in about 10 minutes. The runner relays provider events at
about one per controller round trip, so the sidecar output fell several
minutes behind. A semantic tool result was then delivered through
`tool.resolve`. While it waited for the reply, the transport read more
than 512 backlogged events, failed with `ACPX sidecar exceeded the
buffered event limit`, and poisoned the transport. The turn failed with
`acpx_provider_transport_failed`, and recovery blocked the task.
**Expected behavior**
A backlog that the sidecar itself accepted does not fail a request. Only
a backlog that is too large for memory fails.
**Steps to reproduce**
1. Let a sidecar write more than 512 small events before it writes the
reply to a pending command.
2. The command fails with `ACPX sidecar exceeded the buffered event
limit`, and the transport is poisoned.
**Relevant logs or output**
```
ACPX sidecar exceeded the buffered event limit
turn.failed code=acpx_provider_transport_failed recoverable=false
```
No related public PR exists. I searched open and closed PRs for this
limit. The slow event relay that builds the backlog is a separate
problem.
## What Changed
- `acpx_sidecar_transport.rs`:
- Each buffered event keeps the byte length of its frame. The transport
keeps a running byte total.
- The buffer fails above 16 MiB (`MAX_BUFFERED_EVENT_BYTES`, four times
the sidecar's 4 MiB unflushed output bound
`SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES`), or above 65,536 events (one event
per 256 bytes of the budget) as a backstop for tiny frames.
- `poll_event` and `defer_polled_event` keep the byte total correct. A
poisoned transport resets it.
- `fake-acpx-sidecar.rs`: new `backlog`, `large-backlog` and
`count-flood` modes. `flood` now writes 18 MiB of 64 KiB events instead
of 513 small events.
- `tests/acpx_sidecar_transport.rs`: new tests for a backlog of 2,048
small events, for freeing the byte budget as events drain, and for a
flood of tiny events. The existing flood test still checks the error.
## Verification
- `cargo test --release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --test acpx_sidecar_transport` (in
`packages/paperclip-runner`): 23 passed.
- `buffers_a_backlog_of_small_events_before_a_response` fails before
this change (2,048 events are above the old 512 limit).
`frees_the_byte_budget_as_buffered_events_drain` needs correct
accounting: two backlogs of 12 MiB pass only when drained events free
their bytes.
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`
pass.
- `cargo test --release --locked -p paperclip-runner-core
--no-fail-fast`: 1,128 passed, 4 failed. The failures are
`codex_provider::tests::fenced_receipt_output_keeps_prefix_when_stdout_never_closes`
and
`fenced_receipt_output_reports_capacity_without_dropping_known_frames`
("failed to signal owned process group: Operation not permitted"), and
`stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement`
and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay
exit timing). None of them uses the ACPX sidecar transport. They come
from the macOS sandbox where I ran the suite. CI covers them on Linux.
- `cargo test --release -p paperclip-runner-core --lib acpx`: 99 passed.
`--test acpx_provider_resolutions --test acpx_sidecar_stopped_time`:
passed.
- I did not run a live provider. After the deploy, a long turn with many
tool events should not fail with `buffered event limit`.
## Risks
- Low risk. The change is local to the transport receive buffer.
- The memory bound is now 16 MiB of frames instead of 512 events. Parsed
events use more memory than their frames, so the worst case is a few
times 16 MiB in the runner process.
- The sidecar's own 4 MiB unflushed output bound does not change. A
burst that is longer than that still fails in the sidecar with
`output_buffer_limit_exceeded`. The slow relay that builds the backlog
is a separate fix.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
089748fbf9
commit
254744f77c
3 files changed
+128
-18
No files matched your search
@@ -18,7 +18,9 @@ use crate::provider_failure::validated_provider_failure;
|
||||
use crate::stable_identity::{is_stable_id, DURABLE_STABLE_ID_CHARS, SHORT_STABLE_ID_CHARS};
|
||||
|
||||
pub const ACPX_SIDECAR_MAX_FRAME_BYTES: usize = 1024 * 1024;
|
||||
const MAX_BUFFERED_EVENTS: usize = 512;
|
||||
const SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES: usize = 4 * ACPX_SIDECAR_MAX_FRAME_BYTES;
|
||||
const MAX_BUFFERED_EVENT_BYTES: usize = 4 * SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES;
|
||||
const MAX_BUFFERED_EVENTS: usize = MAX_BUFFERED_EVENT_BYTES / 256;
|
||||
const MAX_EVENT_POLL_TIMEOUT: Duration = Duration::from_secs(120);
|
||||
const MAX_JSON_SAFE_INTEGER: u64 = 9_007_199_254_740_991;
|
||||
/// A response wait blocks in slices of at most this long.
|
||||
@@ -128,7 +130,9 @@ pub struct AcpxSidecarTransport {
|
||||
request_deadline: Option<Instant>,
|
||||
next_request_id: u64,
|
||||
last_event_sequence: u64,
|
||||
buffered_events: VecDeque<AcpxSidecarEvent>,
|
||||
buffered_events: VecDeque<(AcpxSidecarEvent, usize)>,
|
||||
buffered_event_bytes: usize,
|
||||
last_polled_event_bytes: usize,
|
||||
stderr_tail: BoundedLogBuffer,
|
||||
stderr_categories: BTreeSet<&'static str>,
|
||||
poisoned: bool,
|
||||
@@ -439,6 +443,8 @@ impl AcpxSidecarTransport {
|
||||
next_request_id: 1,
|
||||
last_event_sequence: 0,
|
||||
buffered_events: VecDeque::new(),
|
||||
buffered_event_bytes: 0,
|
||||
last_polled_event_bytes: 0,
|
||||
stderr_tail: BoundedLogBuffer::new(32, 8 * 1024),
|
||||
stderr_categories: BTreeSet::new(),
|
||||
poisoned: false,
|
||||
@@ -464,7 +470,7 @@ impl AcpxSidecarTransport {
|
||||
}
|
||||
|
||||
pub(crate) fn buffered_events(&self) -> impl ExactSizeIterator<Item = &AcpxSidecarEvent> {
|
||||
self.buffered_events.iter()
|
||||
self.buffered_events.iter().map(|(event, _)| event)
|
||||
}
|
||||
|
||||
pub fn request(
|
||||
@@ -550,22 +556,33 @@ impl AcpxSidecarTransport {
|
||||
"ACPX sidecar event poll timeout must not exceed 120 s",
|
||||
));
|
||||
}
|
||||
if let Some(event) = self.buffered_events.pop_front() {
|
||||
self.last_polled_event_bytes = 0;
|
||||
if let Some((event, bytes)) = self.buffered_events.pop_front() {
|
||||
self.buffered_event_bytes -= bytes;
|
||||
self.last_polled_event_bytes = bytes;
|
||||
return Ok(Some(event));
|
||||
}
|
||||
if timeout.is_zero() {
|
||||
return Ok(None);
|
||||
}
|
||||
let result = self.poll_event_inner(timeout);
|
||||
if result.is_err() {
|
||||
self.poison();
|
||||
match self.poll_event_inner(timeout) {
|
||||
Ok(Some((event, bytes))) => {
|
||||
self.last_polled_event_bytes = bytes;
|
||||
Ok(Some(event))
|
||||
}
|
||||
Ok(None) => Ok(None),
|
||||
Err(error) => {
|
||||
self.poison();
|
||||
Err(error)
|
||||
}
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
pub(crate) fn defer_polled_event(&mut self, event: AcpxSidecarEvent) {
|
||||
// poll_event already validated its sequence and freed its FIFO slot.
|
||||
self.buffered_events.push_front(event);
|
||||
let bytes = std::mem::take(&mut self.last_polled_event_bytes);
|
||||
self.buffered_event_bytes += bytes;
|
||||
self.buffered_events.push_front((event, bytes));
|
||||
}
|
||||
|
||||
pub fn shutdown(&mut self) -> Result<(), LocalRunnerError> {
|
||||
@@ -767,7 +784,7 @@ impl AcpxSidecarTransport {
|
||||
continue;
|
||||
};
|
||||
match parse_frame(&line)? {
|
||||
ParsedFrame::Event(event) => self.buffer_event(event)?,
|
||||
ParsedFrame::Event(event) => self.buffer_event(event, line.len())?,
|
||||
ParsedFrame::Response(response) => {
|
||||
if self.abandoned_observation == Some(response.id) {
|
||||
self.abandoned_observation = None;
|
||||
@@ -811,7 +828,7 @@ impl AcpxSidecarTransport {
|
||||
fn poll_event_inner(
|
||||
&mut self,
|
||||
timeout: Duration,
|
||||
) -> Result<Option<AcpxSidecarEvent>, LocalRunnerError> {
|
||||
) -> Result<Option<(AcpxSidecarEvent, usize)>, LocalRunnerError> {
|
||||
let deadline = Instant::now() + timeout;
|
||||
loop {
|
||||
let Some(line) = self.receive_stdout_line(
|
||||
@@ -824,7 +841,7 @@ impl AcpxSidecarTransport {
|
||||
match parse_frame(&line)? {
|
||||
ParsedFrame::Event(event) => {
|
||||
self.validate_event_sequence(event.sequence)?;
|
||||
return Ok(Some(event));
|
||||
return Ok(Some((event, line.len())));
|
||||
}
|
||||
ParsedFrame::Response(response)
|
||||
if self.abandoned_observation == Some(response.id) =>
|
||||
@@ -882,14 +899,21 @@ impl AcpxSidecarTransport {
|
||||
}
|
||||
}
|
||||
|
||||
fn buffer_event(&mut self, event: AcpxSidecarEvent) -> Result<(), LocalRunnerError> {
|
||||
if self.buffered_events.len() >= MAX_BUFFERED_EVENTS {
|
||||
fn buffer_event(
|
||||
&mut self,
|
||||
event: AcpxSidecarEvent,
|
||||
bytes: usize,
|
||||
) -> Result<(), LocalRunnerError> {
|
||||
if self.buffered_events.len() >= MAX_BUFFERED_EVENTS
|
||||
|| self.buffered_event_bytes.saturating_add(bytes) > MAX_BUFFERED_EVENT_BYTES
|
||||
{
|
||||
return Err(LocalRunnerError::invalid(
|
||||
"ACPX sidecar exceeded the buffered event limit",
|
||||
));
|
||||
}
|
||||
self.validate_event_sequence(event.sequence)?;
|
||||
self.buffered_events.push_back(event);
|
||||
self.buffered_event_bytes += bytes;
|
||||
self.buffered_events.push_back((event, bytes));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -994,6 +1018,7 @@ impl AcpxSidecarTransport {
|
||||
}
|
||||
self.poisoned = true;
|
||||
self.buffered_events.clear();
|
||||
self.buffered_event_bytes = 0;
|
||||
let _ = self.process.terminate_group();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -682,8 +682,30 @@ fn run() -> Result<(), Box<dyn std::error::Error>> {
|
||||
write_json(&mut stdout, &success(id, command, &request))?;
|
||||
}
|
||||
"flood" => {
|
||||
for sequence in 1..=513 {
|
||||
write_event(&mut stdout, sequence)?;
|
||||
for _ in 0..288 {
|
||||
write_padded_event(&mut stdout, next_sequence, 64 * 1024)?;
|
||||
next_sequence += 1;
|
||||
}
|
||||
write_json(&mut stdout, &success(id, command, &request))?;
|
||||
}
|
||||
"count-flood" => {
|
||||
for _ in 0..=65_536 {
|
||||
write_event(&mut stdout, next_sequence)?;
|
||||
next_sequence += 1;
|
||||
}
|
||||
write_json(&mut stdout, &success(id, command, &request))?;
|
||||
}
|
||||
"backlog" => {
|
||||
for _ in 0..2_048 {
|
||||
write_event(&mut stdout, next_sequence)?;
|
||||
next_sequence += 1;
|
||||
}
|
||||
write_json(&mut stdout, &success(id, command, &request))?;
|
||||
}
|
||||
"large-backlog" => {
|
||||
for _ in 0..192 {
|
||||
write_padded_event(&mut stdout, next_sequence, 64 * 1024)?;
|
||||
next_sequence += 1;
|
||||
}
|
||||
write_json(&mut stdout, &success(id, command, &request))?;
|
||||
}
|
||||
@@ -1490,6 +1512,20 @@ fn write_event(output: &mut impl Write, sequence: u64) -> io::Result<()> {
|
||||
)
|
||||
}
|
||||
|
||||
fn write_padded_event(output: &mut impl Write, sequence: u64, padding: usize) -> io::Result<()> {
|
||||
write_json(
|
||||
output,
|
||||
&json!({
|
||||
"protocolVersion": GENERATED_ACPX_SIDECAR_PROTOCOL_VERSION,
|
||||
"sequence": sequence,
|
||||
"eventType": "runtime.diagnostic",
|
||||
"runId": null,
|
||||
"turnId": null,
|
||||
"payload": { "code": "fake_event", "message": "x".repeat(padding) },
|
||||
}),
|
||||
)
|
||||
}
|
||||
|
||||
fn write_json(output: &mut impl Write, value: &Value) -> io::Result<()> {
|
||||
serde_json::to_writer(&mut *output, value)?;
|
||||
output.write_all(b"\n")?;
|
||||
|
||||
+50
-1
@@ -91,13 +91,62 @@ fn a_poisoned_transport_discards_events_buffered_before_failure() {
|
||||
|
||||
#[test]
|
||||
fn bounds_events_while_waiting_for_a_response() {
|
||||
let mut transport = transport("flood", Duration::from_secs(2));
|
||||
let mut transport = transport("flood", Duration::from_secs(10));
|
||||
let error = transport
|
||||
.request(GeneratedAcpxSidecarCommand::Initialize, json!({}))
|
||||
.expect_err("event flood must fail");
|
||||
assert!(error.to_string().contains("buffered event limit"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bounds_tiny_events_by_count_while_waiting_for_a_response() {
|
||||
let mut transport = transport("count-flood", Duration::from_secs(20));
|
||||
let error = transport
|
||||
.request(GeneratedAcpxSidecarCommand::Initialize, json!({}))
|
||||
.expect_err("a flood of tiny events must fail");
|
||||
assert!(error.to_string().contains("buffered event limit"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn buffers_a_backlog_of_small_events_before_a_response() {
|
||||
let mut transport = transport("backlog", Duration::from_secs(10));
|
||||
transport
|
||||
.request(GeneratedAcpxSidecarCommand::Initialize, json!({}))
|
||||
.expect("a backlog within the byte budget should not fail the request");
|
||||
for expected in 1..=2_048 {
|
||||
let event = transport
|
||||
.poll_event(Duration::ZERO)
|
||||
.expect("buffered event should parse")
|
||||
.expect("buffered event should exist");
|
||||
assert_eq!(event.sequence, expected);
|
||||
}
|
||||
assert!(transport
|
||||
.poll_event(Duration::ZERO)
|
||||
.expect("drained buffer should not fail")
|
||||
.is_none());
|
||||
transport.shutdown().expect("fake sidecar should stop");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn frees_the_byte_budget_as_buffered_events_drain() {
|
||||
let mut transport = transport("large-backlog", Duration::from_secs(10));
|
||||
let mut expected = 1;
|
||||
for _ in 0..2 {
|
||||
transport
|
||||
.request(GeneratedAcpxSidecarCommand::SessionRead, json!({}))
|
||||
.expect("a drained buffer should admit the next backlog");
|
||||
for _ in 0..192 {
|
||||
let event = transport
|
||||
.poll_event(Duration::ZERO)
|
||||
.expect("buffered event should parse")
|
||||
.expect("buffered event should exist");
|
||||
assert_eq!(event.sequence, expected);
|
||||
expected += 1;
|
||||
}
|
||||
}
|
||||
transport.shutdown().expect("fake sidecar should stop");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_an_oversized_stdout_frame() {
|
||||
let mut transport = transport("oversized", Duration::from_secs(1));
|
||||
|
||||
Reference in new issue
Block a user