mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner talks to ACPX providers through a sidecar process.
The sidecar writes events and command replies as JSON lines on stdout.
> - While the transport waits for a command reply, it buffers every
event that arrives before the reply. It failed when 512 events were
buffered.
> - The sidecar allows up to 4 MiB of unflushed stdout. A backlog of
small events passes 512 events long before any memory concern, for
example during a long turn with several subagents while the runner
relays events slowly.
> - When the limit hits, the transport is poisoned and the turn fails
with `acpx_provider_transport_failed`, which is not recoverable. The
task is then blocked.
> - This pull request bounds the receive buffer by frame bytes,
consistent with the sidecar's own output bound, and keeps a count
backstop for floods of tiny frames.
> - The benefit is that a normal event backlog no longer kills the turn,
and the buffer stays bounded in memory.
## Linked Issues or Issue Description
**What happened?**
A routine run started several subagents, which made about 190 GitHub
tool calls in about 10 minutes. The runner relays provider events at
about one per controller round trip, so the sidecar output fell several
minutes behind. A semantic tool result was then delivered through
`tool.resolve`. While it waited for the reply, the transport read more
than 512 backlogged events, failed with `ACPX sidecar exceeded the
buffered event limit`, and poisoned the transport. The turn failed with
`acpx_provider_transport_failed`, and recovery blocked the task.
**Expected behavior**
A backlog that the sidecar itself accepted does not fail a request. Only
a backlog that is too large for memory fails.
**Steps to reproduce**
1. Let a sidecar write more than 512 small events before it writes the
reply to a pending command.
2. The command fails with `ACPX sidecar exceeded the buffered event
limit`, and the transport is poisoned.
**Relevant logs or output**
```
ACPX sidecar exceeded the buffered event limit
turn.failed code=acpx_provider_transport_failed recoverable=false
```
No related public PR exists. I searched open and closed PRs for this
limit. The slow event relay that builds the backlog is a separate
problem.
## What Changed
- `acpx_sidecar_transport.rs`:
- Each buffered event keeps the byte length of its frame. The transport
keeps a running byte total.
- The buffer fails above 16 MiB (`MAX_BUFFERED_EVENT_BYTES`, four times
the sidecar's 4 MiB unflushed output bound
`SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES`), or above 65,536 events (one event
per 256 bytes of the budget) as a backstop for tiny frames.
- `poll_event` and `defer_polled_event` keep the byte total correct. A
poisoned transport resets it.
- `fake-acpx-sidecar.rs`: new `backlog`, `large-backlog` and
`count-flood` modes. `flood` now writes 18 MiB of 64 KiB events instead
of 513 small events.
- `tests/acpx_sidecar_transport.rs`: new tests for a backlog of 2,048
small events, for freeing the byte budget as events drain, and for a
flood of tiny events. The existing flood test still checks the error.
## Verification
- `cargo test --release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --test acpx_sidecar_transport` (in
`packages/paperclip-runner`): 23 passed.
- `buffers_a_backlog_of_small_events_before_a_response` fails before
this change (2,048 events are above the old 512 limit).
`frees_the_byte_budget_as_buffered_events_drain` needs correct
accounting: two backlogs of 12 MiB pass only when drained events free
their bytes.
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`
pass.
- `cargo test --release --locked -p paperclip-runner-core
--no-fail-fast`: 1,128 passed, 4 failed. The failures are
`codex_provider::tests::fenced_receipt_output_keeps_prefix_when_stdout_never_closes`
and
`fenced_receipt_output_reports_capacity_without_dropping_known_frames`
("failed to signal owned process group: Operation not permitted"), and
`stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement`
and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay
exit timing). None of them uses the ACPX sidecar transport. They come
from the macOS sandbox where I ran the suite. CI covers them on Linux.
- `cargo test --release -p paperclip-runner-core --lib acpx`: 99 passed.
`--test acpx_provider_resolutions --test acpx_sidecar_stopped_time`:
passed.
- I did not run a live provider. After the deploy, a long turn with many
tool events should not fail with `buffered event limit`.
## Risks
- Low risk. The change is local to the transport receive buffer.
- The memory bound is now 16 MiB of frames instead of 512 events. Parsed
events use more memory than their frames, so the worst case is a few
times 16 MiB in the runner process.
- The sidecar's own 4 MiB unflushed output bound does not change. A
burst that is longer than that still fails in the sidecar with
`output_buffer_limit_exceeded`. The slow relay that builds the backlog
is a separate fix.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>