mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 05:16:50 +01:00
fa14631ba230fe3e831c73840d330cbb670a2bca
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Priya RamanandPaperclip
|
fa14631ba2 |
fix(adapter-utils): charge the duplex gateway body reserve for its true unknown-length peak
The duplex gateway charged an unknown-length request body one time, but readBody retains a chunk list and the Buffer.concat result together for one instant, so its true retained peak is up to two times the body cap. Name that factor as DUPLEX_GATEWAY_UNKNOWN_LENGTH_BODY_RESERVE_FACTOR and charge 2 * maxBodyBytes plus the fixed frame allowance for an unknown-length read. A declared-length read still charges the declared length plus the same fixed allowance, and a bodyless request still charges the zero-length reserve. Update the retained-peak comment to state the true unknown-length peak. The file gateway reserve is unchanged: FILE_GATEWAY_BODY_RESERVE_FACTOR stays 4. Add a regression test that sends four concurrent cap-sized chunked requests with no Content-Length header, holds the host forward so the reserves overlap, and sets the bound to admit exactly three reserves of 2 * maxBodyBytes + the frame allowance. It asserts the fourth request gets a 503 bridge_busy, then asserts a later request is admitted once a terminal releases a reserve. Proven against the pre-fix commit first: the test fails there (all four requests admit, none shed) and passes after this fix. Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Priya RamanandPaperclip
|
84f3a8eaf9 |
fix(adapter-utils): charge a bodyless request the zero-length reserve, not the worst-case cap
A request with no Content-Length header and no Transfer-Encoding header has no body (RFC 9112 section 6.3). declaredBodyBytes returned null for that shape, so both bridge gateways charged the same worst-case maxBodyBytes reserve as an unknown-size chunked body. A plain DELETE takes this exact shape, so two concurrent DELETE requests -- or one DELETE next to one large POST -- exceeded the production inflight bound and got a 503, even though neither request holds any real body memory. Classify the framing in three ways instead of two: a Transfer-Encoding header always means an unknown length; a valid Content-Length header is a declared length; neither header present is a declared length of zero. Both gateways already treat a non-null declared length as the real charge, so this alone fixes the reserve for both gateways with no other call-site change. The zero case still reads through readBody with a declared length of zero, so an unexpected body byte still raises a 413. The charge for a declared length and for an unknown (chunked) length is unchanged in value. The chunked worst case is documented in the comment on the charge: at production defaults the file gateway admits exactly one such request at a time, and this is intentional -- the gateway cannot know a chunked body's size before it reads it. Regression tests prove two concurrent DELETE requests, and one DELETE concurrent with a POST at the body cap, both reach the host worker at production admission defaults; they fail against the prior commit and pass after this fix. A duplex-gateway test proves the same zero-length charge with the duplex charge model unchanged for declared and unknown lengths. Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
d7ed84c775 |
fix(adapter-utils): charge duplex bridge body admission from the declared length
Round 4 of the bridge body admission fix. reserveBodyRead and readBody took a flat maxBodyBytes charge for every active read. That charge was too large for a small control-plane call and too small for the file gateway, whose real retained peak reaches several times the raw body size once it decodes, re-serializes, and re-encodes the body. Read the Content-Length header before the gateway reserves anything. Reject a body that declares more than the cap with a clean 413, before the gateway reserves or reads one byte. When the header is absent (a chunked request), fall back to the cap as the worst case, the same as before. Charge the reserve and the read buffer from the declared length times a named per-gateway factor: - The duplex gateway adds a fixed frame allowance to the declared length, because it holds one body buffer plus at most one in-flight frame that does not grow with the body. - The file gateway multiplies the declared length by a factor of 15, the sum of the four representations it holds: the raw buffer, the decoded string, the JSON-escaped string, and the write buffer. The comment on the constant names each one and its worst-case size. Fix the outbound stdout writer so a stream error settles every queued write, not only the one at the head of the queue. The old top-of-loop check discarded queued entries with no reject call, and the resume path after a stalled write could remove and orphan a line a caller queued after the stream had already failed. Both left a promise that never settled, which stalled the reserve release and shed every later request until the process restarted. Add unit tests for the writer fix and end-to-end tests on both gateways for the declared-length admission model: the pre-reserve 413, the worst-case fallback for a request with no declared length, many small concurrent requests the old flat charge would have shed, and the admission-bound math at the new per-gateway reserve. Co-authored-by: Paperclip <noreply@paperclip.ing> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
b070181aba |
fix(adapter-utils): serialize duplex gateway stdout writes and fix reserve release order
The Stage-2 security re-review found two remaining defects in the in-sandbox duplex gateway. First, the gateway wrote every frame to process.stdout and ignored the boolean write() returns. A writable pipe returns false at its high-water mark, and Node then queues every later write in the stream's own buffer. A slow host reader let one admitted read retain its whole body's worth of encoded frames, not the one-frame margin the retained-peak comment claimed. Fix: route every outbound frame -- the heartbeat, the READY frame, the request envelope, and each body_chunk -- through one serialized writer. The writer issues at most one write at a time and waits for the stream to drain (or to error) before it starts the next queued line, so a slow reader can no longer make the gateway queue more than about one frame per active send. Second, readBody held the chunks array and the Buffer.concat result at the same time, so the transient peak was 2x the raw body while admission charged 1x. Fix: readBody now preallocates one buffer sized to the admission charge and copies each chunk into it in place, then returns a view over the written bytes. This removes the double retention; the tradeoff (a small body now retains the full charge-sized buffer for the life of the request) is documented in the function comment. Third, the admission reserve released inside a finally block on the response promise, so a response timeout could release the reserve while the outbound send still held the body. Fix: the reserve now releases only after both the response settled and the send finished (drained fully or ended by a stream error), tracked independently and released exactly once through an idempotent release function. The send now starts after the pending entry and its timer are registered, so a response timeout no longer stops or races the send. The retained-peak comment is rewritten to describe what the new code actually holds: one body buffer, plus at most one in-flight frame bounded by the frame size limit, times the admitted read count. Tests: a pipe-backpressure regression test proves the writer never queues more than about one bounded frame per active send under a stalled pipe; two isolated writer tests (extracted the same way the frame codec already is) prove the writer serializes writes and ends a stalled write on a stream error; an ordering test proves one request's chunks stay in sequence while a concurrent request interleaves its own; two reserve tests prove the reserve stays held through a stalled send and releases only once the send and the response both finish, for both a resumed pipe and a broken one. Kept unchanged: the 10 MiB + 1 body cap, the 64 MiB inflight bound, the host spill behavior, and the observable 413 / 503 / 502 statuses. Every existing test on the three touched files stays green (189 tests total). tsc is clean. Co-authored-by: Paperclip <noreply@paperclip.ing> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Priya RamanandPaperclip
|
f1af57bbfc |
fix(adapter-utils): stop duplex bridge body-read amplification and bound the retained peak
The in-sandbox gateway charged request-body admission on the raw source bytes, but it retained and forwarded more bytes than it charged. Two defects allowed a body to exceed the charged bytes. First, readBody returned a UTF-8 string. The duplex gateway re-encoded that string to a Buffer. Each invalid byte became the 3-byte U+FFFD replacement, so a cap-sized malformed body expanded to about 3x the charged bytes after admission. Preserve the raw request bytes end to end: readBody now returns the raw Buffer and the duplex gateway uses it directly. The base64 frames and the envelope byte count now equal the wire bytes, so a malformed body cannot expand. Second, the duplex path held the raw body plus the full base64 frame array plus the full encoded-line array at once, about 3.7x the raw body. Stream the body as body_chunk frames one at a time, so the gateway holds the raw body plus at most one frame. The retained peak per read is now about the charged bytes plus 1 MB, so the inflight bound bounds the true retained bytes. The file gateway writes the body as a JSON string that the host reader consumes. It rejects an invalid-UTF-8 body with a clean 400 before it converts the body, so it does not amplify a malformed body and keeps the on-disk payload contract unchanged. Tests: a cap-sized malformed-UTF-8 body forwards byte-for-byte with no expanded frame; the admission bound caps concurrent reads at the retained-byte model; the file gateway answers 400 for an invalid body and forwards nothing. Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Priya RamanandPaperclip
|
de5a1bdbe7 |
feat(duplex): raise bridge body cap to the API JSON limit and add safety bounds
Raise the sandbox-to-host duplex bridge body cap from 256 KiB to the API JSON body limit plus one byte (10 MiB + 1). The bridge is a buffering backstop, not the policy limit, so a body of exactly the API limit passes through and the API issues its own 413. Fix and bound the change with these parts: - Add DEFAULT_JSON_BODY_LIMIT_BYTES and derive the "10mb" string from it so the byte form and the string form cannot drift. A cross-package test asserts the bridge cap equals this byte limit plus one. - Answer a clean 413 request_too_large, not a retryable 502 with the internal message, when a request body passes the bridge cap. readBody throws a typed error that each gateway maps to 413. - Add local in-sandbox body-read admission control. The gateway reserves one worst-case body per active read against a process-owned bound and sheds a new read with 503 when the bound is full. It releases the reserve on every terminal path. This protects the sandbox process; it is separate from the host ledger. - Pin the SSH command output buffer at 1 MiB with its own constant, so it no longer tracks the bridge body cap. - Raise the host pre-bind buffer to 16 MiB, sized from the body cap and the base64 body-chunk expansion, so one in-spec body cannot terminate its route. Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
16b59c9315 |
feat(adapter-utils): stream duplex bridge bodies as sequenced chunks with receive-side spill (#12006)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent adapters use a duplex bridge to send requests and responses across an isolated boundary > - The bridge held each request body and response body in memory on both ends > - Large bodies can exhaust memory and reduce the safe size of adapter traffic > - This pull request sends receive-side bodies as sequenced chunks and spills large bodies to disk > - The benefit is bounded memory use with strict size, order, and cleanup checks ## Linked Issues or Issue Description **What existing behavior does this improve?** The adapter-utils duplex bridge transports request and response bodies across the sandbox boundary. **Current behavior** The bridge stores each complete body in memory on both ends of the duplex channel. **Proposed behavior** The bridge sends body chunks with sequence checks. The receive side keeps bodies up to 1 MiB in memory and spills larger bodies to a temporary file. **Reason and benefit** This change reduces memory pressure and keeps malformed or oversized input on a terminal error path. **Breaking changes** The duplex frame version changes to version 2. The request and response envelopes now carry bodyByteCount, and body_chunk frames carry the body data. ## What Changed - Add version 2 body_chunk frames with 256 KiB raw slices encoded as canonical base64 text. - Add receive-side memory and spill reassembly with per-channel disk and file limits. - Reject malformed, reordered, oversized, truncated, and non-canonical body chunks. - Stream reassembled request bodies to the host forward handler with a web stream and half-duplex request. - Remove spill files on success, failure, channel death, and startup cleanup. ## Verification - Run the adapter-utils type-check. - Run the adapter-utils duplex test suite. - Run all pull request checks. - Run the Greptile review and confirm a 5/5 score with no open findings. ## Risks The wire format changes from version 1 to version 2. Older bridge peers cannot use this protocol. The receive path adds temporary file operations and cleanup paths. The implementation fails closed when a body violates size or sequence rules. ## Model Used OpenAI GPT-5 Codex, tool-enabled coding agent. The exact context-window size and reasoning mode are not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs) - [x] My branch name describes the change (e.g. docs/..., fix/...) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
10d2781a29 |
feat(sandbox): add the duplex bridge broker, gated transport selection, and fixed observability (#11769)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Sandbox adapters provide controlled execution for untrusted provider environments. > - The sandbox channel needs one persistent duplex transport with strict host control. > - The transport must remain off unless the instance setting and provider capability both allow it. > - The host must detect loss, bound resource use, and expose only safe telemetry. > - This pull request adds the broker, gated selection, kill-switch wiring, fixed observability, and real-process proof. > - The benefit is safer sandbox execution with bounded failure behavior and inspectable transport results. ## Linked Issues or Issue Description No public issue exists for this change. The related pull requests are #11738 and #11750. **Problem or motivation** The sandbox duplex channel needs a host-controlled broker, strict transport gates, bounded provider input, and safe loss telemetry. Without these controls, a provider can cause replay, resource growth, unsafe endpoint selection, or data exposure through telemetry. **Proposed solution** Add a host broker with nested time limits, request limits, one-shot loss, and per-id deduplication. Select duplex transport only when the instance setting and provider capability both equal true. Assign the endpoint and nonce on the host. Reject invalid readiness data and use the file bridge on failure. Add fixed redacted telemetry and a real-process end-to-end test harness. **Alternatives considered** Keep the file bridge as the only transport. This avoids new channel behavior but does not provide persistent duplex operation for supported sandbox providers. **Roadmap alignment** This change supports the Cloud / Sandbox agents section in ROADMAP.md. ## What Changed - Add the duplex bridge broker with bounded forward, response, and gateway wait budgets. - Bound concurrent requests, lifetime requests, and request-id bytes before retention or forwarding. - Select duplex transport only when both required gates are true. - Assign the loopback port and nonce on the host and enforce a liveness-only READY frame. - Fall back to the file bridge after invalid readiness, contamination, bind failure, or timeout. - Carry the kill switch through the server, acpx engine, and six local adapters. - Add fixed, redacted duplex telemetry with a provider allowlist. - Add a real-process end-to-end harness for readiness, round trips, loss, and teardown. - Add regression coverage for limits, loss, UTF-8 splits, concurrency, and telemetry dimensions. ## Verification - Adapter-utils, server, and Daytona typechecks pass locally. - Adapter-utils tests pass, including the codec, broker, execution-target sandbox, and real-process harness. - Server kill-switch tests pass. - Live Daytona tests pass with the required provider key and skip without that key. - The root pnpm-lock.yaml file has no diff. - The branch contains ten commits after origin/master. ## Risks - Duplex transport remains disabled unless both gates equal true. - A provider remains an untrusted boundary and needs least-privilege credentials and quotas. - The server telemetry recorder stays deferred; the default recorder does nothing. - A provider that pre-binds the host port causes a fail-closed fallback to the file bridge. - The change adds no database migration and changes no root lockfile. ## Model Used OpenAI GPT-5, exact model family GPT-5, large context window, reasoning, and tool use. The model assisted with Git handoff validation and PR preparation. The implementation commits came from the engineering worktree. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs) - [x] My branch name describes the change (e.g. docs/... or fix/...) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge - [x] I searched the GitHub PR list for similar PRs and confirmed this is not a duplicate --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |