mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
DottaandPaperclip
8812c1941d
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The storage service uploads large workspace snapshots through S3 multipart uploads. > - The current uploader permits four requests but fills each streamed body before it starts the next one. > - Slow payload delivery therefore remains serial even when request acknowledgements overlap. > - This pull request fills bounded part buffers and sends those payloads concurrently through the existing SDK streams. > - This lets the four-request window carry payload data while it retains backpressure and failure cleanup. ## Linked Issues or Issue Description Related implementation: #18659. This fixes payload serialization within that existing multipart owner. A search found no duplicate open storage multipart fix. **What happened?** A controlled 24 MiB upload started part 2 only after part 1 had delivered 99.22% of its bytes. The same sequence occurred for part 3. The four-request cap overlapped acknowledgement waits but did not provide useful overlap during payload delivery. **Expected behavior** Up to four part payloads can travel at the same time. Memory stays bounded. Exact lengths, bytes, ordered completion, cancellation and abort cleanup still apply. **Steps to reproduce** Run the new slow-consumer test against the previous uploader. Hold all part consumers until four requests have started. The previous implementation starts one request and the assertion fails. The revised implementation starts four requests and then verifies every payload hash and byte count. **Paperclip version or commit** Reproduced on the source from `585c885168`. The final branch is rebased onto `ae1f781cac`. **Deployment mode** The S3 storage provider. Local tests use fake credentials and an owned loopback HTTP server. No hosted writes were made. ## What Changed - Share a 32 MiB part-buffer allowance across uploads. Stream up to four buffered payloads concurrently through the existing SDK requests. When the allowance is full, use the existing streaming path without an upload queue. - Keep the original streaming path for dynamic parts larger than 8 MiB. This retains the provider part-count limit without larger buffers. - Extend the existing tests for concurrent slow consumers, the shared memory window and cancellation in both paths. Verify allowance release after source and request failures. Check part identity independently of acknowledgement order. ## Verification - Baseline RED: the new controlled test failed with one request started instead of four. Its cleanup released and awaited the upload. - Final review-repair head: 26/26 tests passed across `storage-s3-multipart.test.ts`, `storage-s3-failure-diagnostics.test.ts` and `storage/config.test.ts`. - These tests include the real S3 HTTP client, exact payload hashes and ordered parts, excess and truncated input, early acknowledgements, sibling failure, abort cleanup, stalled input and the bounded four-part window. - Strict server TypeScript, module boundary checks and `git diff --check` passed on the final review-repair head. CI then caught an optional SDK `Key` type in the new test fixture. A focused test typecheck reproduced both TS2345 errors. The fixture now binds its required controlled key; that typecheck and the named shared-window regression pass. The uploader source is unchanged by this repair. - Controlled 24 MiB comparison: 640.58 ms before and 393.88 ms after. The new requests started within 7.37 ms. Both runs used 1 ms per consumed chunk and 250 ms acknowledgement delays. This measures the controlled fixture only. It does not predict hosted S3 throughput or total migration time. - Review RED: two blocked uploads consumed 64 MiB of source chunks under the original per-upload buffer scheme. The new shared allowance kept the source reads within 32 MiB plus stream lookahead. Three new controls passed for concurrent uploads and allowance release after source/request failures. - The first treatment run passed 12/13 cases. Its remaining test assumed completion order matched part order. The test now binds lengths to part numbers and still checks the ordered completion request. An initial typecheck setup attempt lacked the built plugin SDK; the canonical dependency build and final typecheck passed. - Final head `fbadf4f1cf427b896bbc255ffe2e7ad480ce06d3`: [full CI](https://github.com/paperclipai/paperclip/actions/runs/42373571408) passed, including build, test shards, Typecheck + Release Registry and `ci / verify`. All 61 reported contexts are terminal: 56 success, four skipped and one neutral review. Greptile and Paperclip both gave a fresh 5/5 for this head. No review threads remain open. ## Risks - Multipart uploads share at most 32 MiB of live part buffers per process, plus existing stream and source-chunk overhead. The previous path used smaller streaming buffers. With no buffer allowance left, an upload uses the original streaming path. It does not wait for a shared queue. - Objects whose dynamic part size exceeds 8 MiB retain the current serial payload behavior. No new object-size ceiling is introduced. - Request bodies remain non-replayable streams. The SDK retry policy, multipart protocol, cleanup policy, object format and public storage API remain unchanged. - No disk spool or cache is added. All current-head CI gates and both reviews passed. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, tool use and code execution. The context window size is not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>