## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The storage service uploads large workspace snapshots through S3
multipart uploads.
> - The current uploader permits four requests but fills each streamed
body before it starts the next one.
> - Slow payload delivery therefore remains serial even when request
acknowledgements overlap.
> - This pull request fills bounded part buffers and sends those
payloads concurrently through the existing SDK streams.
> - This lets the four-request window carry payload data while it
retains backpressure and failure cleanup.
## Linked Issues or Issue Description
Related implementation: #18659. This fixes payload serialization within
that existing multipart owner. A search found no duplicate open storage
multipart fix.
**What happened?**
A controlled 24 MiB upload started part 2 only after part 1 had
delivered 99.22% of its bytes. The same sequence occurred for part 3.
The four-request cap overlapped acknowledgement waits but did not
provide useful overlap during payload delivery.
**Expected behavior**
Up to four part payloads can travel at the same time. Memory stays
bounded. Exact lengths, bytes, ordered completion, cancellation and
abort cleanup still apply.
**Steps to reproduce**
Run the new slow-consumer test against the previous uploader. Hold all
part consumers until four requests have started. The previous
implementation starts one request and the assertion fails. The revised
implementation starts four requests and then verifies every payload hash
and byte count.
**Paperclip version or commit**
Reproduced on the source from `585c885168`. The final branch is rebased
onto `ae1f781cac`.
**Deployment mode**
The S3 storage provider. Local tests use fake credentials and an owned
loopback HTTP server. No hosted writes were made.
## What Changed
- Share a 32 MiB part-buffer allowance across uploads. Stream up to four
buffered payloads concurrently through the existing SDK requests. When
the allowance is full, use the existing streaming path without an upload
queue.
- Keep the original streaming path for dynamic parts larger than 8 MiB.
This retains the provider part-count limit without larger buffers.
- Extend the existing tests for concurrent slow consumers, the shared
memory window and cancellation in both paths. Verify allowance release
after source and request failures. Check part identity independently of
acknowledgement order.
## Verification
- Baseline RED: the new controlled test failed with one request started
instead of four. Its cleanup released and awaited the upload.
- Final review-repair head: 26/26 tests passed across
`storage-s3-multipart.test.ts`, `storage-s3-failure-diagnostics.test.ts`
and `storage/config.test.ts`.
- These tests include the real S3 HTTP client, exact payload hashes and
ordered parts, excess and truncated input, early acknowledgements,
sibling failure, abort cleanup, stalled input and the bounded four-part
window.
- Strict server TypeScript, module boundary checks and `git diff
--check` passed on the final review-repair head. CI then caught an
optional SDK `Key` type in the new test fixture. A focused test
typecheck reproduced both TS2345 errors. The fixture now binds its
required controlled key; that typecheck and the named shared-window
regression pass. The uploader source is unchanged by this repair.
- Controlled 24 MiB comparison: 640.58 ms before and 393.88 ms after.
The new requests started within 7.37 ms. Both runs used 1 ms per
consumed chunk and 250 ms acknowledgement delays. This measures the
controlled fixture only. It does not predict hosted S3 throughput or
total migration time.
- Review RED: two blocked uploads consumed 64 MiB of source chunks under
the original per-upload buffer scheme. The new shared allowance kept the
source reads within 32 MiB plus stream lookahead. Three new controls
passed for concurrent uploads and allowance release after source/request
failures.
- The first treatment run passed 12/13 cases. Its remaining test assumed
completion order matched part order. The test now binds lengths to part
numbers and still checks the ordered completion request. An initial
typecheck setup attempt lacked the built plugin SDK; the canonical
dependency build and final typecheck passed.
- Final head `fbadf4f1cf427b896bbc255ffe2e7ad480ce06d3`: [full
CI](https://github.com/paperclipai/paperclip/actions/runs/42373571408)
passed, including build, test shards, Typecheck + Release Registry and
`ci / verify`. All 61 reported contexts are terminal: 56 success, four
skipped and one neutral review. Greptile and Paperclip both gave a fresh
5/5 for this head. No review threads remain open.
## Risks
- Multipart uploads share at most 32 MiB of live part buffers per
process, plus existing stream and source-chunk overhead. The previous
path used smaller streaming buffers. With no buffer allowance left, an
upload uses the original streaming path. It does not wait for a shared
queue.
- Objects whose dynamic part size exceeds 8 MiB retain the current
serial payload behavior. No new object-size ceiling is introduced.
- Request bodies remain non-replayable streams. The SDK retry policy,
multipart protocol, cleanup policy, object format and public storage API
remain unchanged.
- No disk spool or cache is added. All current-head CI gates and both
reviews passed.
## Model Used
OpenAI `gpt-6.1-sol`, with reasoning, tool use and code execution. The
context window size is not exposed by this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Server tests that need a real PostgreSQL process must use
./helpers/embedded-postgres.ts instead of constructing embedded-postgres
directly.
The shared helper creates a throwaway data directory and a reserved-safe
loopback port for each test database. This protects the live Paperclip
control-plane Postgres from server vitest runs; see PAP-2033 for the incident
that introduced this guard.
