mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server saves many objects to S3-compatible storage: skill packages, attachments, recovery archives and manifests > - Object storage answers a burst with `503 SlowDown`. Most large and file-backed uploads already back off (#20165, #20313, #20772, #21010) > - A small upload with no source file still had almost no retry. The S3 SDK never resends a stream body, so a stream got one attempt. A Buffer got the SDK's three closely spaced attempts > - Hosted production and staging reported `SlowDown` on `put_object` from request handlers, for example right after a skill import fetched its files. The error continues on the current production release > - This pull request holds a small stream's bytes within the existing buffered-payload budget, and backs off a throttled small upload like a file upload > - The benefit is that a short request-rate limit no longer fails a small save ## Linked Issues or Issue Description Refs #20165, #20313, #20772, #21010 (the same backoff for other upload paths). **What happened?** A request handler saved a small object with `storage.putFile` and a stream body (for example a skill package archive), or with a small Buffer. Object storage answered `503 SlowDown` ("Please reduce your request rate."). The stream was not sent again. The Buffer was sent three times within about a second. The request failed with a 500. **Expected behavior** A request-rate limit slows the upload down. It does not fail the save. **Steps to reproduce** 1. Point the S3 provider at an endpoint that answers the first `PUT` with `503 SlowDown`. 2. Call `putFile` with a small `Readable` body and no source file. 3. On `master` it throws `SlowDown: Please reduce your request rate.` after one `PUT`. **Paperclip version or commit** `master` at `201abacd6e`. ## What Changed - `server/src/storage/s3-multipart.ts`: `withBufferedUploadBody` holds a small stream body within the shared buffered-payload budget that multipart already uses. It checks the declared length. `sendThrottledRequestWithRetry` uses the same jittered exponential backoff as file uploads. It resends only request-rate limits (SlowDown, 503, 429). - `server/src/storage/s3-provider.ts`: the small `putObject` path without a source file sends Buffer bodies and buffered stream bodies through that retry. When the budget is full, the stream is sent once, as before. - `server/src/__tests__/storage-s3-small-upload-throttle.test.ts` (new): a real S3 client against a local endpoint. ## Verification - `cd server && npx vitest run src/__tests__/storage-s3-small-upload-throttle.test.ts`: with the `master` source files, 3 of 4 cases fail (the stream and Buffer cases reject with `SlowDown`). With this change, 4 of 4 pass. - `cd server && npx vitest run src/__tests__/storage-* src/storage`: 17 files, 135 tests pass on top of `201abacd6e`, including `does not repeat a write that timed out`. - I did not run the full server typecheck locally. The sandbox ran out of memory. CI must cover it. ## Risks - Low. A small stream is now held in memory, at most 16 MiB, inside the existing 32 MiB process budget. If the budget is full, behavior is unchanged. - Only request-rate limits are resent. The store did not apply those writes. Other write failures keep their SDK handling, so a timed-out write is still not repeated. - A stream that ends early or runs long now fails before any request is sent. - Conditional publication puts (`putObjectConditional`) are unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution) and read-only Sentry reads. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no doc change is necessary: this is internal storage behavior) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>