Files
PaperClipAI/server
fa96716bb5 fix(storage): back off a throttled small upload that has no source file (#22404)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server saves many objects to S3-compatible storage: skill
packages, attachments, recovery archives and manifests
> - Object storage answers a burst with `503 SlowDown`. Most large and
file-backed uploads already back off (#20165, #20313, #20772, #21010)
> - A small upload with no source file still had almost no retry. The S3
SDK never resends a stream body, so a stream got one attempt. A Buffer
got the SDK's three closely spaced attempts
> - Hosted production and staging reported `SlowDown` on `put_object`
from request handlers, for example right after a skill import fetched
its files. The error continues on the current production release
> - This pull request holds a small stream's bytes within the existing
buffered-payload budget, and backs off a throttled small upload like a
file upload
> - The benefit is that a short request-rate limit no longer fails a
small save

## Linked Issues or Issue Description

Refs #20165, #20313, #20772, #21010 (the same backoff for other upload
paths).

**What happened?**

A request handler saved a small object with `storage.putFile` and a
stream body (for example a skill package archive), or with a small
Buffer. Object storage answered `503 SlowDown` ("Please reduce your
request rate."). The stream was not sent again. The Buffer was sent
three times within about a second. The request failed with a 500.

**Expected behavior**

A request-rate limit slows the upload down. It does not fail the save.

**Steps to reproduce**

1. Point the S3 provider at an endpoint that answers the first `PUT`
with `503 SlowDown`.
2. Call `putFile` with a small `Readable` body and no source file.
3. On `master` it throws `SlowDown: Please reduce your request rate.`
after one `PUT`.

**Paperclip version or commit**

`master` at `201abacd6e`.

## What Changed

- `server/src/storage/s3-multipart.ts`: `withBufferedUploadBody` holds a
small stream body within the shared buffered-payload budget that
multipart already uses. It checks the declared length.
`sendThrottledRequestWithRetry` uses the same jittered exponential
backoff as file uploads. It resends only request-rate limits (SlowDown,
503, 429).
- `server/src/storage/s3-provider.ts`: the small `putObject` path
without a source file sends Buffer bodies and buffered stream bodies
through that retry. When the budget is full, the stream is sent once, as
before.
- `server/src/__tests__/storage-s3-small-upload-throttle.test.ts` (new):
a real S3 client against a local endpoint.

## Verification

- `cd server && npx vitest run
src/__tests__/storage-s3-small-upload-throttle.test.ts`: with the
`master` source files, 3 of 4 cases fail (the stream and Buffer cases
reject with `SlowDown`). With this change, 4 of 4 pass.
- `cd server && npx vitest run src/__tests__/storage-* src/storage`: 17
files, 135 tests pass on top of `201abacd6e`, including `does not repeat
a write that timed out`.
- I did not run the full server typecheck locally. The sandbox ran out
of memory. CI must cover it.

## Risks

- Low. A small stream is now held in memory, at most 16 MiB, inside the
existing 32 MiB process budget. If the budget is full, behavior is
unchanged.
- Only request-rate limits are resent. The store did not apply those
writes. Other write failures keep their SDK handling, so a timed-out
write is still not repeated.
- A stream that ends early or runs long now fails before any request is
sent.
- Conditional publication puts (`putObjectConditional`) are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with
tool use (shell, file edit, test execution) and read-only Sentry reads.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
doc change is necessary: this is internal storage behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
2026-11-20 05:37:17 -08:00
..