fix(storage): back off a throttled small upload that has no source file (#22404)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server saves many objects to S3-compatible storage: skill
packages, attachments, recovery archives and manifests
> - Object storage answers a burst with `503 SlowDown`. Most large and
file-backed uploads already back off (#20165, #20313, #20772, #21010)
> - A small upload with no source file still had almost no retry. The S3
SDK never resends a stream body, so a stream got one attempt. A Buffer
got the SDK's three closely spaced attempts
> - Hosted production and staging reported `SlowDown` on `put_object`
from request handlers, for example right after a skill import fetched
its files. The error continues on the current production release
> - This pull request holds a small stream's bytes within the existing
buffered-payload budget, and backs off a throttled small upload like a
file upload
> - The benefit is that a short request-rate limit no longer fails a
small save

## Linked Issues or Issue Description

Refs #20165, #20313, #20772, #21010 (the same backoff for other upload
paths).

**What happened?**

A request handler saved a small object with `storage.putFile` and a
stream body (for example a skill package archive), or with a small
Buffer. Object storage answered `503 SlowDown` ("Please reduce your
request rate."). The stream was not sent again. The Buffer was sent
three times within about a second. The request failed with a 500.

**Expected behavior**

A request-rate limit slows the upload down. It does not fail the save.

**Steps to reproduce**

1. Point the S3 provider at an endpoint that answers the first `PUT`
with `503 SlowDown`.
2. Call `putFile` with a small `Readable` body and no source file.
3. On `master` it throws `SlowDown: Please reduce your request rate.`
after one `PUT`.

**Paperclip version or commit**

`master` at `201abacd6e`.

## What Changed

- `server/src/storage/s3-multipart.ts`: `withBufferedUploadBody` holds a
small stream body within the shared buffered-payload budget that
multipart already uses. It checks the declared length.
`sendThrottledRequestWithRetry` uses the same jittered exponential
backoff as file uploads. It resends only request-rate limits (SlowDown,
503, 429).
- `server/src/storage/s3-provider.ts`: the small `putObject` path
without a source file sends Buffer bodies and buffered stream bodies
through that retry. When the budget is full, the stream is sent once, as
before.
- `server/src/__tests__/storage-s3-small-upload-throttle.test.ts` (new):
a real S3 client against a local endpoint.

## Verification

- `cd server && npx vitest run
src/__tests__/storage-s3-small-upload-throttle.test.ts`: with the
`master` source files, 3 of 4 cases fail (the stream and Buffer cases
reject with `SlowDown`). With this change, 4 of 4 pass.
- `cd server && npx vitest run src/__tests__/storage-* src/storage`: 17
files, 135 tests pass on top of `201abacd6e`, including `does not repeat
a write that timed out`.
- I did not run the full server typecheck locally. The sandbox ran out
of memory. CI must cover it.

## Risks

- Low. A small stream is now held in memory, at most 16 MiB, inside the
existing 32 MiB process budget. If the budget is full, behavior is
unchanged.
- Only request-rate limits are resent. The store did not apply those
writes. Other write failures keep their SDK handling, so a timed-out
write is still not repeated.
- A stream that ends early or runs long now fails before any request is
sent.
- Conditional publication puts (`putObjectConditional`) are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with
tool use (shell, file edit, test execution) and read-only Sentry reads.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
doc change is necessary: this is internal storage behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-11-20 05:37:17 -08:00
1 parent 7ef731ec7e
commit fa96716bb5
3 files changed
+219 -7

No files matched your search

@@ -0,0 +1,180 @@
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import type { AddressInfo } from "node:net";
import { Readable } from "node:stream";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { createS3StorageProvider } from "../storage/s3-provider.js";
import { createStorageService } from "../storage/service.js";
type Seen = { method: string; key: string; body: Buffer };
const seen: Seen[] = [];
let respond: (req: IncomingMessage, res: ServerResponse, index: number) => void = (_req, res) => res.end();
const server = createServer((req, res) => {
const chunks: Buffer[] = [];
req.on("data", chunk => chunks.push(chunk as Buffer));
req.on("end", () => {
seen.push({ method: req.method ?? "", key: (req.url ?? "").split("?")[0]!.split("/").pop() ?? "", body: Buffer.concat(chunks) });
respond(req, res, seen.length - 1);
});
});
let endpoint = "";
const savedEnv = { ...process.env };
beforeAll(async () => {
await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
endpoint = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
process.env.AWS_ACCESS_KEY_ID = "fixture-access-key";
process.env.AWS_SECRET_ACCESS_KEY = "fixture-secret-key";
delete process.env.AWS_SESSION_TOKEN;
delete process.env.AWS_PROFILE;
});
afterAll(async () => {
process.env = savedEnv;
server.closeAllConnections();
await new Promise<void>(resolve => server.close(() => resolve()));
});
beforeEach(() => { seen.length = 0; vi.spyOn(Math, "random").mockReturnValue(0); });
afterEach(() => { vi.restoreAllMocks(); });
const SLOW_DOWN = `<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><RequestId>req-503</RequestId></Error>`;
function throttleFirstPuts(count: number) {
respond = (req, res, index) => {
if (req.method === "PUT" && index < count) {
res.writeHead(503, { "content-type": "application/xml" });
res.end(SLOW_DOWN);
return;
}
res.writeHead(200, { etag: '"saved"' });
res.end();
};
}
function storage() {
return createStorageService(createS3StorageProvider({ bucket: "bucket", region: "us-east-1", endpoint, forcePathStyle: true }));
}
const sha256 = "0".repeat(64);
describe("S3 throttling of small uploads without a source file", () => {
it("resends a small streamed upload after SlowDown", async () => {
const content = Buffer.from("skill package archive bytes");
throttleFirstPuts(1);
await expect(storage().putFile({ companyId: "company", namespace: "skill-packages", objectKey: "company/skill-packages/files.tar.gz",
originalFilename: "files.tar.gz", contentType: "application/gzip", body: Readable.from([content]),
byteSize: content.length, sha256 })).resolves.toMatchObject({ byteSize: content.length });
expect(seen.map(entry => entry.method)).toEqual(["PUT", "PUT"]);
expect(seen[1]!.body.equals(content)).toBe(true);
});
it("keeps backing off a small buffered upload after the SDK's own attempts are throttled", async () => {
const content = Buffer.from('{"files":[]}');
throttleFirstPuts(3);
await expect(storage().putFile({ companyId: "company", namespace: "skill-packages", objectKey: "company/skill-packages/manifest.json",
originalFilename: "manifest.json", contentType: "application/json", body: content }))
.resolves.toMatchObject({ byteSize: content.length });
expect(seen.map(entry => entry.method)).toEqual(["PUT", "PUT", "PUT", "PUT"]);
expect(seen[3]!.body.equals(content)).toBe(true);
});
it("rejects a streamed upload that ends before its declared size without sending it", async () => {
throttleFirstPuts(0);
await expect(storage().putFile({ companyId: "company", namespace: "skill-packages", objectKey: "company/skill-packages/files.tar.gz",
originalFilename: "files.tar.gz", contentType: "application/gzip", body: Readable.from([Buffer.from("short")]),
byteSize: 10, sha256 })).rejects.toThrow("S3 upload source was truncated");
expect(seen.filter(entry => entry.method === "PUT")).toEqual([]);
});
it("does not resend a small upload the store refused for good", async () => {
respond = (_req, res) => {
res.writeHead(403, { "content-type": "application/xml" });
res.end(`<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>denied</Message></Error>`);
};
await expect(storage().putFile({ companyId: "company", namespace: "skill-packages", objectKey: "company/skill-packages/files.tar.gz",
originalFilename: "files.tar.gz", contentType: "application/gzip", body: Readable.from([Buffer.from("bytes")]),
byteSize: 5, sha256 })).rejects.toMatchObject({ name: "AccessDenied" });
expect(seen.filter(entry => entry.method === "PUT")).toHaveLength(1);
});
});
describe("S3 small streamed uploads while the buffer budget is full", () => {
const MiB = 1024 * 1024;
const slowDown = (res: ServerResponse) => { res.writeHead(503, { "content-type": "application/xml" }); res.end(SLOW_DOWN); };
const ok = (res: ServerResponse) => { res.writeHead(200, { etag: '"saved"' }); res.end(); };
const putStream = (key: string, content: Buffer, signal?: AbortSignal) => storage().putFile({ companyId: "company",
namespace: "skill-packages", objectKey: `company/skill-packages/${key}`, originalFilename: key, contentType: "application/gzip",
body: Readable.from([content]), byteSize: content.length, sha256, signal });
const awsChunkedPayload = (body: Buffer) => {
const chunks: Buffer[] = [];
for (let at = 0; at < body.length;) {
const end = body.indexOf("\r\n", at);
const size = end < 0 ? Number.NaN : Number.parseInt(body.subarray(at, end).toString(), 16);
if (!Number.isInteger(size) || size < 0) return undefined;
if (size === 0) return Buffer.concat(chunks);
chunks.push(body.subarray(end + 2, end + 2 + size));
at = end + 4 + size;
}
return undefined;
};
const puts = (key: string) => seen.filter(entry => entry.method === "PUT" && entry.key === key);
it("streams another small upload once, intact, and gets the budget back from a failed and a cancelled upload", async () => {
const held = new Map<string, ServerResponse>();
let bothHeld!: () => void;
const holding = new Promise<void>(resolve => { bothHeld = resolve; });
respond = (req, res) => {
if (req.method === "PUT" && seen.at(-1)!.key.startsWith("held-")) {
held.set(seen.at(-1)!.key, res);
if (held.size === 2) bothHeld();
return;
}
if (req.method === "PUT" && seen.at(-1)!.key === "throttled") return slowDown(res);
ok(res);
};
const cancel = new AbortController();
const failing = putStream("held-failing", Buffer.alloc(16 * MiB, 1));
const cancelled = putStream("held-cancelled", Buffer.alloc(16 * MiB, 2), cancel.signal);
failing.catch(() => undefined);
cancelled.catch(() => undefined);
await holding;
const content = Buffer.from("skill package archive bytes");
await expect(putStream("fallback", content)).resolves.toMatchObject({ byteSize: content.length });
expect(puts("fallback").length).toBe(1);
expect(awsChunkedPayload(puts("fallback")[0]!.body)?.equals(content)).toBe(true);
await expect(putStream("throttled", content)).rejects.toMatchObject({ name: "SlowDown" });
expect(puts("throttled").length).toBe(1);
const refused = held.get("held-failing")!;
refused.writeHead(403, { "content-type": "application/xml" });
refused.end(`<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>denied</Message></Error>`);
await expect(failing).rejects.toMatchObject({ name: "AccessDenied" });
cancel.abort(new Error("upload cancelled"));
await expect(cancelled).rejects.toThrow();
held.get("held-cancelled")!.destroy();
const throttledOnce = new Set<string>();
respond = (req, res) => {
const key = seen.at(-1)!.key;
if (req.method === "PUT" && !throttledOnce.has(key)) { throttledOnce.add(key); return slowDown(res); }
ok(res);
};
const first = Buffer.alloc(16 * MiB, 3);
const second = Buffer.alloc(16 * MiB, 4);
const outcomes = await Promise.allSettled([putStream("after-first", first), putStream("after-second", second)]);
expect(outcomes.map(outcome => outcome.status === "fulfilled" ? "saved" : (outcome.reason as Error).name)).toEqual(["saved", "saved"]);
expect(puts("after-first").length).toBe(2);
expect(puts("after-second").length).toBe(2);
expect(puts("after-first")[1]!.body.equals(first)).toBe(true);
expect(puts("after-second")[1]!.body.equals(second)).toBe(true);
});
});
+30 -3
View File
@@ -4,7 +4,7 @@ import { recordStorageBackoff, recordStoragePart, recordStorageThrottle, withSto
import { once } from "node:events";
import { createReadStream } from "node:fs";
import { stat } from "node:fs/promises";
import { PassThrough, Readable } from "node:stream";
import { addAbortSignal, PassThrough, Readable } from "node:stream";
import { setTimeout as delay } from "node:timers/promises";
import type { PutObjectInput, PutObjectReceipt } from "./types.js";
import { withS3FailureDiagnostics } from "./s3-failure.js";
@@ -186,7 +186,15 @@ function isTransientFileRequestFailure(error: unknown) {
|| (typeof name === "string" && TRANSIENT_FILE_REQUEST_NAMES.has(name));
}
export async function sendFileRequestWithRetry<T>(signal: AbortSignal | undefined, send: () => Promise<T>): Promise<T> {
export function sendFileRequestWithRetry<T>(signal: AbortSignal | undefined, send: () => Promise<T>): Promise<T> {
return sendRequestWithRetry(signal, send, true);
}
export function sendThrottledRequestWithRetry<T>(signal: AbortSignal | undefined, send: () => Promise<T>): Promise<T> {
return sendRequestWithRetry(signal, send, false);
}
async function sendRequestWithRetry<T>(signal: AbortSignal | undefined, send: () => Promise<T>, retryTransient: boolean): Promise<T> {
for (let failures = 0, throttles = 0; ;) {
signal?.throwIfAborted();
try {
@@ -202,13 +210,32 @@ export async function sendFileRequestWithRetry<T>(signal: AbortSignal | undefine
await delay(wait, undefined, { signal });
continue;
}
if (!isTransientFileRequestFailure(error) || ++failures >= FILE_PART_ATTEMPTS) throw error;
if (!retryTransient || !isTransientFileRequestFailure(error) || ++failures >= FILE_PART_ATTEMPTS) throw error;
recordStorageBackoff(failures * 500);
await delay(failures * 500, undefined, { signal });
}
}
}
export async function withBufferedUploadBody(source: Readable, length: number, signal: AbortSignal | undefined,
send: (body: Buffer) => Promise<void>): Promise<boolean> {
if (bufferedPayloadBytes + length > bufferedPayloadLimit) return false;
bufferedPayloadBytes += length;
try {
const payload = Buffer.allocUnsafe(length);
let position = 0;
for await (const chunk of signal ? addAbortSignal(signal, source) : source) {
const bytes = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk as Uint8Array);
if (position + bytes.length > length) throw new Error("S3 upload source exceeds its declared size");
bytes.copy(payload, position);
position += bytes.length;
}
if (position !== length) throw new Error("S3 upload source was truncated");
await send(payload);
return true;
} finally { bufferedPayloadBytes -= length; }
}
/** Base64 MD5 of an inclusive byte range, as the Content-MD5 header expects. Hashed off the server's thread. */
export async function fileRangeMd5(file: string, start: number, end: number, signal?: AbortSignal) {
const digest = await digestFileOffThread({ file, start, end, algorithm: "md5" }, signal);
+9 -4
View File
@@ -9,7 +9,8 @@ import {
ListMultipartUploadsCommand,
AbortMultipartUploadCommand,
} from "@aws-sdk/client-s3";
import { assertSourceFileSize, fileRangeMd5, putS3Multipart, putS3MultipartFile, sendFileRequestWithRetry } from "./s3-multipart.js";
import { assertSourceFileSize, fileRangeMd5, putS3Multipart, putS3MultipartFile, sendFileRequestWithRetry,
sendThrottledRequestWithRetry, withBufferedUploadBody } from "./s3-multipart.js";
import { createReadStream } from "node:fs";
import { createHash } from "node:crypto";
import { recordS3BodyFailure, withS3FailureDiagnostics } from "./s3-failure.js";
@@ -271,17 +272,21 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
await putS3Multipart(client, bucket, key, input);
return;
}
const send = () => withS3FailureDiagnostics("put_object", () => client.send(
const send = (body: Buffer | Readable) => withS3FailureDiagnostics("put_object", () => client.send(
new PutObjectCommand({
Bucket: bucket,
Key: key,
Body: input.body,
Body: body,
ContentType: input.contentType,
ContentLength: input.contentLength,
}),
{ abortSignal: input.signal },
), input.signal);
await (Buffer.isBuffer(input.body) ? sendThroughStorageOutage(send, input.signal) : send());
const sendBytes = async (body: Buffer) => {
await sendThroughStorageOutage(() => sendThrottledRequestWithRetry(input.signal, () => send(body)), input.signal);
};
if (Buffer.isBuffer(input.body)) return sendBytes(input.body);
if (!(await withBufferedUploadBody(input.body, input.contentLength, input.signal, sendBytes))) await send(input.body);
},
async getObject(input): Promise<GetObjectResult> {