Files
PaperClipAI/server/src/storage/s3-multipart.ts
T
fa96716bb5 fix(storage): back off a throttled small upload that has no source file (#22404)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server saves many objects to S3-compatible storage: skill
packages, attachments, recovery archives and manifests
> - Object storage answers a burst with `503 SlowDown`. Most large and
file-backed uploads already back off (#20165, #20313, #20772, #21010)
> - A small upload with no source file still had almost no retry. The S3
SDK never resends a stream body, so a stream got one attempt. A Buffer
got the SDK's three closely spaced attempts
> - Hosted production and staging reported `SlowDown` on `put_object`
from request handlers, for example right after a skill import fetched
its files. The error continues on the current production release
> - This pull request holds a small stream's bytes within the existing
buffered-payload budget, and backs off a throttled small upload like a
file upload
> - The benefit is that a short request-rate limit no longer fails a
small save

## Linked Issues or Issue Description

Refs #20165, #20313, #20772, #21010 (the same backoff for other upload
paths).

**What happened?**

A request handler saved a small object with `storage.putFile` and a
stream body (for example a skill package archive), or with a small
Buffer. Object storage answered `503 SlowDown` ("Please reduce your
request rate."). The stream was not sent again. The Buffer was sent
three times within about a second. The request failed with a 500.

**Expected behavior**

A request-rate limit slows the upload down. It does not fail the save.

**Steps to reproduce**

1. Point the S3 provider at an endpoint that answers the first `PUT`
with `503 SlowDown`.
2. Call `putFile` with a small `Readable` body and no source file.
3. On `master` it throws `SlowDown: Please reduce your request rate.`
after one `PUT`.

**Paperclip version or commit**

`master` at `201abacd6e`.

## What Changed

- `server/src/storage/s3-multipart.ts`: `withBufferedUploadBody` holds a
small stream body within the shared buffered-payload budget that
multipart already uses. It checks the declared length.
`sendThrottledRequestWithRetry` uses the same jittered exponential
backoff as file uploads. It resends only request-rate limits (SlowDown,
503, 429).
- `server/src/storage/s3-provider.ts`: the small `putObject` path
without a source file sends Buffer bodies and buffered stream bodies
through that retry. When the budget is full, the stream is sent once, as
before.
- `server/src/__tests__/storage-s3-small-upload-throttle.test.ts` (new):
a real S3 client against a local endpoint.

## Verification

- `cd server && npx vitest run
src/__tests__/storage-s3-small-upload-throttle.test.ts`: with the
`master` source files, 3 of 4 cases fail (the stream and Buffer cases
reject with `SlowDown`). With this change, 4 of 4 pass.
- `cd server && npx vitest run src/__tests__/storage-* src/storage`: 17
files, 135 tests pass on top of `201abacd6e`, including `does not repeat
a write that timed out`.
- I did not run the full server typecheck locally. The sandbox ran out
of memory. CI must cover it.

## Risks

- Low. A small stream is now held in memory, at most 16 MiB, inside the
existing 32 MiB process budget. If the budget is full, behavior is
unchanged.
- Only request-rate limits are resent. The store did not apply those
writes. Other write failures keep their SDK handling, so a timed-out
write is still not repeated.
- A stream that ends early or runs long now fails before any request is
sent.
- Conditional publication puts (`putObjectConditional`) are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with
tool use (shell, file edit, test execution) and read-only Sentry reads.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
doc change is necessary: this is internal storage behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
2026-11-20 05:37:17 -08:00

349 lines
19 KiB
TypeScript

import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, type S3Client } from "@aws-sdk/client-s3";
import { sendThroughStorageOutage } from "./storage-unavailable.js";
import { recordStorageBackoff, recordStoragePart, recordStorageThrottle, withStorageRequestStats } from "./request-stats.js";
import { once } from "node:events";
import { createReadStream } from "node:fs";
import { stat } from "node:fs/promises";
import { addAbortSignal, PassThrough, Readable } from "node:stream";
import { setTimeout as delay } from "node:timers/promises";
import type { PutObjectInput, PutObjectReceipt } from "./types.js";
import { withS3FailureDiagnostics } from "./s3-failure.js";
import { cleanupStorageFailure, STORAGE_FAILURE_CLEANUP_MS } from "./failure-cleanup.js";
import { digestFileOffThread } from "./file-crypto-pool.js";
const bufferedPartSize = 8 * 1024 * 1024;
const bufferedPayloadLimit = 4 * bufferedPartSize;
let bufferedPayloadBytes = 0;
const inFlightUploads = new Set<{ abort: (reason: unknown) => void; settled: Promise<void> }>();
function trackUpload(abort: (reason: unknown) => void) {
let settle!: () => void;
const entry = { abort, settled: new Promise<void>(resolve => { settle = resolve; }) };
inFlightUploads.add(entry);
return () => { inFlightUploads.delete(entry); settle(); };
}
function settledWithin(entries: Array<{ settled: Promise<void> }>, ms: number) {
let timer: ReturnType<typeof setTimeout> | undefined;
return Promise.race([Promise.all(entries.map(entry => entry.settled)).then(() => undefined),
new Promise<void>(resolve => { timer = setTimeout(resolve, ms); })]).finally(() => clearTimeout(timer));
}
export async function abortInFlightMultipartUploads(reason: unknown, options: { graceMs?: number; timeoutMs?: number } = {}) {
if (options.graceMs && inFlightUploads.size) await settledWithin([...inFlightUploads], options.graceMs);
const entries = [...inFlightUploads];
if (!entries.length) return 0;
for (const entry of entries) entry.abort(reason);
await settledWithin(entries, options.timeoutMs ?? 2 * STORAGE_FAILURE_CLEANUP_MS);
return entries.length;
}
/** Each part is streamed too: multipart must not become another whole-file
* buffer or a fixed part-count ceiling below the storage provider's limits. */
export async function putS3Multipart(client: S3Client, bucket: string, key: string, input: PutObjectInput) {
const source = input.body instanceof Readable ? input.body : Readable.from([input.body]);
const iterator = source[Symbol.asyncIterator]();
const controller = new AbortController();
const active = new Set<Promise<void>>();
const bodies = new Set<PassThrough>();
let failed = false;
let failure: unknown;
const stop = (error: unknown) => {
if (failed) return;
failed = true;
failure = error;
controller.abort(error);
source.destroy();
for (const body of bodies) body.destroy();
};
const check = () => { if (failed) throw failure; };
const onAbort = () => stop(input.signal!.reason);
input.signal?.addEventListener("abort", onAbort, { once: true });
if (input.signal?.aborted) onAbort();
const untrack = trackUpload(stop);
const readNext = async () => {
controller.signal.throwIfAborted();
let abort!: () => void;
const interrupted = new Promise<never>((_resolve, reject) => {
abort = () => reject(controller.signal.reason);
controller.signal.addEventListener("abort", abort, { once: true });
});
try { return await Promise.race([iterator.next(), interrupted]); }
finally { controller.signal.removeEventListener("abort", abort); }
};
let pending = Buffer.alloc(0);
const readPart = async (length: number, write: (chunk: Buffer) => Promise<void>) => {
while (length > 0) {
check();
if (!pending.length) {
const next = await readNext();
check();
if (next.done) throw new Error("S3 upload source was truncated");
pending = Buffer.isBuffer(next.value) ? next.value : Buffer.from(next.value);
if (!pending.length) continue;
}
const count = Math.min(length, pending.length, 64 * 1024), chunk = pending.subarray(0, count);
pending = pending.subarray(count); length -= count;
await write(chunk);
}
};
let uploadId: string | undefined;
// Scale part sizes to the S3 part-count limit while retaining stream backpressure.
const partSize = Math.max(bufferedPartSize, Math.ceil(input.contentLength / 10_000));
try {
check();
uploadId = (await withS3FailureDiagnostics("create_multipart_upload", () => client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }), { abortSignal: controller.signal }), controller.signal)).UploadId;
if (!uploadId) throw new Error("S3 did not return a multipart upload id");
const parts: { ETag: string; PartNumber: number }[] = [];
let partNumber = 0;
for (let offset = 0; offset < input.contentLength;) {
while (active.size >= 4) { await Promise.race(active); check(); }
check();
const length = Math.min(partSize, input.contentLength - offset);
const number = ++partNumber;
let buffered: Buffer | null = null;
if (partSize <= bufferedPartSize && bufferedPayloadBytes + length <= bufferedPayloadLimit) {
bufferedPayloadBytes += length;
try {
const payload = Buffer.allocUnsafe(length);
let position = 0;
await readPart(length, async chunk => { chunk.copy(payload, position); position += chunk.length; });
buffered = payload;
} catch (error) { bufferedPayloadBytes -= length; throw error; }
}
let remaining = length;
const body = new PassThrough({ highWaterMark: 64 * 1024 });
bodies.add(body);
const write = async (chunk: Buffer) => {
check(); remaining -= chunk.length;
if (!body.write(chunk)) await once(body, "drain", { signal: controller.signal });
};
const sending = (async () => {
const result = await withS3FailureDiagnostics("upload_part", () => client.send(new UploadPartCommand({ Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: number, Body: body, ContentLength: length }), { abortSignal: controller.signal }), controller.signal);
if (remaining || !body.readableEnded || !result.ETag) throw new Error("S3 upload part is incomplete");
parts[number - 1] = { ETag: result.ETag, PartNumber: number };
})().catch(stop);
const producing = buffered ? (async () => {
for (let position = 0; position < buffered.length; position += 64 * 1024)
await write(buffered.subarray(position, Math.min(position + 64 * 1024, buffered.length)));
body.end();
})().catch(stop) : Promise.resolve();
const uploading = Promise.all([sending, producing]).then(() => undefined)
.finally(() => { if (buffered) bufferedPayloadBytes -= length; active.delete(uploading); bodies.delete(body); body.destroy(); });
active.add(uploading);
if (!buffered) {
await readPart(length, write);
body.end();
}
offset += length;
}
if (pending.length || !(await readNext()).done) throw new Error("S3 upload source exceeds its declared size");
await Promise.all(active);
check();
await withS3FailureDiagnostics("complete_multipart_upload", () => client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts } }), { abortSignal: controller.signal }), controller.signal);
} catch (error) {
stop(error);
await Promise.all(active);
if (uploadId) await cleanupStorageFailure(signal => withStorageRequestStats(() => client.send(new AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId }), { abortSignal: signal })));
throw failure;
} finally {
input.signal?.removeEventListener("abort", onAbort);
source.destroy();
await cleanupStorageFailure(async () => iterator.return?.());
untrack();
}
}
export const FILE_PART_CONCURRENCY = 8;
const FILE_PART_ATTEMPTS = 3;
/** A provider that asks for a lower request rate gets more, longer-spaced attempts. */
export const THROTTLED_PART_ATTEMPTS = 8;
const THROTTLE_BACKOFF_BASE_MS = 1_000;
const THROTTLE_BACKOFF_MAX_MS = 20_000;
/**
* S3 answers a request-rate limit with 503 SlowDown; compatible gateways also
* use 429. The SDK never retries a part because its body is a stream, so the
* upload must slow down itself.
*/
function isRequestRateLimited(error: unknown) {
if (!error || typeof error !== "object") return false;
const { name, $metadata } = error as { name?: unknown; $metadata?: { httpStatusCode?: unknown } };
return name === "SlowDown" || $metadata?.httpStatusCode === 503 || $metadata?.httpStatusCode === 429;
}
const TRANSIENT_FILE_REQUEST_STATUS = new Set([408, 429, 500, 502, 503, 504]);
const TRANSIENT_FILE_REQUEST_CODES = new Set(["ECONNRESET", "ECONNREFUSED", "ECONNABORTED", "ETIMEDOUT", "EPIPE", "EAI_AGAIN",
"ENOTFOUND", "ERR_STREAM_PREMATURE_CLOSE", "UND_ERR_SOCKET"]);
const TRANSIENT_FILE_REQUEST_NAMES = new Set(["RequestTimeout", "TimeoutError", "NetworkingError", "InternalError", "ServiceUnavailable"]);
function isTransientFileRequestFailure(error: unknown) {
if (!error || typeof error !== "object") return false;
const { name, code, $metadata } = error as { name?: unknown; code?: unknown; $metadata?: { httpStatusCode?: unknown } };
const status = $metadata?.httpStatusCode;
if (typeof status === "number") return TRANSIENT_FILE_REQUEST_STATUS.has(status);
return (typeof code === "string" && TRANSIENT_FILE_REQUEST_CODES.has(code))
|| (typeof name === "string" && TRANSIENT_FILE_REQUEST_NAMES.has(name));
}
export function sendFileRequestWithRetry<T>(signal: AbortSignal | undefined, send: () => Promise<T>): Promise<T> {
return sendRequestWithRetry(signal, send, true);
}
export function sendThrottledRequestWithRetry<T>(signal: AbortSignal | undefined, send: () => Promise<T>): Promise<T> {
return sendRequestWithRetry(signal, send, false);
}
async function sendRequestWithRetry<T>(signal: AbortSignal | undefined, send: () => Promise<T>, retryTransient: boolean): Promise<T> {
for (let failures = 0, throttles = 0; ;) {
signal?.throwIfAborted();
try {
return await send();
} catch (error) {
if (signal?.aborted) throw error;
if (isRequestRateLimited(error)) {
recordStorageThrottle();
if (++throttles >= THROTTLED_PART_ATTEMPTS) throw error;
const ceiling = Math.min(THROTTLE_BACKOFF_MAX_MS, THROTTLE_BACKOFF_BASE_MS * 2 ** (throttles - 1));
const wait = Math.floor(Math.random() * ceiling);
recordStorageBackoff(wait);
await delay(wait, undefined, { signal });
continue;
}
if (!retryTransient || !isTransientFileRequestFailure(error) || ++failures >= FILE_PART_ATTEMPTS) throw error;
recordStorageBackoff(failures * 500);
await delay(failures * 500, undefined, { signal });
}
}
}
export async function withBufferedUploadBody(source: Readable, length: number, signal: AbortSignal | undefined,
send: (body: Buffer) => Promise<void>): Promise<boolean> {
if (bufferedPayloadBytes + length > bufferedPayloadLimit) return false;
bufferedPayloadBytes += length;
try {
const payload = Buffer.allocUnsafe(length);
let position = 0;
for await (const chunk of signal ? addAbortSignal(signal, source) : source) {
const bytes = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk as Uint8Array);
if (position + bytes.length > length) throw new Error("S3 upload source exceeds its declared size");
bytes.copy(payload, position);
position += bytes.length;
}
if (position !== length) throw new Error("S3 upload source was truncated");
await send(payload);
return true;
} finally { bufferedPayloadBytes -= length; }
}
/** Base64 MD5 of an inclusive byte range, as the Content-MD5 header expects. Hashed off the server's thread. */
export async function fileRangeMd5(file: string, start: number, end: number, signal?: AbortSignal) {
const digest = await digestFileOffThread({ file, start, end, algorithm: "md5" }, signal);
return Buffer.from(digest.hex, "hex").toString("base64");
}
export async function assertSourceFileSize(file: string, contentLength: number) {
const stats = await stat(file);
if (!stats.isFile() || stats.size !== contentLength) throw new Error("S3 upload source file does not match its declared size");
}
/** A file source is read per part instead of through one stream. Parts upload
* in parallel without the shared memory window, each carries Content-MD5 so
* the backend rejects bytes it did not receive intact, and a failed part is
* resent from disk instead of failing the whole upload. When the backend
* limits the request rate, the part backs off with jitter and the upload
* halves its parts in flight, once per round of throttled requests. */
export async function putS3MultipartFile(
client: S3Client, bucket: string, key: string, input: PutObjectInput & { sourceFile: string },
): Promise<PutObjectReceipt> {
const file = input.sourceFile;
const controller = new AbortController();
const onAbort = () => controller.abort(input.signal!.reason);
input.signal?.addEventListener("abort", onAbort, { once: true });
if (input.signal?.aborted) onAbort();
const untrack = trackUpload(reason => { if (!controller.signal.aborted) controller.abort(reason); });
// This path never reads the stream body; release its file descriptor now.
if (input.body instanceof Readable) input.body.destroy();
let uploadId: string | undefined;
try {
await assertSourceFileSize(file, input.contentLength);
controller.signal.throwIfAborted();
const partSize = Math.max(bufferedPartSize, Math.ceil(input.contentLength / 10_000));
const count = Math.ceil(input.contentLength / partSize);
uploadId = (await sendThroughStorageOutage(() => withS3FailureDiagnostics("create_multipart_upload", () => client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }), { abortSignal: controller.signal }), controller.signal), controller.signal)).UploadId;
if (!uploadId) throw new Error("S3 did not return a multipart upload id");
const id = uploadId;
const parts: { ETag: string; PartNumber: number }[] = new Array(count);
// Workers at or above this slot stop taking new parts. A reduction counts
// only for requests sent after the previous one, so a burst of parallel
// throttled parts halves the rate once rather than once per part.
let concurrency = FILE_PART_CONCURRENCY, reductions = 0;
const partMd5 = (index: number) => fileRangeMd5(file, index * partSize, Math.min((index + 1) * partSize, input.contentLength) - 1, controller.signal);
// The first round of parts is digested before any of them is sent. They
// leave together, as they did when this thread digested them side by side,
// so a rate limit still answers one burst rather than a trickle of requests.
// Every digest of the round has ended, also a failed or cancelled one,
// before a part goes on: none still reads the file when the upload settles.
const firstRound = Array.from({ length: Math.min(FILE_PART_CONCURRENCY, count) }, (_, index) => partMd5(index));
const firstRoundEnded = Promise.allSettled(firstRound);
const sendPart = async (index: number) => {
const start = index * partSize, length = Math.min(partSize, input.contentLength - start), end = start + length - 1;
if (index < firstRound.length) await firstRoundEnded;
const contentMd5 = await (firstRound[index] ?? partMd5(index));
for (let failures = 0, throttles = 0; ;) {
controller.signal.throwIfAborted();
const body = createReadStream(file, { start, end, highWaterMark: 64 * 1024 });
const sentAfter = reductions;
try {
const result = await withS3FailureDiagnostics("upload_part", () => client.send(new UploadPartCommand({
Bucket: bucket, Key: key, UploadId: id, PartNumber: index + 1, Body: body, ContentLength: length, ContentMD5: contentMd5,
}), { abortSignal: controller.signal }), controller.signal);
if (!result.ETag) throw new Error("S3 upload part is incomplete");
parts[index] = { ETag: result.ETag, PartNumber: index + 1 };
recordStoragePart();
return;
} catch (error) {
// A part read from disk can be resent; the stream path cannot.
if (controller.signal.aborted) throw error;
if (isRequestRateLimited(error)) {
recordStorageThrottle();
if (++throttles >= THROTTLED_PART_ATTEMPTS) throw error;
if (sentAfter === reductions) { concurrency = Math.max(1, Math.floor(concurrency / 2)); reductions++; }
const ceiling = Math.min(THROTTLE_BACKOFF_MAX_MS, THROTTLE_BACKOFF_BASE_MS * 2 ** (throttles - 1));
const wait = Math.floor(Math.random() * ceiling);
recordStorageBackoff(wait);
await delay(wait, undefined, { signal: controller.signal });
continue;
}
if (++failures >= FILE_PART_ATTEMPTS) throw error;
recordStorageBackoff(failures * 500);
await delay(failures * 500, undefined, { signal: controller.signal });
} finally { body.destroy(); }
}
};
let next = 0;
const worker = async (slot: number) => {
while (next < count && slot < concurrency) {
const index = next++;
try { await sendThroughStorageOutage(() => sendPart(index), controller.signal); }
catch (error) { if (!controller.signal.aborted) controller.abort(error); throw error; }
}
};
// Wait for every sibling to settle before the multipart upload is aborted.
const settled = await Promise.allSettled(Array.from({ length: Math.min(FILE_PART_CONCURRENCY, count) }, (_, slot) => worker(slot)));
const rejected = settled.find((entry): entry is PromiseRejectedResult => entry.status === "rejected");
if (rejected) throw controller.signal.aborted && controller.signal.reason !== undefined ? controller.signal.reason : rejected.reason;
controller.signal.throwIfAborted();
await sendThroughStorageOutage(() => withS3FailureDiagnostics("complete_multipart_upload", () => client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: id, MultipartUpload: { Parts: parts } }), { abortSignal: controller.signal }), controller.signal), controller.signal);
uploadId = undefined;
return { integrity: "server_verified_md5" };
} catch (error) {
const failedUpload = uploadId;
if (failedUpload) await cleanupStorageFailure(signal => withStorageRequestStats(() => client.send(new AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: failedUpload }), { abortSignal: signal })));
throw error;
} finally {
input.signal?.removeEventListener("abort", onAbort);
untrack();
}
}