mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-15 18:56:05 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces save their files as encrypted snapshots in object
storage. Snapshots over 16 MiB use S3 multipart uploads.
> - A server restart can cut off a snapshot upload in the middle of an
`UploadPart` request.
> - The server aborts a multipart upload only in-process, on an error or
an abort signal. A SIGKILL can never do that. On SIGTERM, the drain step
timed out while the store held the part, and the process exited without
`AbortMultipartUpload`.
> - The incomplete upload then stays in the bucket. Nothing records its
upload id and nothing finds it again, so incomplete uploads pile up and
the operator pays for their parts.
> - This pull request aborts the incomplete uploads of each save attempt
when that attempt can no longer win, with no polling and no schema
change.
> - The benefit is that no incomplete upload outlives its save attempt.
## Linked Issues or Issue Description
**What happened?**
A restart-chaos wave (Daytona, R1 workload, storage fault ST4,
`--storage-trigger snapshot-part`, faults K1 and K2, master
`fea4530a30`) passed all 3 tasks. All 3 committed snapshots read back
intact. But the storage audit found 3 incomplete multipart uploads. Each
one was the exact upload that a restart cut off in the middle of
`UploadPart`. The retried save then published a different attempt of the
same snapshot.
**Expected behavior**
When a save attempt can no longer become the snapshot's object, its
incomplete multipart uploads are aborted. The storage audit shows 0
incomplete uploads after the wave.
**Steps to reproduce**
1. Run a snapshot save larger than 16 MiB against S3.
2. Kill the server (SIGKILL), or send SIGTERM while the store holds an
`UploadPart` request.
3. Let the save retry and publish.
4. List the bucket's multipart uploads. The cut-off attempt's upload is
still there.
**Paperclip version or commit**
master `fea4530a30` and later.
**Relevant logs or output**
`shutdown step timed out; continuing shutdown without it
{"step":"delivery_work_stop","durationMs":20001}`, then `Paperclip did
not stop within 30000ms; sending SIGKILL`.
## What Changed
- The S3 storage provider and the storage service get
`abortIncompleteUploads(objectKey)`. It lists the multipart uploads
under that exact key (`ListMultipartUploads` with the key as the
prefix), skips other keys that only share the prefix, and aborts each
one. `NoSuchUpload` counts as already done. The local-disk provider has
no multipart uploads and does nothing.
- Save publication: the save journals every attempt's object key in
`metadata.uploads` before any remote I/O. When one attempt moves the
snapshot from `pending` to `uploaded`, no other attempt can win any
more. The winning process now aborts the incomplete uploads of every
other journaled attempt key. Each abort is best effort and bounded to 5
s. A snapshot with one attempt sends no extra request.
- A losing uploader that is still live: if its upload fails after
another attempt has published the snapshot, it now returns the published
snapshot (after it verifies it) instead of the error. So an abort from
the winner never fails a save that is still running in another process.
- Snapshot retention: before it deletes a retired snapshot's objects
(this includes saves that #20502 retires as abandoned), retention aborts
the incomplete uploads of each attempt key. A store that refuses the
listing (for example a missing `s3:ListBucketMultipartUploads`
permission) logs a warning and does not block the deletion.
- Abandoned saves: when the ownership reconciler retires an abandoned
save (`remote_snapshot_save` and `remote_snapshot_ledger` settlement,
from #20502), it now aborts the incomplete uploads of every journaled
attempt key after its transaction commits. The owner run is terminal and
the save work is settled, so no live uploader can own those uploads.
This is the durable path for SIGKILL: the reconciler pass after boot
settles the dead save and cleans its uploads.
- SIGTERM drain: the multipart upload code keeps an in-process set of
running uploads. Shutdown aborts them right away when
`delivery_work_stop` times out. It also aborts them after the heartbeat
run drain, before the finalizer and workspace-recovery waits, after a 5
s grace for them to finish. It waits at most 10 s for the
`AbortMultipartUpload` calls. Shutdown does nothing extra when no upload
runs.
- No polling and no periodic sweep. All triggers are events that already
exist: publication, abandonment settlement, retention of a retired
snapshot, and shutdown.
Previous-release story (`rules/upgrade-safe-run-state.mdc`): this adds
no persisted state and changes no saved shape. A snapshot that the
previous release saved with cut-off attempts in `metadata.uploads`
carries forward. Its incomplete uploads are aborted when the new release
publishes a retry of that snapshot, or when retention retires the
snapshot.
## Verification
- Red before, green after (all fail on master, pass with this change):
- `server/src/services/remote-workspace-snapshot-storage.test.ts`:
"aborts the incomplete upload of an attempt a restart cut off once a
retried save publishes" and "adopts the published save when another
uploader aborts this attempt's upload".
- `server/src/services/remote-workspace-snapshot-retention.test.ts`:
"aborts each attempt's incomplete multipart upload before deleting a
retired snapshot's objects". A second test proves that a refused listing
does not block retention.
- `server/src/__tests__/storage-s3-multipart-shutdown.test.ts`: file and
streamed uploads whose part the store holds are aborted at shutdown, and
the provider aborts only the exact key across listing pages.
- `server/src/__tests__/remote-snapshot-ledger-settlement.test.ts`:
"aborts the incomplete multipart uploads of every attempt when the
reconciler retires an abandoned save" (both settlement kinds; a key
outside the snapshot's prefix is skipped).
- `tests/restart-chaos/storage-orphan-uploads.test.ts` (MinIO): the real
provider aborts both cut-off uploads of one attempt key, and keeps a
sibling attempt and a longer key that shares the prefix.
- `cd server && npx vitest run src/__tests__/storage-s3 src/storage
src/__tests__/remote-snapshot-ledger-settlement.test.ts
src/services/remote-workspace-snapshot
src/services/remote-workspace-transport.test.ts
src/__tests__/remote-workspace-file-resources.test.ts`: 19 files, 517
tests pass; with `src/services/recovery`, `src/__tests__/ownership*` and
`heartbeat-workspace-busy.test.ts`, 32 files and 580 tests pass.
- `cd server && npx tsc --noEmit` and `node
scripts/typecheck-test-projects.mjs tests/restart-chaos` pass.
- Live proof, Daytona R1, ST4, `--storage-trigger snapshot-part`, K1/K2,
seed 1033, the evidence wave's flags, image
`paperclip-daytona-runner@sha256:05a4a092…` (controllerMatch: match):
- Before (master `fea4530a30`): `w-20261110T130541-s1033`: 3/3 pass, **3
incomplete multipart uploads**.
- `w-20261110T134623-s1033` on an earlier revision of this PR (publish +
retention hooks only): 2 incomplete uploads remained. Those saves were
abandoned, not retried, so neither hook ran. This led to the settlement
hook.
- After (head `8bc8fc66d0`): `w-20261110T141144-s1033`: 3/3 tasks pass,
exit 0, every fired restart proved it cut the held `UploadPart`, the
reconciler settled 3 abandoned saves, shutdown logged one in-flight
abort, and the storage audit shows **0 incomplete multipart uploads**
and 3 committed snapshots that read back intact.
## Risks
- A live uploader in another process can lose its incomplete upload when
a different attempt publishes first. That uploader now adopts the
published snapshot, so its save still succeeds.
- Narrow gap: an uploader that recorded its attempt but had not yet sent
`CreateMultipartUpload` when the winner published, and that then crashes
during its upload, leaves an incomplete upload until retention retires
the snapshot. An S3 lifecycle rule for incomplete uploads stays a good
backstop.
- Stores without `ListMultipartUploads` permission log a warning and
keep the previous behavior.
- Shutdown can take up to 10 s longer, and only when an upload is still
running after the drain.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
315 lines
17 KiB
TypeScript
315 lines
17 KiB
TypeScript
import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, type S3Client } from "@aws-sdk/client-s3";
|
|
import { sendThroughStorageOutage } from "./storage-unavailable.js";
|
|
import { recordStorageBackoff, recordStoragePart, recordStorageThrottle, withStorageRequestStats } from "./request-stats.js";
|
|
import { createHash } from "node:crypto";
|
|
import { once } from "node:events";
|
|
import { createReadStream } from "node:fs";
|
|
import { stat } from "node:fs/promises";
|
|
import { PassThrough, Readable } from "node:stream";
|
|
import { setTimeout as delay } from "node:timers/promises";
|
|
import type { PutObjectInput, PutObjectReceipt } from "./types.js";
|
|
import { withS3FailureDiagnostics } from "./s3-failure.js";
|
|
import { cleanupStorageFailure, STORAGE_FAILURE_CLEANUP_MS } from "./failure-cleanup.js";
|
|
|
|
const bufferedPartSize = 8 * 1024 * 1024;
|
|
const bufferedPayloadLimit = 4 * bufferedPartSize;
|
|
let bufferedPayloadBytes = 0;
|
|
const inFlightUploads = new Set<{ abort: (reason: unknown) => void; settled: Promise<void> }>();
|
|
|
|
function trackUpload(abort: (reason: unknown) => void) {
|
|
let settle!: () => void;
|
|
const entry = { abort, settled: new Promise<void>(resolve => { settle = resolve; }) };
|
|
inFlightUploads.add(entry);
|
|
return () => { inFlightUploads.delete(entry); settle(); };
|
|
}
|
|
|
|
function settledWithin(entries: Array<{ settled: Promise<void> }>, ms: number) {
|
|
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
return Promise.race([Promise.all(entries.map(entry => entry.settled)).then(() => undefined),
|
|
new Promise<void>(resolve => { timer = setTimeout(resolve, ms); })]).finally(() => clearTimeout(timer));
|
|
}
|
|
|
|
export async function abortInFlightMultipartUploads(reason: unknown, options: { graceMs?: number; timeoutMs?: number } = {}) {
|
|
if (options.graceMs && inFlightUploads.size) await settledWithin([...inFlightUploads], options.graceMs);
|
|
const entries = [...inFlightUploads];
|
|
if (!entries.length) return 0;
|
|
for (const entry of entries) entry.abort(reason);
|
|
await settledWithin(entries, options.timeoutMs ?? 2 * STORAGE_FAILURE_CLEANUP_MS);
|
|
return entries.length;
|
|
}
|
|
|
|
/** Each part is streamed too: multipart must not become another whole-file
|
|
* buffer or a fixed part-count ceiling below the storage provider's limits. */
|
|
export async function putS3Multipart(client: S3Client, bucket: string, key: string, input: PutObjectInput) {
|
|
const source = input.body instanceof Readable ? input.body : Readable.from([input.body]);
|
|
const iterator = source[Symbol.asyncIterator]();
|
|
const controller = new AbortController();
|
|
const active = new Set<Promise<void>>();
|
|
const bodies = new Set<PassThrough>();
|
|
let failed = false;
|
|
let failure: unknown;
|
|
const stop = (error: unknown) => {
|
|
if (failed) return;
|
|
failed = true;
|
|
failure = error;
|
|
controller.abort(error);
|
|
source.destroy();
|
|
for (const body of bodies) body.destroy();
|
|
};
|
|
const check = () => { if (failed) throw failure; };
|
|
const onAbort = () => stop(input.signal!.reason);
|
|
input.signal?.addEventListener("abort", onAbort, { once: true });
|
|
if (input.signal?.aborted) onAbort();
|
|
const untrack = trackUpload(stop);
|
|
const readNext = async () => {
|
|
controller.signal.throwIfAborted();
|
|
let abort!: () => void;
|
|
const interrupted = new Promise<never>((_resolve, reject) => {
|
|
abort = () => reject(controller.signal.reason);
|
|
controller.signal.addEventListener("abort", abort, { once: true });
|
|
});
|
|
try { return await Promise.race([iterator.next(), interrupted]); }
|
|
finally { controller.signal.removeEventListener("abort", abort); }
|
|
};
|
|
let pending = Buffer.alloc(0);
|
|
const readPart = async (length: number, write: (chunk: Buffer) => Promise<void>) => {
|
|
while (length > 0) {
|
|
check();
|
|
if (!pending.length) {
|
|
const next = await readNext();
|
|
check();
|
|
if (next.done) throw new Error("S3 upload source was truncated");
|
|
pending = Buffer.isBuffer(next.value) ? next.value : Buffer.from(next.value);
|
|
if (!pending.length) continue;
|
|
}
|
|
const count = Math.min(length, pending.length, 64 * 1024), chunk = pending.subarray(0, count);
|
|
pending = pending.subarray(count); length -= count;
|
|
await write(chunk);
|
|
}
|
|
};
|
|
let uploadId: string | undefined;
|
|
// Scale part sizes to the S3 part-count limit while retaining stream backpressure.
|
|
const partSize = Math.max(bufferedPartSize, Math.ceil(input.contentLength / 10_000));
|
|
try {
|
|
check();
|
|
uploadId = (await withS3FailureDiagnostics("create_multipart_upload", () => client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }), { abortSignal: controller.signal }), controller.signal)).UploadId;
|
|
if (!uploadId) throw new Error("S3 did not return a multipart upload id");
|
|
const parts: { ETag: string; PartNumber: number }[] = [];
|
|
let partNumber = 0;
|
|
for (let offset = 0; offset < input.contentLength;) {
|
|
while (active.size >= 4) { await Promise.race(active); check(); }
|
|
check();
|
|
const length = Math.min(partSize, input.contentLength - offset);
|
|
const number = ++partNumber;
|
|
let buffered: Buffer | null = null;
|
|
if (partSize <= bufferedPartSize && bufferedPayloadBytes + length <= bufferedPayloadLimit) {
|
|
bufferedPayloadBytes += length;
|
|
try {
|
|
const payload = Buffer.allocUnsafe(length);
|
|
let position = 0;
|
|
await readPart(length, async chunk => { chunk.copy(payload, position); position += chunk.length; });
|
|
buffered = payload;
|
|
} catch (error) { bufferedPayloadBytes -= length; throw error; }
|
|
}
|
|
let remaining = length;
|
|
const body = new PassThrough({ highWaterMark: 64 * 1024 });
|
|
bodies.add(body);
|
|
const write = async (chunk: Buffer) => {
|
|
check(); remaining -= chunk.length;
|
|
if (!body.write(chunk)) await once(body, "drain", { signal: controller.signal });
|
|
};
|
|
const sending = (async () => {
|
|
const result = await withS3FailureDiagnostics("upload_part", () => client.send(new UploadPartCommand({ Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: number, Body: body, ContentLength: length }), { abortSignal: controller.signal }), controller.signal);
|
|
if (remaining || !body.readableEnded || !result.ETag) throw new Error("S3 upload part is incomplete");
|
|
parts[number - 1] = { ETag: result.ETag, PartNumber: number };
|
|
})().catch(stop);
|
|
const producing = buffered ? (async () => {
|
|
for (let position = 0; position < buffered.length; position += 64 * 1024)
|
|
await write(buffered.subarray(position, Math.min(position + 64 * 1024, buffered.length)));
|
|
body.end();
|
|
})().catch(stop) : Promise.resolve();
|
|
const uploading = Promise.all([sending, producing]).then(() => undefined)
|
|
.finally(() => { if (buffered) bufferedPayloadBytes -= length; active.delete(uploading); bodies.delete(body); body.destroy(); });
|
|
active.add(uploading);
|
|
if (!buffered) {
|
|
await readPart(length, write);
|
|
body.end();
|
|
}
|
|
offset += length;
|
|
}
|
|
if (pending.length || !(await readNext()).done) throw new Error("S3 upload source exceeds its declared size");
|
|
await Promise.all(active);
|
|
check();
|
|
await withS3FailureDiagnostics("complete_multipart_upload", () => client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts } }), { abortSignal: controller.signal }), controller.signal);
|
|
} catch (error) {
|
|
stop(error);
|
|
await Promise.all(active);
|
|
if (uploadId) await cleanupStorageFailure(signal => withStorageRequestStats(() => client.send(new AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId }), { abortSignal: signal })));
|
|
throw failure;
|
|
} finally {
|
|
input.signal?.removeEventListener("abort", onAbort);
|
|
source.destroy();
|
|
await cleanupStorageFailure(async () => iterator.return?.());
|
|
untrack();
|
|
}
|
|
}
|
|
|
|
export const FILE_PART_CONCURRENCY = 8;
|
|
const FILE_PART_ATTEMPTS = 3;
|
|
/** A provider that asks for a lower request rate gets more, longer-spaced attempts. */
|
|
export const THROTTLED_PART_ATTEMPTS = 8;
|
|
const THROTTLE_BACKOFF_BASE_MS = 1_000;
|
|
const THROTTLE_BACKOFF_MAX_MS = 20_000;
|
|
const FILE_READ_BYTES = 1024 * 1024;
|
|
|
|
/**
|
|
* S3 answers a request-rate limit with 503 SlowDown; compatible gateways also
|
|
* use 429. The SDK never retries a part because its body is a stream, so the
|
|
* upload must slow down itself.
|
|
*/
|
|
function isRequestRateLimited(error: unknown) {
|
|
if (!error || typeof error !== "object") return false;
|
|
const { name, $metadata } = error as { name?: unknown; $metadata?: { httpStatusCode?: unknown } };
|
|
return name === "SlowDown" || $metadata?.httpStatusCode === 503 || $metadata?.httpStatusCode === 429;
|
|
}
|
|
|
|
const TRANSIENT_FILE_REQUEST_STATUS = new Set([408, 429, 500, 502, 503, 504]);
|
|
const TRANSIENT_FILE_REQUEST_CODES = new Set(["ECONNRESET", "ECONNREFUSED", "ECONNABORTED", "ETIMEDOUT", "EPIPE", "EAI_AGAIN",
|
|
"ENOTFOUND", "ERR_STREAM_PREMATURE_CLOSE", "UND_ERR_SOCKET"]);
|
|
const TRANSIENT_FILE_REQUEST_NAMES = new Set(["RequestTimeout", "TimeoutError", "NetworkingError", "InternalError", "ServiceUnavailable"]);
|
|
|
|
function isTransientFileRequestFailure(error: unknown) {
|
|
if (!error || typeof error !== "object") return false;
|
|
const { name, code, $metadata } = error as { name?: unknown; code?: unknown; $metadata?: { httpStatusCode?: unknown } };
|
|
const status = $metadata?.httpStatusCode;
|
|
if (typeof status === "number") return TRANSIENT_FILE_REQUEST_STATUS.has(status);
|
|
return (typeof code === "string" && TRANSIENT_FILE_REQUEST_CODES.has(code))
|
|
|| (typeof name === "string" && TRANSIENT_FILE_REQUEST_NAMES.has(name));
|
|
}
|
|
|
|
export async function sendFileRequestWithRetry<T>(signal: AbortSignal | undefined, send: () => Promise<T>): Promise<T> {
|
|
for (let failures = 0, throttles = 0; ;) {
|
|
signal?.throwIfAborted();
|
|
try {
|
|
return await send();
|
|
} catch (error) {
|
|
if (signal?.aborted) throw error;
|
|
if (isRequestRateLimited(error)) {
|
|
recordStorageThrottle();
|
|
if (++throttles >= THROTTLED_PART_ATTEMPTS) throw error;
|
|
const ceiling = Math.min(THROTTLE_BACKOFF_MAX_MS, THROTTLE_BACKOFF_BASE_MS * 2 ** (throttles - 1));
|
|
const wait = Math.floor(Math.random() * ceiling);
|
|
recordStorageBackoff(wait);
|
|
await delay(wait, undefined, { signal });
|
|
continue;
|
|
}
|
|
if (!isTransientFileRequestFailure(error) || ++failures >= FILE_PART_ATTEMPTS) throw error;
|
|
recordStorageBackoff(failures * 500);
|
|
await delay(failures * 500, undefined, { signal });
|
|
}
|
|
}
|
|
}
|
|
|
|
/** Base64 MD5 of an inclusive byte range, as the Content-MD5 header expects. */
|
|
export async function fileRangeMd5(file: string, start: number, end: number, signal?: AbortSignal) {
|
|
const digest = createHash("md5");
|
|
for await (const chunk of createReadStream(file, { start, end, signal, highWaterMark: FILE_READ_BYTES })) digest.update(chunk);
|
|
return digest.digest("base64");
|
|
}
|
|
|
|
export async function assertSourceFileSize(file: string, contentLength: number) {
|
|
const stats = await stat(file);
|
|
if (!stats.isFile() || stats.size !== contentLength) throw new Error("S3 upload source file does not match its declared size");
|
|
}
|
|
|
|
/** A file source is read per part instead of through one stream. Parts upload
|
|
* in parallel without the shared memory window, each carries Content-MD5 so
|
|
* the backend rejects bytes it did not receive intact, and a failed part is
|
|
* resent from disk instead of failing the whole upload. When the backend
|
|
* limits the request rate, the part backs off with jitter and the upload
|
|
* halves its parts in flight, once per round of throttled requests. */
|
|
export async function putS3MultipartFile(
|
|
client: S3Client, bucket: string, key: string, input: PutObjectInput & { sourceFile: string },
|
|
): Promise<PutObjectReceipt> {
|
|
const file = input.sourceFile;
|
|
const controller = new AbortController();
|
|
const onAbort = () => controller.abort(input.signal!.reason);
|
|
input.signal?.addEventListener("abort", onAbort, { once: true });
|
|
if (input.signal?.aborted) onAbort();
|
|
const untrack = trackUpload(reason => { if (!controller.signal.aborted) controller.abort(reason); });
|
|
// This path never reads the stream body; release its file descriptor now.
|
|
if (input.body instanceof Readable) input.body.destroy();
|
|
let uploadId: string | undefined;
|
|
try {
|
|
await assertSourceFileSize(file, input.contentLength);
|
|
controller.signal.throwIfAborted();
|
|
const partSize = Math.max(bufferedPartSize, Math.ceil(input.contentLength / 10_000));
|
|
const count = Math.ceil(input.contentLength / partSize);
|
|
uploadId = (await sendThroughStorageOutage(() => withS3FailureDiagnostics("create_multipart_upload", () => client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }), { abortSignal: controller.signal }), controller.signal), controller.signal)).UploadId;
|
|
if (!uploadId) throw new Error("S3 did not return a multipart upload id");
|
|
const id = uploadId;
|
|
const parts: { ETag: string; PartNumber: number }[] = new Array(count);
|
|
// Workers at or above this slot stop taking new parts. A reduction counts
|
|
// only for requests sent after the previous one, so a burst of parallel
|
|
// throttled parts halves the rate once rather than once per part.
|
|
let concurrency = FILE_PART_CONCURRENCY, reductions = 0;
|
|
const sendPart = async (index: number) => {
|
|
const start = index * partSize, length = Math.min(partSize, input.contentLength - start), end = start + length - 1;
|
|
const contentMd5 = await fileRangeMd5(file, start, end, controller.signal);
|
|
for (let failures = 0, throttles = 0; ;) {
|
|
controller.signal.throwIfAborted();
|
|
const body = createReadStream(file, { start, end, highWaterMark: 64 * 1024 });
|
|
const sentAfter = reductions;
|
|
try {
|
|
const result = await withS3FailureDiagnostics("upload_part", () => client.send(new UploadPartCommand({
|
|
Bucket: bucket, Key: key, UploadId: id, PartNumber: index + 1, Body: body, ContentLength: length, ContentMD5: contentMd5,
|
|
}), { abortSignal: controller.signal }), controller.signal);
|
|
if (!result.ETag) throw new Error("S3 upload part is incomplete");
|
|
parts[index] = { ETag: result.ETag, PartNumber: index + 1 };
|
|
recordStoragePart();
|
|
return;
|
|
} catch (error) {
|
|
// A part read from disk can be resent; the stream path cannot.
|
|
if (controller.signal.aborted) throw error;
|
|
if (isRequestRateLimited(error)) {
|
|
recordStorageThrottle();
|
|
if (++throttles >= THROTTLED_PART_ATTEMPTS) throw error;
|
|
if (sentAfter === reductions) { concurrency = Math.max(1, Math.floor(concurrency / 2)); reductions++; }
|
|
const ceiling = Math.min(THROTTLE_BACKOFF_MAX_MS, THROTTLE_BACKOFF_BASE_MS * 2 ** (throttles - 1));
|
|
const wait = Math.floor(Math.random() * ceiling);
|
|
recordStorageBackoff(wait);
|
|
await delay(wait, undefined, { signal: controller.signal });
|
|
continue;
|
|
}
|
|
if (++failures >= FILE_PART_ATTEMPTS) throw error;
|
|
recordStorageBackoff(failures * 500);
|
|
await delay(failures * 500, undefined, { signal: controller.signal });
|
|
} finally { body.destroy(); }
|
|
}
|
|
};
|
|
let next = 0;
|
|
const worker = async (slot: number) => {
|
|
while (next < count && slot < concurrency) {
|
|
const index = next++;
|
|
try { await sendThroughStorageOutage(() => sendPart(index), controller.signal); }
|
|
catch (error) { if (!controller.signal.aborted) controller.abort(error); throw error; }
|
|
}
|
|
};
|
|
// Wait for every sibling to settle before the multipart upload is aborted.
|
|
const settled = await Promise.allSettled(Array.from({ length: Math.min(FILE_PART_CONCURRENCY, count) }, (_, slot) => worker(slot)));
|
|
const rejected = settled.find((entry): entry is PromiseRejectedResult => entry.status === "rejected");
|
|
if (rejected) throw controller.signal.aborted && controller.signal.reason !== undefined ? controller.signal.reason : rejected.reason;
|
|
controller.signal.throwIfAborted();
|
|
await sendThroughStorageOutage(() => withS3FailureDiagnostics("complete_multipart_upload", () => client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: id, MultipartUpload: { Parts: parts } }), { abortSignal: controller.signal }), controller.signal), controller.signal);
|
|
uploadId = undefined;
|
|
return { integrity: "server_verified_md5" };
|
|
} catch (error) {
|
|
const failedUpload = uploadId;
|
|
if (failedUpload) await cleanupStorageFailure(signal => withStorageRequestStats(() => client.send(new AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: failedUpload }), { abortSignal: signal })));
|
|
throw error;
|
|
} finally {
|
|
input.signal?.removeEventListener("abort", onAbort);
|
|
untrack();
|
|
}
|
|
}
|