Files
PaperClipAI/server/src/__tests__/storage-s3-multipart.test.ts
T
DottaandClaude Fable 5.2 1af987690d perf(storage): hash and encrypt workspace snapshots off the server thread (#21138)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process serves every request and agent turn on one
thread. The same process saves each remote workspace as one encrypted
archive in object storage, and restores it before a turn
> - `server/src/services/remote-workspace-snapshot-storage.ts` hashed,
encrypted and decrypted every byte of those archives on that thread.
`server/src/storage/s3-multipart.ts` computed the Content-MD5 of each
uploaded part on that thread too
> - Hashes and ciphers are synchronous in Node. A CPU profile of a busy
hosted server showed this work on the main thread, which was at 93% of
one core
> - Each megabyte of a snapshot delays every request and turn that waits
behind it
> - This pull request moves those passes to a small, bounded set of
worker threads. The checks, their order, their failures and the stored
format do not change
> - The benefit is that a 1 GiB save uses about 0.7 s of the server's
thread instead of about 2.8 s, and an unrelated request during the save
is answered in 1.3 ms (p99) instead of 16.7 ms

## Linked Issues or Issue Description

No public issue exists. I searched open and closed pull requests for
`workspace snapshot hash encrypt`, `event loop` and `archive io`.
Related merged work: #19900 gave these stream passes 1 MiB buffers,
#20554 removed a read-back from migration, #20722 took runner artifact
hashes off the event loop, #20709 records long event-loop blocks. No
open pull request covers this change.

**What happened?**

A 30 second CPU profile of the main thread of a hosted server (commit
1d153f62b6, Node 24, main thread at 93% of one core) showed:

- the two stream transforms in the snapshot save at 3.9% and 1.6%
inclusive. The first hashes the archive (SHA-256). The second hashes the
ciphertext (SHA-256);
- native hash `update` at 5.4% self time for the whole process. About
1.0 s of it was under those two transforms and 0.25 s was under
`fileRangeMd5`;
- the AES-256-GCM cipher of the same pipeline, and Buffer allocation
under the file read streams. Garbage collection was 25% of the thread.

What the code did with each byte, on the server's thread:

| Path | Pass | Digest or cipher | Purpose |
| --- | --- | --- | --- |
| Save | read archive | SHA-256 of plaintext | bound into the wrapped
key, checked on each restore |
| Save | same read | AES-256-GCM | confidentiality and authenticity |
| Save | same read | SHA-256 of stored bytes | recorded on the snapshot
row, checked on each restore and read-back |
| Save | read each part (S3) | MD5 | `Content-MD5`: the object store
rejects a part it did not receive intact |
| Save, read-back callers | read the kept file | SHA-256 of plaintext |
proof that the published file is the saved archive |
| Restore | download | SHA-256 of stored bytes | integrity of the object
against the snapshot row |
| Restore | read ciphertext | AES-256-GCM | authenticity |
| Restore | read plaintext | SHA-256 of plaintext | integrity against
the digest in the wrapped key |
| Restore | read plaintext again | none (copy) | publication beside the
destination |

**Expected behavior**

Digest and cipher passes over a snapshot do not run on the thread that
serves requests. Every check stays, with the same failure.

**Steps to reproduce**

1. On the parent commit, run `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (copy the
script from this branch).
2. Read `upload.mainThreadCpuMs`, `upload.mainThreadBusyMs` and
`upload.unrelatedRequest`. The benchmark uses the real save and restore
functions, the real S3 client, and a local object store that checks each
part's `Content-MD5`. A separate process requests a trivial route from
the measured server and times each reply with its own clock.

**Paperclip version or commit**

`master` at 9e910ff9ae. The profile is from 1d153f62b6.

**Deployment mode**

All modes that save remote workspaces. The profile is from a hosted
server with S3 storage.

**Node.js version**

Profile: Node 24. Local benchmark and tests: Node 26.4.0 on darwin
arm64.

## What Changed

- `server/src/storage/file-crypto-pool.ts` (new): runs file digests and
AES-256-GCM passes on worker threads.
- Bounds: at most 2 jobs run at once. Each worker has two reused 1 MiB
read buffers and one write in flight. A waiting job holds file names and
byte ranges, not file contents. Waiting jobs are served in order.
- Cancellation: a waiting job is cancelled at once. A running job stops
at its next chunk, closes its files, and then the caller sees the same
`AbortError` as before.
- Shutdown: an idle worker does not keep the process alive. A busy
worker does, until its job ends or its caller's signal aborts it.
- Key handling: the pool copies the data key for the worker, moves that
copy (it is not cloned), and the worker erases it. A copy that never
reaches a worker is erased on this thread. The caller keeps and erases
its own key as before.
- The worker program is sent as source text, as
`remote-workspace-snapshot-archive.ts` already does. It needs no extra
file in the build or the runtime bundle.
- Save (`uploadWorkspaceSnapshot`): the one read that digests the
archive, encrypts it and digests the stored bytes runs in a worker. The
identity check around that read (`archive_changed`) stays on this thread
and is unchanged.
- Part upload (`fileRangeMd5`): the `Content-MD5` of each part is
computed in a worker. The first round of parts is digested before any
part is sent, so the parts still leave together and a rate limit still
answers one burst (the existing throttle test covers this). Every digest
of that round has ended, also after a failure or a cancellation, before
the upload settles.
- Restore (`loadWorkspaceSnapshotArchive`): the download no longer
hashes on this thread. One worker digests the downloaded object while
another decrypts it into private scratch. The results are then checked
in the previous order: object size and digest, header, key, GCM tag.
- Restore, one pass less: the decrypted archive was read twice (digest,
then copy). It is now digested while it is copied beside the
destination, and linked to the destination only when the digest matches.
The digest covers exactly the published bytes.
- Read-back callers (`verifiedArchivePath`): the digest of the kept file
runs in a worker, through the same open descriptor. The file identity
checks are unchanged.
- `server/scripts/benchmark-workspace-snapshot-crypto.ts` (new): the
reproduction and benchmark.
- Docs: `doc/run-log-events.md` (what each `storage.snapshot.*` phase
now does) and `doc/DEVELOPING.md` (the worker threads and the
benchmark).

Module boundaries (`rules/module-boundaries.mdc`): the pool is in
`server/src/storage/` because both consumers are parts of the same
upload, and storage must not import from services. Its contract is file
paths, byte ranges, an algorithm, and AES-GCM parameters. The snapshot
format (magic bytes, additional data, key wrapping, the order of checks)
stays in the snapshot service. The part layout and `Content-MD5` stay in
the S3 module.

Not changed, and why:

- The stored format, the ledger, attempt journaling, atomic publication,
and `recoverWorkspaceSnapshotUpload` /
`recoverWorkspaceSnapshotArchive`.
- `verifyObject` (full read-back for a store that did not verify the
upload, and for a retried save) still hashes on this thread. Its bytes
arrive from the network on this thread. To move it, the object must
first be written to disk.
- Socket writes of part bodies stay on this thread. I measured 1 MiB
part-body read chunks in place of 64 KiB: main-thread time did not
change and the longest event-loop delay increased, so I did not keep it.
- `remote-workspace-transport.ts` also hashes an archive on this thread.
It is outside the snapshot storage path.

## Verification

Regression tests, red then green (`cd server && pnpm exec vitest run
src/services/remote-workspace-snapshot-storage.test.ts
src/storage/file-crypto-pool.test.ts`):

- Parent implementation with the tests of this branch: 3 tests fail.
- `keeps encrypting a save while the server thread is occupied by other
work`: `expected 4194323 to be 25166144`. The test blocks the thread
until the encrypted file is complete. On the parent the file stops at 4
MiB.
- `answers unrelated requests while a save is being encrypted, and
digests or encrypts none of it on the server thread`: `expected 8388627
to be less than 65536` (bytes hashed on the thread).
- `digests and decrypts none of a save or a restore on the server thread
when the store verified the upload`: `expected 33556772 to be less than
65536`.
- This branch: 58 of 58 and 6 of 6 pass.
- The tests use synchronization, not time. A named pipe keeps a save in
the encrypt step for as long as the test feeds it. Deadlines are 30 s
and matter only for a failure.
- Integrity tests that pass on both the parent implementation and this
branch, which shows that the failures are the same: a truncated object
(`object_hash_mismatch`), an oversized object (`object_size_mismatch`),
an archive that grows, shrinks or is rewritten in place after its
identity was read (`archive_changed`, before any upload), a ciphertext
that was changed with the recorded digest changed to match (GCM
failure), a plaintext that differs from the digest in the key
(`plaintext_hash_mismatch`, staged copy removed), and the order object,
header, key.
- Cancellation: during encryption the save rejects with `AbortError`,
and leaves no scratch file, no object and no recorded attempt. During
the store write it leaves no object.
- Part digests (`src/__tests__/storage-s3-multipart.test.ts`): `leaves
no part digest running or waiting when the source is cancelled|removed
during the first round`. Before the review fix in a354242865 it failed
with `{ running: 2, waiting: 0 }` at multipart cleanup.
- Pool tests: digests equal single-threaded digests for a file, a range
and a descriptor; encrypt and decrypt round trip; each damaged
ciphertext is reported; at most 2 jobs run and the others wait in order;
a waiting job and a running job are both cancelled.

Benchmark, same machine, 5 alternating pairs (parent implementation,
then this branch) for each size. Values are medians, with the range in
parentheses.

Command: `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-workspace-snapshot-crypto.ts --mib=1024` (and
`--mib=200`).

1 GiB archive:

| Measure | Save, before | Save, after | Restore, before | Restore,
after |
| --- | --- | --- | --- | --- |
| Main-thread CPU time | 2,834 ms (2,717 to 3,007) | 742 ms (658 to 881)
| 1,602 ms (1,378 to 1,837) | 510 ms (470 to 589) |
| Main-thread busy time (event-loop utilization) | 2,938 ms | 858 ms |
1,673 ms | 597 ms |
| Total process CPU time | 6,735 ms | 4,762 ms | 5,564 ms | 3,306 ms |
| Wall time | 3,914 ms (3,656 to 4,253) | 2,580 ms (2,507 to 3,200) |
2,502 ms (1,756 to 2,922) | 1,799 ms (1,508 to 2,492) |
| Longest event-loop delay | 12.6 ms | 6.0 ms | 10.4 ms | 5.4 ms |
| Event-loop delay p99 | 9.2 ms | 2.0 ms | 2.1 ms | 1.8 ms |
| Unrelated request p99 (p50 is 0.2 to 0.3 ms in all runs) | 16.7 ms |
1.3 ms | 4.7 ms | 1.3 ms |
| Unrelated request, slowest | 18.5 ms | 4.5 ms | 10.0 ms | 3.5 ms |
| Peak resident memory | 495 MiB | 408 MiB | 524 MiB | 476 MiB |

200 MiB archive:

| Measure | Save, before | Save, after | Restore, before | Restore,
after |
| --- | --- | --- | --- | --- |
| Main-thread CPU time | 571 ms | 181 ms | 378 ms | 141 ms |
| Main-thread busy time | 602 ms | 236 ms | 395 ms | 167 ms |
| Total process CPU time | 1,235 ms | 935 ms | 1,385 ms | 840 ms |
| Wall time | 842 ms | 679 ms | 538 ms | 382 ms |
| Longest event-loop delay | 12.4 ms | 4.0 ms | 7.3 ms | 2.9 ms |
| Unrelated request p99 | 16.5 ms | 2.0 ms | 1.9 ms | 1.9 ms |
| Peak resident memory | 437 MiB | 398 MiB | 510 MiB | 424 MiB |

How to read these numbers:

- The parent did not block the thread for seconds at one time. It worked
in 1 MiB slices, so the longest single delay was about 12 ms. The cost
was the sum: the thread was busy for about 75% of a 1 GiB save. That sum
is what the profile showed, and it is what decreases.
- A worker thread moves CPU time; it does not remove it. Total CPU time
also decreased here (about 29% for a save, 41% for a restore) because
the workers reuse two buffers in place of one new 1 MiB Buffer for each
chunk, and a restore reads the decrypted archive once in place of twice.
- Wall time decreased because the part digests no longer wait for the
thread that sends the parts, and a restore digests and decrypts at the
same time.
- What remains on the server's thread is network and file streaming:
socket writes of the parts, the download, and the database calls.
- Limits: the machine was shared with other work (1-minute load average
8 to 14 on 18 cores) and timings varied between runs, which is why the
pairs alternate. Earlier runs on a busier machine (load 15 to 28) showed
the same CPU-time ratios with wall times up to 4 times longer for both
implementations. The object store was on loopback without TLS.

Other checks:

- `cd server && pnpm exec vitest run src/storage
src/services/remote-workspace-snapshot-storage.test.ts
src/services/remote-workspace-snapshot-reuse.test.ts
src/__tests__/remote-workspace-file-resources.test.ts` and the 8
`src/__tests__/storage-s3-*.test.ts` files: 249 tests in 15 files pass.
- `tsc --noEmit` over the changed TypeScript files: clean. `pnpm
check:module-boundaries`: pass.
- The pool, bundled with the options of
`server/scripts/bundle-runtime.mjs` and run with plain `node`: encrypt,
decrypt and digest round trip, and the process exits.
- `pnpm verify` exits 2 (broader verification required, because of the
new file in `server/scripts`). Required CI supplies that coverage.
- Not verified: the hosted server after deploy, Linux x86 with Node 24
outside CI, a real object store with TLS, and archives larger than 1
GiB.

## Risks

- This is integrity and confidentiality code. The mitigation is that
each check and its failure is pinned by a test that passes on the parent
implementation and on this branch, and that the format is unchanged, so
objects saved before and after this change restore with either version.
- Order of work in a restore: the ciphertext is now decrypted into
private scratch while the object digest is computed, not after it. No
output is used before the digest, header, key and tag checks pass, and
the scratch directory is removed on every failure. A damaged object is
reported with the same error, but after the decrypt work.
- The staged copy beside the destination (`.workspace-snapshot-*.tmp`)
is now written before the plaintext digest is known, and after the GCM
tag is verified. It is removed on a mismatch. The destination itself is
linked only after the digest matches, as before.
- Resource trade-off: up to 2 more threads, each with about 2 MiB of
buffers and a small heap (64 MB limit), that stay for the life of the
process once used. In exchange the server's thread does about 70% less
work for each snapshot. With more than 2 snapshot passes at one time,
the others wait. A part digest can wait behind a long encryption.
- Run-log timing shifts: `storage.snapshot.digest` is now about 0 ms,
`storage.snapshot.verify_plaintext` includes the copy, and
`storage.snapshot.publish` is only the link. `doc/run-log-events.md`
describes this.
- A worker that stops unexpectedly fails its job with
`file_crypto_worker_exited`. The save or restore then fails as for any
other I/O error, and the next job starts a new worker.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code with
tool use (shell, file edits, local test and benchmark runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:11:37 -06:00

602 lines
34 KiB
TypeScript

import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, DeleteObjectCommand, PutObjectCommand, S3Client } from "@aws-sdk/client-s3";
import { Readable } from "node:stream";
import { createHash } from "node:crypto";
import { createServer, type ServerResponse } from "node:http";
import type { AddressInfo } from "node:net";
import { describe, expect, it, vi } from "vitest";
import { FILE_PART_CONCURRENCY, putS3Multipart, putS3MultipartFile } from "../storage/s3-multipart.js";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { createS3StorageProvider } from "../storage/s3-provider.js";
import { createStorageService } from "../storage/service.js";
import { STORAGE_FAILURE_CLEANUP_MS } from "../storage/failure-cleanup.js";
import { fileCryptoPoolLoad } from "../storage/file-crypto-pool.js";
describe("streamed S3 multipart uploads", () => {
it("delivers four slow part payloads concurrently before any part is drained", async () => {
const client = new S3Client({ region: "us-east-1" }), started = new Set<number>(), received = new Map<number, number>();
let release!: () => void, active = 0, peak = 0, completed = false;
const reading = new Promise<void>(resolve => { release = resolve; });
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
const number = command.input.PartNumber!; started.add(number);
await reading; active++; peak = Math.max(peak, active);
let bytes = 0; const digest = createHash("sha256");
for await (const chunk of command.input.Body as Readable) {
digest.update(chunk); bytes += chunk.length;
await new Promise(resolve => setTimeout(resolve, 1));
}
expect(bytes).toBe(8 * 1024 * 1024);
expect(digest.digest("hex")).toBe(createHash("sha256").update(Buffer.alloc(bytes, number)).digest("hex"));
received.set(number, bytes); active--;
return { ETag: `part-${number}` };
}
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
expect(command.input.MultipartUpload.Parts).toEqual([1, 2, 3, 4].map(PartNumber => ({ PartNumber, ETag: `part-${PartNumber}` })));
completed = true; return {};
});
const body = Readable.from((async function* () {
for (let index = 0; index < 512; index++) yield Buffer.alloc(64 * 1024, Math.floor(index / 128) + 1);
})());
const saving = putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream", contentLength: 32 * 1024 * 1024, body }).catch(error => error);
try {
await expect.poll(() => started.size, { timeout: 1500 }).toBe(4);
expect(received.size).toBe(0); release();
expect(await saving).toBeUndefined();
expect(peak).toBe(4); expect(received.size).toBe(4); expect(completed).toBe(true);
} finally { release(); await saving; send.mockRestore(); client.destroy(); }
});
it("sends concurrent exact part bodies through the real S3 HTTP client", async () => {
const received = new Map<number, string>();
const responses = new Map<number, ServerResponse>();
let completedParts = "";
const server = createServer((request, response) => {
const url = new URL(request.url!, "http://localhost");
const digest = createHash("sha256");
let text = "";
request.on("data", chunk => { digest.update(chunk); if (request.method === "POST") text += chunk; });
request.on("end", () => {
if (request.method === "PUT") {
const number = Number(url.searchParams.get("partNumber"));
received.set(number, digest.digest("hex"));
responses.set(number, response);
} else if (url.searchParams.has("uploads")) {
response.end('<InitiateMultipartUploadResult><UploadId>upload</UploadId></InitiateMultipartUploadResult>');
} else {
completedParts = text;
response.end('<CompleteMultipartUploadResult><ETag>complete</ETag></CompleteMultipartUploadResult>');
}
});
});
await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); });
const client = new S3Client({ region: "us-east-1", endpoint: `http://127.0.0.1:${(server.address() as AddressInfo).port}`,
forcePathStyle: true, requestChecksumCalculation: "WHEN_REQUIRED", credentials: { accessKeyId: "fixture", secretAccessKey: "fixture" } });
const cancellation = new AbortController();
const body = Readable.from((async function* () {
for (let index = 0; index < 512; index++) yield Buffer.alloc(64 * 1024, Math.floor(index / 128) + 1);
})());
const saving = putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream", contentLength: 32 * 1024 * 1024, body, signal: cancellation.signal }).catch(error => error);
try {
await expect.poll(() => received.size, { timeout: 3000 }).toBe(4);
for (const number of [4, 2, 3, 1]) {
expect(received.get(number)).toBe(createHash("sha256").update(Buffer.alloc(8 * 1024 * 1024, number)).digest("hex"));
responses.get(number)!.writeHead(200, { ETag: `part-${number}` }).end();
}
expect(await saving).toBeUndefined();
expect([...completedParts.matchAll(/<PartNumber>(\d+)<\/PartNumber>/g)].map(match => Number(match[1]))).toEqual([1, 2, 3, 4]);
} finally {
cancellation.abort();
await saving;
client.destroy();
server.closeAllConnections();
await new Promise<void>(resolve => server.close(() => resolve()));
}
});
it("overlaps slow part acknowledgements and completes exact bytes in source order", async () => {
const client = new S3Client({ region: "us-east-1" });
const received = new Map<number, Buffer>();
const acknowledgements = new Map<number, () => void>();
let completed = false;
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
const chunks: Buffer[] = [];
for await (const chunk of command.input.Body as Readable) chunks.push(chunk);
received.set(command.input.PartNumber!, Buffer.concat(chunks));
await new Promise<void>(resolve => acknowledgements.set(command.input.PartNumber!, resolve));
return { ETag: `part-${command.input.PartNumber}` };
}
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
expect(command.input.MultipartUpload.Parts).toEqual([1, 2, 3, 4].map(PartNumber => ({ PartNumber, ETag: `part-${PartNumber}` })));
completed = true;
return {};
});
const length = 25 * 1024 * 1024;
const body = Readable.from((async function* () {
for (let offset = 0; offset < length; offset += 64 * 1024) yield Buffer.alloc(Math.min(64 * 1024, length - offset), offset / (64 * 1024) % 251);
})());
const saving = putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream", contentLength: length, body });
try {
await expect.poll(() => received.size, { timeout: 1000 }).toBe(4);
expect(completed).toBe(false);
for (const number of [4, 2, 3, 1]) acknowledgements.get(number)!();
await saving;
const actual = Buffer.concat([1, 2, 3, 4].map(number => received.get(number)!));
expect(actual.length).toBe(length);
for (let offset = 0; offset < length; offset += 64 * 1024) {
expect(actual.subarray(offset, Math.min(length, offset + 64 * 1024)).every(byte => byte === offset / (64 * 1024) % 251)).toBe(true);
}
expect(completed).toBe(true);
} finally {
const release = setInterval(() => { for (const acknowledge of acknowledgements.values()) acknowledge(); }, 1);
try { await saving; } finally { clearInterval(release); send.mockRestore(); client.destroy(); }
}
});
it("streams exact ordered parts without buffering the full response", async () => {
const client = new S3Client({ region: "us-east-1" });
const lengths = new Map<number, number>();
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
expect(command.input.Body).toBeInstanceOf(Readable);
let length = 0;
for await (const chunk of command.input.Body as Readable) {
expect(chunk.length).toBeLessThanOrEqual(64 * 1024);
length += chunk.length;
}
expect(length).toBe(command.input.ContentLength);
lengths.set(command.input.PartNumber!, length);
return { ETag: `part-${command.input.PartNumber}` };
}
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
expect(command.input.MultipartUpload.Parts).toEqual([1, 2, 3].map(PartNumber => ({ PartNumber, ETag: `part-${PartNumber}` })));
return {};
});
const body = Readable.from((async function* () { for (let i = 0; i < 272; i++) yield Buffer.alloc(64 * 1024); })());
try {
await putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "text/plain", contentLength: 17 * 1024 * 1024, body });
expect([...lengths].sort(([a], [b]) => a - b)).toEqual([[1, 8], [2, 8], [3, 1]].map(([part, mb]) => [part, mb * 1024 * 1024]));
} finally { send.mockRestore(); client.destroy(); }
});
it("keeps source backpressure and at most four part requests while acknowledgements stall", async () => {
const client = new S3Client({ region: "us-east-1" });
let releaseReading!: () => void, releaseAcknowledgements!: () => void;
const reading = new Promise<void>(resolve => { releaseReading = resolve; });
const acknowledgements = new Promise<void>(resolve => { releaseAcknowledgements = resolve; });
let yielded = 0, active = 0, peak = 0, received = 0;
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
active++; peak = Math.max(peak, active);
await reading;
for await (const chunk of command.input.Body as Readable) expect(chunk.length).toBeLessThanOrEqual(64 * 1024);
received++;
await acknowledgements;
active--;
return { ETag: `part-${command.input.PartNumber}` };
}
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
return {};
});
const body = Readable.from((async function* () {
for (let index = 0; index < 1600; index++) { yielded++; yield Buffer.alloc(64 * 1024); }
})());
const saving = putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream", contentLength: 100 * 1024 * 1024, body });
try {
await expect.poll(() => active).toBe(4);
expect(yielded).toBeLessThanOrEqual(4 * 128 + 3);
releaseReading();
await expect.poll(() => received).toBe(4);
expect(yielded).toBeLessThanOrEqual(4 * 128 + 3);
expect(peak).toBe(4);
releaseAcknowledgements();
await saving;
expect(received).toBe(13);
expect(yielded).toBe(1600);
expect(peak).toBe(4);
} finally { releaseReading(); releaseAcknowledgements(); await saving; send.mockRestore(); client.destroy(); }
});
it("shares the buffered payload window across simultaneous uploads", async () => {
const client = new S3Client({ region: "us-east-1" });
let release!: () => void, yielded = 0;
const reading = new Promise<void>(resolve => { release = resolve; });
const started = new Map<string, number>();
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: command.input.Key };
if (command instanceof UploadPartCommand) {
const key = command.input.Key!;
started.set(key, (started.get(key) ?? 0) + 1);
await reading;
let bytes = 0;
for await (const chunk of command.input.Body as Readable) bytes += chunk.length;
expect(bytes).toBe(command.input.ContentLength);
return { ETag: `part-${command.input.PartNumber}` };
}
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
return {};
});
const saving = Promise.all(["a", "b"].map(key => putS3Multipart(client, "bucket", key, {
objectKey: key, contentType: "application/octet-stream", contentLength: 32 * 1024 * 1024,
body: Readable.from((async function* () {
for (let index = 0; index < 512; index++) { yielded++; yield Buffer.alloc(64 * 1024); }
})()),
})));
try {
await expect.poll(() => ["a", "b"].every(key => (started.get(key) ?? 0) >= 3)).toBe(true);
expect(yielded).toBeLessThanOrEqual(4 * 128 + 8);
release(); await saving;
expect(yielded).toBe(1024);
expect([...started.values()]).toEqual([4, 4]);
} finally { release(); await saving; send.mockRestore(); client.destroy(); }
});
it.each(["source", "request"])("releases the shared payload allowance after a %s failure", async failure => {
const client = new S3Client({ region: "us-east-1" });
let release!: () => void, yielded = 0, started = 0;
const reading = new Promise<void>(resolve => { release = resolve; });
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: command.input.Key };
if (command instanceof UploadPartCommand) {
if (command.input.Key === "failed") throw new Error("request failed");
started++; await reading;
for await (const _chunk of command.input.Body as Readable) {}
return { ETag: `part-${command.input.PartNumber}` };
}
expect(command instanceof CompleteMultipartUploadCommand || command instanceof AbortMultipartUploadCommand).toBe(true);
return {};
});
const first = Readable.from((async function* () {
if (failure === "source") throw new Error("source failed");
yield Buffer.alloc(8 * 1024 * 1024);
})());
let saving: Promise<void> | undefined;
try {
await expect(putS3Multipart(client, "bucket", "failed", {
objectKey: "failed", contentType: "application/octet-stream", contentLength: 8 * 1024 * 1024, body: first,
})).rejects.toThrow(`${failure} failed`);
saving = putS3Multipart(client, "bucket", "next", {
objectKey: "next", contentType: "application/octet-stream", contentLength: 32 * 1024 * 1024,
body: Readable.from((async function* () {
for (let index = 0; index < 512; index++) { yielded++; yield Buffer.alloc(64 * 1024); }
})()),
});
await expect.poll(() => started).toBe(4);
expect(yielded).toBe(512);
release(); await saving;
} finally { release(); await saving; send.mockRestore(); client.destroy(); }
});
it("settles cancelled siblings before aborting a failed multipart upload", async () => {
const client = new S3Client({ region: "us-east-1" });
const cancellations: AbortSignal[] = [];
let failPart!: () => void, releaseSiblings!: () => void;
const settlement = new Promise<void>(resolve => { releaseSiblings = resolve; });
let received = 0, settled = 0, aborted = false, completed = false;
const send = vi.spyOn(client, "send").mockImplementation(async (command: any, options: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
for await (const _chunk of command.input.Body as Readable) {}
received++;
if (command.input.PartNumber === 2) return new Promise((_resolve, reject) => { failPart = () => reject(new Error("second part failed")); });
cancellations.push(options.abortSignal);
await new Promise<void>(resolve => options.abortSignal.addEventListener("abort", () => resolve(), { once: true }));
await settlement;
settled++;
throw options.abortSignal.reason;
}
if (command instanceof CompleteMultipartUploadCommand) { completed = true; return {}; }
expect(command).toBeInstanceOf(AbortMultipartUploadCommand);
expect(settled).toBe(3);
aborted = true;
return {};
});
const body = Readable.from((async function* () { for (let index = 0; index < 512; index++) yield Buffer.alloc(64 * 1024); })());
const saving = putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream", contentLength: 32 * 1024 * 1024, body }).catch(error => error);
try {
await expect.poll(() => received).toBe(4);
failPart();
await expect.poll(() => cancellations.every(signal => signal.aborted)).toBe(true);
expect(aborted).toBe(false);
releaseSiblings();
expect((await saving).message).toBe("second part failed");
expect(aborted).toBe(true);
expect(completed).toBe(false);
expect(body.destroyed).toBe(true);
} finally { failPart?.(); releaseSiblings(); await saving; send.mockRestore(); client.destroy(); }
});
it.each(["truncated", "excess", "source error"])("rejects %s input without publishing an object", async kind => {
const client = new S3Client({ region: "us-east-1" });
let aborted = false;
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
for await (const _chunk of command.input.Body as Readable) {}
return { ETag: `part-${command.input.PartNumber}` };
}
expect(command).toBeInstanceOf(AbortMultipartUploadCommand);
aborted = true;
return {};
});
const expected = { truncated: "S3 upload source was truncated", excess: "S3 upload source exceeds its declared size", "source error": "source unavailable" };
const body = Readable.from((async function* () {
for (let index = 0; index < (kind === "excess" ? 274 : 270); index++) yield Buffer.alloc(64 * 1024);
if (kind === "source error") throw new Error("source unavailable");
})());
try {
await expect(putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream", contentLength: 17 * 1024 * 1024, body })).rejects.toThrow(expected[kind as keyof typeof expected]);
expect(aborted).toBe(true);
expect(body.destroyed).toBe(true);
} finally { send.mockRestore(); client.destroy(); }
});
it("aborts an incomplete upload and closes its source on failure", async () => {
const client = new S3Client({ region: "us-east-1" });
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) throw new Error("storage unavailable");
expect(command).toBeInstanceOf(AbortMultipartUploadCommand);
return {};
});
const body = Readable.from([Buffer.from("abc")]);
try {
await expect(putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "text/plain", contentLength: 3, body })).rejects.toThrow("storage unavailable");
expect(send).toHaveBeenCalledTimes(3);
expect(body.destroyed).toBe(true);
} finally { send.mockRestore(); client.destroy(); }
});
it("bounds stalled multipart abort and object cleanup after cancellation", async () => {
vi.useFakeTimers();
const cancellation = new AbortController();
const cleanupSignals: AbortSignal[] = [];
const send = vi.spyOn(S3Client.prototype, "send").mockImplementation(async (command: any, options: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) return new Promise((_resolve, reject) => {
options.abortSignal.addEventListener("abort", () => reject(cancellation.signal.reason), { once: true });
});
if (command instanceof AbortMultipartUploadCommand || command instanceof DeleteObjectCommand) {
cleanupSignals.push(options.abortSignal);
return new Promise(() => {});
}
expect(command).toBeInstanceOf(PutObjectCommand);
return {};
});
const storage = createStorageService(createS3StorageProvider({ bucket: "test", region: "us-east-1" }));
const body = Readable.from([Buffer.alloc(17 * 1024 * 1024)]);
try {
const saving = storage.putFile({ companyId: "company", namespace: "skill-packages", objectKey: "company/package/archive",
originalFilename: "archive", contentType: "application/gzip", body, byteSize: 17 * 1024 * 1024, sha256: "a".repeat(64), signal: cancellation.signal }).catch(error => error);
await vi.advanceTimersByTimeAsync(0);
cancellation.abort(new Error("cancelled upload"));
await vi.advanceTimersByTimeAsync(STORAGE_FAILURE_CLEANUP_MS * 2 + 1);
expect((await saving).message).toBe("cancelled upload");
expect(cleanupSignals).toHaveLength(2);
expect(cleanupSignals.every(signal => signal.aborted)).toBe(true);
expect(body.destroyed).toBe(true);
await expect(storage.putFile({ companyId: "company", namespace: "skill-packages", originalFilename: "next", contentType: "text/plain", body: Buffer.from("next") })).resolves.toMatchObject({ byteSize: 4 });
} finally { send.mockRestore(); vi.useRealTimers(); }
});
it.each([128 * 1024, 8 * 1024 * 1024 * 10_000 + 1])("bounds cancellation while an asynchronous source read is stalled (declared bytes: %s)", async contentLength => {
vi.useFakeTimers();
const client = new S3Client({ region: "us-east-1" });
const cancellation = new AbortController();
let release!: () => void, waiting = false, settled = false, partLength: number | undefined;
const sourceGate = new Promise<void>(resolve => { release = resolve; });
const body = Readable.from((async function* () {
yield Buffer.alloc(64 * 1024);
waiting = true;
await sourceGate;
yield Buffer.alloc(64 * 1024);
})());
const send = vi.spyOn(client, "send").mockImplementation(async (command: any, options: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
partLength = command.input.ContentLength;
(command.input.Body as Readable).resume();
return new Promise((_resolve, reject) => options.abortSignal.addEventListener("abort", () => reject(options.abortSignal.reason), { once: true }));
}
expect(command).toBeInstanceOf(AbortMultipartUploadCommand);
return {};
});
const saving = putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "text/plain", contentLength, body, signal: cancellation.signal })
.catch(error => error).finally(() => { settled = true; });
try {
await vi.advanceTimersByTimeAsync(0);
expect(waiting).toBe(true);
if (contentLength > 8 * 1024 * 1024 * 10_000) expect(partLength).toBe(8 * 1024 * 1024 + 1);
cancellation.abort(new Error("cancelled during source read"));
await vi.advanceTimersByTimeAsync(STORAGE_FAILURE_CLEANUP_MS + 1);
expect(settled).toBe(true);
expect((await saving).message).toBe("cancelled during source read");
expect(body.destroyed).toBe(true);
} finally {
release();
await vi.advanceTimersByTimeAsync(STORAGE_FAILURE_CLEANUP_MS + 1);
await saving;
send.mockRestore(); client.destroy(); vi.useRealTimers();
}
});
});
describe("file-backed S3 multipart uploads", () => {
const partBytes = 8 * 1024 * 1024;
async function sourceFile(parts: number, tail = 0) {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "pc-s3-file-part-"));
const file = path.join(dir, "source.bin");
const handle = await fs.open(file, "wx", 0o600);
try {
for (let index = 0; index < parts; index++) await handle.write(Buffer.alloc(partBytes, index + 1));
if (tail) await handle.write(Buffer.alloc(tail, 0xee));
} finally { await handle.close(); }
return { dir, file, size: parts * partBytes + tail };
}
const md5 = (bytes: Buffer) => createHash("md5").update(bytes).digest("base64");
async function drain(body: unknown) {
const chunks: Buffer[] = [];
for await (const chunk of body as Readable) chunks.push(Buffer.from(chunk));
return Buffer.concat(chunks);
}
it("uploads ranges in parallel and sends each part's Content-MD5", async () => {
const source = await sourceFile(10, 4096);
const client = new S3Client({ region: "us-east-1" });
let active = 0, peak = 0, release!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const seen = new Map<number, { length: number; md5: string; declared: string | undefined }>();
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
active++; peak = Math.max(peak, active);
if (active === FILE_PART_CONCURRENCY) release();
await gate;
const bytes = await drain(command.input.Body);
expect(bytes.length).toBe(command.input.ContentLength);
seen.set(command.input.PartNumber!, { length: bytes.length, md5: md5(bytes), declared: command.input.ContentMD5 });
active--;
return { ETag: `part-${command.input.PartNumber}` };
}
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
expect(command.input.MultipartUpload.Parts.map((part: { PartNumber: number }) => part.PartNumber)).toEqual([...Array(11).keys()].map(n => n + 1));
return {};
});
const body = Readable.from([Buffer.from("unused stream body")]);
try {
const receipt = await putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
contentLength: source.size, body, sourceFile: source.file });
expect(receipt).toEqual({ integrity: "server_verified_md5" });
expect(peak).toBe(FILE_PART_CONCURRENCY);
expect(body.destroyed).toBe(true);
expect(seen.size).toBe(11);
for (const [number, part] of seen) {
expect(part.declared).toBe(part.md5);
expect(part.length).toBe(number === 11 ? 4096 : partBytes);
}
expect(seen.get(3)!.md5).toBe(md5(Buffer.alloc(partBytes, 3)));
} finally { send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
});
it("resends a failed part from the file instead of failing the upload", async () => {
const source = await sourceFile(3);
const client = new S3Client({ region: "us-east-1" });
const attempts = new Map<number, number>();
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
const number = command.input.PartNumber!;
attempts.set(number, (attempts.get(number) ?? 0) + 1);
const bytes = await drain(command.input.Body);
if (number === 2 && attempts.get(number) === 1) throw new Error("An error was encountered in a non-retryable streaming request.");
expect(md5(bytes)).toBe(command.input.ContentMD5);
return { ETag: `part-${number}` };
}
if (command instanceof AbortMultipartUploadCommand) throw new Error("must not abort a recovered upload");
return {};
});
try {
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
contentLength: source.size, body: Buffer.alloc(0), sourceFile: source.file })).resolves.toEqual({ integrity: "server_verified_md5" });
expect(attempts.get(2)).toBe(2);
expect(send.mock.calls.some(([command]) => command instanceof CompleteMultipartUploadCommand)).toBe(true);
} finally { send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
});
it("aborts the upload only after every in-flight part settles when a part keeps failing", async () => {
const source = await sourceFile(4);
const client = new S3Client({ region: "us-east-1" });
let inFlight = 0, abortedWhileInFlight = false;
const send = vi.spyOn(client, "send").mockImplementation(async (command: any, options?: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
inFlight++;
try {
if (command.input.PartNumber === 1) throw new Error("storage refused part");
await new Promise<void>((_resolve, reject) => options?.abortSignal?.addEventListener("abort", () => reject(options.abortSignal!.reason), { once: true }));
return { ETag: "never" };
} finally { inFlight--; }
}
if (command instanceof AbortMultipartUploadCommand) { abortedWhileInFlight = inFlight > 0; return {}; }
throw new Error("must not complete a failed upload");
});
try {
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
contentLength: source.size, body: Buffer.alloc(0), sourceFile: source.file })).rejects.toThrow("storage refused part");
const partOneAttempts = send.mock.calls.filter(([command]) => command instanceof UploadPartCommand && command.input.PartNumber === 1).length;
expect(partOneAttempts).toBe(3);
expect(send.mock.calls.filter(([command]) => command instanceof AbortMultipartUploadCommand)).toHaveLength(1);
expect(abortedWhileInFlight).toBe(false);
} finally { send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
});
it.each(["cancelled", "removed"])("leaves no part digest running or waiting when the source is %s during the first round", async reason => {
const source = await sourceFile(FILE_PART_CONCURRENCY + 2);
const client = new S3Client({ region: "us-east-1" }), cancellation = new AbortController();
let digestsAtCleanup: ReturnType<typeof fileCryptoPoolLoad> | undefined;
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) {
// The digests of the first round start right after this reply.
if (reason === "cancelled") setImmediate(() => cancellation.abort(new Error("run cancelled")));
else await fs.rm(source.file);
return { UploadId: "upload" };
}
if (command instanceof AbortMultipartUploadCommand) { digestsAtCleanup = fileCryptoPoolLoad(); return {}; }
throw new Error("must not send a part or complete the upload");
});
try {
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
contentLength: source.size, body: Buffer.alloc(0), sourceFile: source.file, signal: cancellation.signal })).rejects.toThrow();
expect(digestsAtCleanup).toEqual({ running: 0, waiting: 0 });
expect(fileCryptoPoolLoad()).toEqual({ running: 0, waiting: 0 });
} finally { send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
});
it("refuses a source file whose size differs from the declared length before starting an upload", async () => {
const source = await sourceFile(3);
const client = new S3Client({ region: "us-east-1" });
const send = vi.spyOn(client, "send");
try {
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
contentLength: source.size + 1, body: Buffer.alloc(0), sourceFile: source.file })).rejects.toThrow("does not match its declared size");
expect(send).not.toHaveBeenCalled();
} finally { send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
});
it("sends a small file source as one PUT with Content-MD5 through the real S3 HTTP client", async () => {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "pc-s3-small-file-"));
const file = path.join(dir, "small.bin"), bytes = Buffer.from("verified snapshot bytes\n".repeat(1000));
await fs.writeFile(file, bytes, { mode: 0o600 });
const requests: Array<{ method: string; md5: string | undefined; body: Buffer }> = [];
const server = createServer((request, response) => {
const chunks: Buffer[] = [];
request.on("data", chunk => chunks.push(chunk));
request.on("end", () => {
requests.push({ method: request.method!, md5: request.headers["content-md5"] as string | undefined, body: Buffer.concat(chunks) });
response.writeHead(200, { ETag: '"small"' }).end();
});
});
await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); });
vi.stubEnv("AWS_ACCESS_KEY_ID", "fixture"); vi.stubEnv("AWS_SECRET_ACCESS_KEY", "fixture");
const service = createStorageService(createS3StorageProvider({ bucket: "bucket", region: "us-east-1", forcePathStyle: true,
endpoint: `http://127.0.0.1:${(server.address() as AddressInfo).port}` }));
try {
const result = await service.putFile({ companyId: "company", objectKey: "company/workspace-snapshots/small.pcws", namespace: "workspace-snapshots",
originalFilename: null, contentType: "application/octet-stream", body: Readable.from([bytes]), byteSize: bytes.length,
sha256: createHash("sha256").update(bytes).digest("hex"), sourceFile: file });
expect(result.integrity).toBe("server_verified_md5");
expect(requests).toHaveLength(1);
expect(requests[0]).toMatchObject({ method: "PUT", md5: md5(bytes) });
expect(requests[0]!.body.equals(bytes)).toBe(true);
} finally {
vi.unstubAllEnvs();
server.closeAllConnections();
await new Promise<void>(resolve => server.close(() => resolve()));
await fs.rm(dir, { recursive: true, force: true });
}
});
});