mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - With remote (sandbox) execution, every agent turn ends by saving the
workspace: capture an archive, encrypt it, upload it to object storage,
and verify it.
> - On a busy S3-backed instance, the upload step is the slowest part of
a save. A 1.5 GB workspace can hold a run open for minutes after the
agent finished.
> - The upload step reads the archive several times, uploads through one
stream with 4 parts in flight, and then downloads the whole object again
to verify it.
> - Parts that use the stream path share one process-wide 32 MiB buffer
window, so saves that run at the same time slow each other down.
> - This pull request uploads parts in parallel straight from the
encrypted file, lets the backend verify every part with Content-MD5, and
replaces the full read-back with a size check for those verified
uploads.
> - The benefit is faster saves, a smaller run-finalization window, and
no interference between concurrent saves.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Remote workspace snapshot upload (`uploadWorkspaceSnapshot`) to
S3-compatible storage.
**Subsystem affected**
Server storage (`server/src/storage/*`) and remote workspace snapshot
storage.
**Current behavior**
For one save, the upload step:
1. hashes the archive;
2. reads the archive again to encrypt it into a temporary file;
3. hashes the encrypted file;
4. uploads it as one stream with 4 parts in flight;
5. downloads the whole object to verify it.
On one hosted instance over 7 days (724 saves, median 731 MB), the
upload step ran at a median of 6.5 MB/s. With no other save running it
reached 11–14 MB/s. With six or more saves at the same time it fell to
4.4 MB/s. From the same region and with the same part settings, a plain
client reached 31 MB/s to the same kind of bucket.
**Proposed behavior**
Upload parts in parallel from byte ranges of the encrypted file, with
Content-MD5 on each part. Confirm only the stored size after a verified
upload. Digest the archive in the same pass that encrypts it.
**Reason and benefit**
Saves finish faster and hold their run open for less time. Concurrent
saves no longer serialize on the shared buffer window. A transient part
failure is retried from disk instead of failing the whole save.
**Breaking changes**
None. `StorageProvider.putObject` may now return a receipt. Providers
that return nothing keep the current behavior, including the full
read-back.
Related: none found.
## What Changed
- `storage/types.ts`, `storage/service.ts`: an upload can name its
`sourceFile`, a private file whose bytes are exactly the body.
`putObject` may return `{ integrity: "server_verified_md5" }`, and
`putFile` passes it through as `PutFileResult.integrity`.
- `storage/s3-multipart.ts`: new `putS3MultipartFile`.
- Each part reads its own byte range of the file. Up to 8 parts upload
at once, with no shared buffer window.
- Each part carries the Content-MD5 of its range.
- A failed part is resent from disk, up to 3 attempts.
- The first part that runs out of attempts stops its siblings. The
multipart upload is aborted only after every part settles.
- A source file whose size differs from the declared length is refused
before any request.
- `storage/s3-provider.ts`:
- A `sourceFile` larger than 16 MiB uses `putS3MultipartFile`. A smaller
one uses one PUT with Content-MD5.
- These requests use a second client with `requestChecksumCalculation:
"WHEN_REQUIRED"`, so bodies are plain and the backend verifies
Content-MD5 directly. The default client wraps bodies in aws-chunked
framing with a CRC32 trailer.
- Other uploads are unchanged.
- `services/remote-workspace-snapshot-storage.ts`:
- Encryption digests the plaintext in the same read. An unchanged file
identity (dev, inode, size, mtime, ctime) across that read replaces the
separate digest pass. A change still fails with `archive_changed` before
any upload.
- The encrypted file is still digested from disk, so the recorded hash
covers the exact bytes that the parts read.
- The upload passes `sourceFile`. When the provider reports
`server_verified_md5`, a HEAD confirms the stored size instead of
downloading the object.
- The full read-back stays for:
- other providers (local disk);
- recovery of earlier attempts;
- the caller that keeps a verified archive file.
## Verification
- `pnpm --filter @paperclipai/server typecheck`: exit 0.
- `vitest run src/__tests__/storage-s3-multipart.test.ts
src/services/remote-workspace-snapshot-storage.test.ts`: 59 passed. New
tests:
- every part carries the Content-MD5 of its range, and 8 parts are in
flight;
- a failed part is resent and the upload completes;
- a part that keeps failing aborts the upload only after in-flight
siblings settle;
- a size mismatch is refused before any request;
- a small file goes as one PUT with Content-MD5, through a real HTTP
server;
- the read-back is skipped only for a server-verified upload, and the
snapshot still restores;
- the read-back stays when the caller keeps the read-back file;
- a stored size mismatch is rejected;
- an archive that changes during encryption is rejected before upload.
- With the source change reverted, the three new snapshot tests fail.
- Real S3-compatible bucket (a Railway bucket, run from a workstation):
- A 3 MiB single PUT stored exactly: HEAD size, ETag and read-back
SHA-256 all matched.
- A 96 MiB, 13-part upload stored exactly, with the same checks.
- A part with a wrong Content-MD5 was rejected with `BadDigest`.
- The same backend stored a wrong `x-amz-checksum-sha256` without
checking it. That is why this change uses Content-MD5.
- Measured from a container in the app's region, 1 GiB with the app's
settings (8 MiB parts, 4 in flight), to the same kind of bucket: 31
MB/s. With 16 in flight: 69 MB/s.
- 77 storage-related suites: the only failures are the same 19 that fail
on unmodified `master` in this checkout. They are in 4 files: missing
bundled connector skills and an unbuilt local runner daemon.
## Risks
- Integrity now depends on the backend enforcing Content-MD5 for file
uploads. AWS S3 and the tested S3-compatible backend enforce it. A
backend that ignores Content-MD5 would accept corrupted parts without a
read-back. A later restore still fails closed: AES-GCM authentication
and the plaintext SHA-256 are checked on load.
- Up to 8 parallel part requests per save, instead of 4 shared ones.
Memory per part is a 64 KiB stream buffer.
- Archive-change detection now uses file identity instead of a second
digest. The archive lives in a private temporary directory that the
capture step creates.
- Medium risk overall: the change is limited to file-sourced S3 uploads
and the snapshot verify step.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs, S3 API probes). AI-assisted
investigation, change and tests, with human direction.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
docs describe this path)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
579 lines
33 KiB
TypeScript
579 lines
33 KiB
TypeScript
import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, DeleteObjectCommand, PutObjectCommand, S3Client } from "@aws-sdk/client-s3";
|
|
import { Readable } from "node:stream";
|
|
import { createHash } from "node:crypto";
|
|
import { createServer, type ServerResponse } from "node:http";
|
|
import type { AddressInfo } from "node:net";
|
|
import { describe, expect, it, vi } from "vitest";
|
|
import { FILE_PART_CONCURRENCY, putS3Multipart, putS3MultipartFile } from "../storage/s3-multipart.js";
|
|
import fs from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { createS3StorageProvider } from "../storage/s3-provider.js";
|
|
import { createStorageService } from "../storage/service.js";
|
|
import { STORAGE_FAILURE_CLEANUP_MS } from "../storage/failure-cleanup.js";
|
|
|
|
describe("streamed S3 multipart uploads", () => {
|
|
it("delivers four slow part payloads concurrently before any part is drained", async () => {
|
|
const client = new S3Client({ region: "us-east-1" }), started = new Set<number>(), received = new Map<number, number>();
|
|
let release!: () => void, active = 0, peak = 0, completed = false;
|
|
const reading = new Promise<void>(resolve => { release = resolve; });
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
const number = command.input.PartNumber!; started.add(number);
|
|
await reading; active++; peak = Math.max(peak, active);
|
|
let bytes = 0; const digest = createHash("sha256");
|
|
for await (const chunk of command.input.Body as Readable) {
|
|
digest.update(chunk); bytes += chunk.length;
|
|
await new Promise(resolve => setTimeout(resolve, 1));
|
|
}
|
|
expect(bytes).toBe(8 * 1024 * 1024);
|
|
expect(digest.digest("hex")).toBe(createHash("sha256").update(Buffer.alloc(bytes, number)).digest("hex"));
|
|
received.set(number, bytes); active--;
|
|
return { ETag: `part-${number}` };
|
|
}
|
|
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
|
|
expect(command.input.MultipartUpload.Parts).toEqual([1, 2, 3, 4].map(PartNumber => ({ PartNumber, ETag: `part-${PartNumber}` })));
|
|
completed = true; return {};
|
|
});
|
|
const body = Readable.from((async function* () {
|
|
for (let index = 0; index < 512; index++) yield Buffer.alloc(64 * 1024, Math.floor(index / 128) + 1);
|
|
})());
|
|
const saving = putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream", contentLength: 32 * 1024 * 1024, body }).catch(error => error);
|
|
try {
|
|
await expect.poll(() => started.size, { timeout: 1500 }).toBe(4);
|
|
expect(received.size).toBe(0); release();
|
|
expect(await saving).toBeUndefined();
|
|
expect(peak).toBe(4); expect(received.size).toBe(4); expect(completed).toBe(true);
|
|
} finally { release(); await saving; send.mockRestore(); client.destroy(); }
|
|
});
|
|
it("sends concurrent exact part bodies through the real S3 HTTP client", async () => {
|
|
const received = new Map<number, string>();
|
|
const responses = new Map<number, ServerResponse>();
|
|
let completedParts = "";
|
|
const server = createServer((request, response) => {
|
|
const url = new URL(request.url!, "http://localhost");
|
|
const digest = createHash("sha256");
|
|
let text = "";
|
|
request.on("data", chunk => { digest.update(chunk); if (request.method === "POST") text += chunk; });
|
|
request.on("end", () => {
|
|
if (request.method === "PUT") {
|
|
const number = Number(url.searchParams.get("partNumber"));
|
|
received.set(number, digest.digest("hex"));
|
|
responses.set(number, response);
|
|
} else if (url.searchParams.has("uploads")) {
|
|
response.end('<InitiateMultipartUploadResult><UploadId>upload</UploadId></InitiateMultipartUploadResult>');
|
|
} else {
|
|
completedParts = text;
|
|
response.end('<CompleteMultipartUploadResult><ETag>complete</ETag></CompleteMultipartUploadResult>');
|
|
}
|
|
});
|
|
});
|
|
await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); });
|
|
const client = new S3Client({ region: "us-east-1", endpoint: `http://127.0.0.1:${(server.address() as AddressInfo).port}`,
|
|
forcePathStyle: true, requestChecksumCalculation: "WHEN_REQUIRED", credentials: { accessKeyId: "fixture", secretAccessKey: "fixture" } });
|
|
const cancellation = new AbortController();
|
|
const body = Readable.from((async function* () {
|
|
for (let index = 0; index < 512; index++) yield Buffer.alloc(64 * 1024, Math.floor(index / 128) + 1);
|
|
})());
|
|
const saving = putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream", contentLength: 32 * 1024 * 1024, body, signal: cancellation.signal }).catch(error => error);
|
|
try {
|
|
await expect.poll(() => received.size, { timeout: 3000 }).toBe(4);
|
|
for (const number of [4, 2, 3, 1]) {
|
|
expect(received.get(number)).toBe(createHash("sha256").update(Buffer.alloc(8 * 1024 * 1024, number)).digest("hex"));
|
|
responses.get(number)!.writeHead(200, { ETag: `part-${number}` }).end();
|
|
}
|
|
expect(await saving).toBeUndefined();
|
|
expect([...completedParts.matchAll(/<PartNumber>(\d+)<\/PartNumber>/g)].map(match => Number(match[1]))).toEqual([1, 2, 3, 4]);
|
|
} finally {
|
|
cancellation.abort();
|
|
await saving;
|
|
client.destroy();
|
|
server.closeAllConnections();
|
|
await new Promise<void>(resolve => server.close(() => resolve()));
|
|
}
|
|
});
|
|
|
|
it("overlaps slow part acknowledgements and completes exact bytes in source order", async () => {
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
const received = new Map<number, Buffer>();
|
|
const acknowledgements = new Map<number, () => void>();
|
|
let completed = false;
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
const chunks: Buffer[] = [];
|
|
for await (const chunk of command.input.Body as Readable) chunks.push(chunk);
|
|
received.set(command.input.PartNumber!, Buffer.concat(chunks));
|
|
await new Promise<void>(resolve => acknowledgements.set(command.input.PartNumber!, resolve));
|
|
return { ETag: `part-${command.input.PartNumber}` };
|
|
}
|
|
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
|
|
expect(command.input.MultipartUpload.Parts).toEqual([1, 2, 3, 4].map(PartNumber => ({ PartNumber, ETag: `part-${PartNumber}` })));
|
|
completed = true;
|
|
return {};
|
|
});
|
|
const length = 25 * 1024 * 1024;
|
|
const body = Readable.from((async function* () {
|
|
for (let offset = 0; offset < length; offset += 64 * 1024) yield Buffer.alloc(Math.min(64 * 1024, length - offset), offset / (64 * 1024) % 251);
|
|
})());
|
|
const saving = putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream", contentLength: length, body });
|
|
try {
|
|
await expect.poll(() => received.size, { timeout: 1000 }).toBe(4);
|
|
expect(completed).toBe(false);
|
|
for (const number of [4, 2, 3, 1]) acknowledgements.get(number)!();
|
|
await saving;
|
|
const actual = Buffer.concat([1, 2, 3, 4].map(number => received.get(number)!));
|
|
expect(actual.length).toBe(length);
|
|
for (let offset = 0; offset < length; offset += 64 * 1024) {
|
|
expect(actual.subarray(offset, Math.min(length, offset + 64 * 1024)).every(byte => byte === offset / (64 * 1024) % 251)).toBe(true);
|
|
}
|
|
expect(completed).toBe(true);
|
|
} finally {
|
|
const release = setInterval(() => { for (const acknowledge of acknowledgements.values()) acknowledge(); }, 1);
|
|
try { await saving; } finally { clearInterval(release); send.mockRestore(); client.destroy(); }
|
|
}
|
|
});
|
|
|
|
it("streams exact ordered parts without buffering the full response", async () => {
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
const lengths = new Map<number, number>();
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
expect(command.input.Body).toBeInstanceOf(Readable);
|
|
let length = 0;
|
|
for await (const chunk of command.input.Body as Readable) {
|
|
expect(chunk.length).toBeLessThanOrEqual(64 * 1024);
|
|
length += chunk.length;
|
|
}
|
|
expect(length).toBe(command.input.ContentLength);
|
|
lengths.set(command.input.PartNumber!, length);
|
|
return { ETag: `part-${command.input.PartNumber}` };
|
|
}
|
|
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
|
|
expect(command.input.MultipartUpload.Parts).toEqual([1, 2, 3].map(PartNumber => ({ PartNumber, ETag: `part-${PartNumber}` })));
|
|
return {};
|
|
});
|
|
const body = Readable.from((async function* () { for (let i = 0; i < 272; i++) yield Buffer.alloc(64 * 1024); })());
|
|
try {
|
|
await putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "text/plain", contentLength: 17 * 1024 * 1024, body });
|
|
expect([...lengths].sort(([a], [b]) => a - b)).toEqual([[1, 8], [2, 8], [3, 1]].map(([part, mb]) => [part, mb * 1024 * 1024]));
|
|
} finally { send.mockRestore(); client.destroy(); }
|
|
});
|
|
it("keeps source backpressure and at most four part requests while acknowledgements stall", async () => {
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
let releaseReading!: () => void, releaseAcknowledgements!: () => void;
|
|
const reading = new Promise<void>(resolve => { releaseReading = resolve; });
|
|
const acknowledgements = new Promise<void>(resolve => { releaseAcknowledgements = resolve; });
|
|
let yielded = 0, active = 0, peak = 0, received = 0;
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
active++; peak = Math.max(peak, active);
|
|
await reading;
|
|
for await (const chunk of command.input.Body as Readable) expect(chunk.length).toBeLessThanOrEqual(64 * 1024);
|
|
received++;
|
|
await acknowledgements;
|
|
active--;
|
|
return { ETag: `part-${command.input.PartNumber}` };
|
|
}
|
|
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
|
|
return {};
|
|
});
|
|
const body = Readable.from((async function* () {
|
|
for (let index = 0; index < 1600; index++) { yielded++; yield Buffer.alloc(64 * 1024); }
|
|
})());
|
|
const saving = putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream", contentLength: 100 * 1024 * 1024, body });
|
|
try {
|
|
await expect.poll(() => active).toBe(4);
|
|
expect(yielded).toBeLessThanOrEqual(4 * 128 + 3);
|
|
releaseReading();
|
|
await expect.poll(() => received).toBe(4);
|
|
expect(yielded).toBeLessThanOrEqual(4 * 128 + 3);
|
|
expect(peak).toBe(4);
|
|
releaseAcknowledgements();
|
|
await saving;
|
|
expect(received).toBe(13);
|
|
expect(yielded).toBe(1600);
|
|
expect(peak).toBe(4);
|
|
} finally { releaseReading(); releaseAcknowledgements(); await saving; send.mockRestore(); client.destroy(); }
|
|
});
|
|
|
|
it("shares the buffered payload window across simultaneous uploads", async () => {
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
let release!: () => void, yielded = 0;
|
|
const reading = new Promise<void>(resolve => { release = resolve; });
|
|
const started = new Map<string, number>();
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: command.input.Key };
|
|
if (command instanceof UploadPartCommand) {
|
|
const key = command.input.Key!;
|
|
started.set(key, (started.get(key) ?? 0) + 1);
|
|
await reading;
|
|
let bytes = 0;
|
|
for await (const chunk of command.input.Body as Readable) bytes += chunk.length;
|
|
expect(bytes).toBe(command.input.ContentLength);
|
|
return { ETag: `part-${command.input.PartNumber}` };
|
|
}
|
|
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
|
|
return {};
|
|
});
|
|
const saving = Promise.all(["a", "b"].map(key => putS3Multipart(client, "bucket", key, {
|
|
objectKey: key, contentType: "application/octet-stream", contentLength: 32 * 1024 * 1024,
|
|
body: Readable.from((async function* () {
|
|
for (let index = 0; index < 512; index++) { yielded++; yield Buffer.alloc(64 * 1024); }
|
|
})()),
|
|
})));
|
|
try {
|
|
await expect.poll(() => ["a", "b"].every(key => (started.get(key) ?? 0) >= 3)).toBe(true);
|
|
expect(yielded).toBeLessThanOrEqual(4 * 128 + 8);
|
|
release(); await saving;
|
|
expect(yielded).toBe(1024);
|
|
expect([...started.values()]).toEqual([4, 4]);
|
|
} finally { release(); await saving; send.mockRestore(); client.destroy(); }
|
|
});
|
|
|
|
it.each(["source", "request"])("releases the shared payload allowance after a %s failure", async failure => {
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
let release!: () => void, yielded = 0, started = 0;
|
|
const reading = new Promise<void>(resolve => { release = resolve; });
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: command.input.Key };
|
|
if (command instanceof UploadPartCommand) {
|
|
if (command.input.Key === "failed") throw new Error("request failed");
|
|
started++; await reading;
|
|
for await (const _chunk of command.input.Body as Readable) {}
|
|
return { ETag: `part-${command.input.PartNumber}` };
|
|
}
|
|
expect(command instanceof CompleteMultipartUploadCommand || command instanceof AbortMultipartUploadCommand).toBe(true);
|
|
return {};
|
|
});
|
|
const first = Readable.from((async function* () {
|
|
if (failure === "source") throw new Error("source failed");
|
|
yield Buffer.alloc(8 * 1024 * 1024);
|
|
})());
|
|
let saving: Promise<void> | undefined;
|
|
try {
|
|
await expect(putS3Multipart(client, "bucket", "failed", {
|
|
objectKey: "failed", contentType: "application/octet-stream", contentLength: 8 * 1024 * 1024, body: first,
|
|
})).rejects.toThrow(`${failure} failed`);
|
|
saving = putS3Multipart(client, "bucket", "next", {
|
|
objectKey: "next", contentType: "application/octet-stream", contentLength: 32 * 1024 * 1024,
|
|
body: Readable.from((async function* () {
|
|
for (let index = 0; index < 512; index++) { yielded++; yield Buffer.alloc(64 * 1024); }
|
|
})()),
|
|
});
|
|
await expect.poll(() => started).toBe(4);
|
|
expect(yielded).toBe(512);
|
|
release(); await saving;
|
|
} finally { release(); await saving; send.mockRestore(); client.destroy(); }
|
|
});
|
|
|
|
it("settles cancelled siblings before aborting a failed multipart upload", async () => {
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
const cancellations: AbortSignal[] = [];
|
|
let failPart!: () => void, releaseSiblings!: () => void;
|
|
const settlement = new Promise<void>(resolve => { releaseSiblings = resolve; });
|
|
let received = 0, settled = 0, aborted = false, completed = false;
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any, options: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
for await (const _chunk of command.input.Body as Readable) {}
|
|
received++;
|
|
if (command.input.PartNumber === 2) return new Promise((_resolve, reject) => { failPart = () => reject(new Error("second part failed")); });
|
|
cancellations.push(options.abortSignal);
|
|
await new Promise<void>(resolve => options.abortSignal.addEventListener("abort", () => resolve(), { once: true }));
|
|
await settlement;
|
|
settled++;
|
|
throw options.abortSignal.reason;
|
|
}
|
|
if (command instanceof CompleteMultipartUploadCommand) { completed = true; return {}; }
|
|
expect(command).toBeInstanceOf(AbortMultipartUploadCommand);
|
|
expect(settled).toBe(3);
|
|
aborted = true;
|
|
return {};
|
|
});
|
|
const body = Readable.from((async function* () { for (let index = 0; index < 512; index++) yield Buffer.alloc(64 * 1024); })());
|
|
const saving = putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream", contentLength: 32 * 1024 * 1024, body }).catch(error => error);
|
|
try {
|
|
await expect.poll(() => received).toBe(4);
|
|
failPart();
|
|
await expect.poll(() => cancellations.every(signal => signal.aborted)).toBe(true);
|
|
expect(aborted).toBe(false);
|
|
releaseSiblings();
|
|
expect((await saving).message).toBe("second part failed");
|
|
expect(aborted).toBe(true);
|
|
expect(completed).toBe(false);
|
|
expect(body.destroyed).toBe(true);
|
|
} finally { failPart?.(); releaseSiblings(); await saving; send.mockRestore(); client.destroy(); }
|
|
});
|
|
|
|
it.each(["truncated", "excess", "source error"])("rejects %s input without publishing an object", async kind => {
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
let aborted = false;
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
for await (const _chunk of command.input.Body as Readable) {}
|
|
return { ETag: `part-${command.input.PartNumber}` };
|
|
}
|
|
expect(command).toBeInstanceOf(AbortMultipartUploadCommand);
|
|
aborted = true;
|
|
return {};
|
|
});
|
|
const expected = { truncated: "S3 upload source was truncated", excess: "S3 upload source exceeds its declared size", "source error": "source unavailable" };
|
|
const body = Readable.from((async function* () {
|
|
for (let index = 0; index < (kind === "excess" ? 274 : 270); index++) yield Buffer.alloc(64 * 1024);
|
|
if (kind === "source error") throw new Error("source unavailable");
|
|
})());
|
|
try {
|
|
await expect(putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream", contentLength: 17 * 1024 * 1024, body })).rejects.toThrow(expected[kind as keyof typeof expected]);
|
|
expect(aborted).toBe(true);
|
|
expect(body.destroyed).toBe(true);
|
|
} finally { send.mockRestore(); client.destroy(); }
|
|
});
|
|
|
|
it("aborts an incomplete upload and closes its source on failure", async () => {
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) throw new Error("storage unavailable");
|
|
expect(command).toBeInstanceOf(AbortMultipartUploadCommand);
|
|
return {};
|
|
});
|
|
const body = Readable.from([Buffer.from("abc")]);
|
|
try {
|
|
await expect(putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "text/plain", contentLength: 3, body })).rejects.toThrow("storage unavailable");
|
|
expect(send).toHaveBeenCalledTimes(3);
|
|
expect(body.destroyed).toBe(true);
|
|
} finally { send.mockRestore(); client.destroy(); }
|
|
});
|
|
it("bounds stalled multipart abort and object cleanup after cancellation", async () => {
|
|
vi.useFakeTimers();
|
|
const cancellation = new AbortController();
|
|
const cleanupSignals: AbortSignal[] = [];
|
|
const send = vi.spyOn(S3Client.prototype, "send").mockImplementation(async (command: any, options: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) return new Promise((_resolve, reject) => {
|
|
options.abortSignal.addEventListener("abort", () => reject(cancellation.signal.reason), { once: true });
|
|
});
|
|
if (command instanceof AbortMultipartUploadCommand || command instanceof DeleteObjectCommand) {
|
|
cleanupSignals.push(options.abortSignal);
|
|
return new Promise(() => {});
|
|
}
|
|
expect(command).toBeInstanceOf(PutObjectCommand);
|
|
return {};
|
|
});
|
|
const storage = createStorageService(createS3StorageProvider({ bucket: "test", region: "us-east-1" }));
|
|
const body = Readable.from([Buffer.alloc(17 * 1024 * 1024)]);
|
|
try {
|
|
const saving = storage.putFile({ companyId: "company", namespace: "skill-packages", objectKey: "company/package/archive",
|
|
originalFilename: "archive", contentType: "application/gzip", body, byteSize: 17 * 1024 * 1024, sha256: "a".repeat(64), signal: cancellation.signal }).catch(error => error);
|
|
await vi.advanceTimersByTimeAsync(0);
|
|
cancellation.abort(new Error("cancelled upload"));
|
|
await vi.advanceTimersByTimeAsync(STORAGE_FAILURE_CLEANUP_MS * 2 + 1);
|
|
expect((await saving).message).toBe("cancelled upload");
|
|
expect(cleanupSignals).toHaveLength(2);
|
|
expect(cleanupSignals.every(signal => signal.aborted)).toBe(true);
|
|
expect(body.destroyed).toBe(true);
|
|
await expect(storage.putFile({ companyId: "company", namespace: "skill-packages", originalFilename: "next", contentType: "text/plain", body: Buffer.from("next") })).resolves.toMatchObject({ byteSize: 4 });
|
|
} finally { send.mockRestore(); vi.useRealTimers(); }
|
|
});
|
|
|
|
it.each([128 * 1024, 8 * 1024 * 1024 * 10_000 + 1])("bounds cancellation while an asynchronous source read is stalled (declared bytes: %s)", async contentLength => {
|
|
vi.useFakeTimers();
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
const cancellation = new AbortController();
|
|
let release!: () => void, waiting = false, settled = false, partLength: number | undefined;
|
|
const sourceGate = new Promise<void>(resolve => { release = resolve; });
|
|
const body = Readable.from((async function* () {
|
|
yield Buffer.alloc(64 * 1024);
|
|
waiting = true;
|
|
await sourceGate;
|
|
yield Buffer.alloc(64 * 1024);
|
|
})());
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any, options: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
partLength = command.input.ContentLength;
|
|
(command.input.Body as Readable).resume();
|
|
return new Promise((_resolve, reject) => options.abortSignal.addEventListener("abort", () => reject(options.abortSignal.reason), { once: true }));
|
|
}
|
|
expect(command).toBeInstanceOf(AbortMultipartUploadCommand);
|
|
return {};
|
|
});
|
|
const saving = putS3Multipart(client, "bucket", "key", { objectKey: "key", contentType: "text/plain", contentLength, body, signal: cancellation.signal })
|
|
.catch(error => error).finally(() => { settled = true; });
|
|
try {
|
|
await vi.advanceTimersByTimeAsync(0);
|
|
expect(waiting).toBe(true);
|
|
if (contentLength > 8 * 1024 * 1024 * 10_000) expect(partLength).toBe(8 * 1024 * 1024 + 1);
|
|
cancellation.abort(new Error("cancelled during source read"));
|
|
await vi.advanceTimersByTimeAsync(STORAGE_FAILURE_CLEANUP_MS + 1);
|
|
expect(settled).toBe(true);
|
|
expect((await saving).message).toBe("cancelled during source read");
|
|
expect(body.destroyed).toBe(true);
|
|
} finally {
|
|
release();
|
|
await vi.advanceTimersByTimeAsync(STORAGE_FAILURE_CLEANUP_MS + 1);
|
|
await saving;
|
|
send.mockRestore(); client.destroy(); vi.useRealTimers();
|
|
}
|
|
});
|
|
});
|
|
|
|
describe("file-backed S3 multipart uploads", () => {
|
|
const partBytes = 8 * 1024 * 1024;
|
|
async function sourceFile(parts: number, tail = 0) {
|
|
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "pc-s3-file-part-"));
|
|
const file = path.join(dir, "source.bin");
|
|
const handle = await fs.open(file, "wx", 0o600);
|
|
try {
|
|
for (let index = 0; index < parts; index++) await handle.write(Buffer.alloc(partBytes, index + 1));
|
|
if (tail) await handle.write(Buffer.alloc(tail, 0xee));
|
|
} finally { await handle.close(); }
|
|
return { dir, file, size: parts * partBytes + tail };
|
|
}
|
|
const md5 = (bytes: Buffer) => createHash("md5").update(bytes).digest("base64");
|
|
async function drain(body: unknown) {
|
|
const chunks: Buffer[] = [];
|
|
for await (const chunk of body as Readable) chunks.push(Buffer.from(chunk));
|
|
return Buffer.concat(chunks);
|
|
}
|
|
|
|
it("uploads ranges in parallel and sends each part's Content-MD5", async () => {
|
|
const source = await sourceFile(10, 4096);
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
let active = 0, peak = 0, release!: () => void;
|
|
const gate = new Promise<void>(resolve => { release = resolve; });
|
|
const seen = new Map<number, { length: number; md5: string; declared: string | undefined }>();
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
active++; peak = Math.max(peak, active);
|
|
if (active === FILE_PART_CONCURRENCY) release();
|
|
await gate;
|
|
const bytes = await drain(command.input.Body);
|
|
expect(bytes.length).toBe(command.input.ContentLength);
|
|
seen.set(command.input.PartNumber!, { length: bytes.length, md5: md5(bytes), declared: command.input.ContentMD5 });
|
|
active--;
|
|
return { ETag: `part-${command.input.PartNumber}` };
|
|
}
|
|
expect(command).toBeInstanceOf(CompleteMultipartUploadCommand);
|
|
expect(command.input.MultipartUpload.Parts.map((part: { PartNumber: number }) => part.PartNumber)).toEqual([...Array(11).keys()].map(n => n + 1));
|
|
return {};
|
|
});
|
|
const body = Readable.from([Buffer.from("unused stream body")]);
|
|
try {
|
|
const receipt = await putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
|
|
contentLength: source.size, body, sourceFile: source.file });
|
|
expect(receipt).toEqual({ integrity: "server_verified_md5" });
|
|
expect(peak).toBe(FILE_PART_CONCURRENCY);
|
|
expect(body.destroyed).toBe(true);
|
|
expect(seen.size).toBe(11);
|
|
for (const [number, part] of seen) {
|
|
expect(part.declared).toBe(part.md5);
|
|
expect(part.length).toBe(number === 11 ? 4096 : partBytes);
|
|
}
|
|
expect(seen.get(3)!.md5).toBe(md5(Buffer.alloc(partBytes, 3)));
|
|
} finally { send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
|
|
});
|
|
|
|
it("resends a failed part from the file instead of failing the upload", async () => {
|
|
const source = await sourceFile(3);
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
const attempts = new Map<number, number>();
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
const number = command.input.PartNumber!;
|
|
attempts.set(number, (attempts.get(number) ?? 0) + 1);
|
|
const bytes = await drain(command.input.Body);
|
|
if (number === 2 && attempts.get(number) === 1) throw new Error("An error was encountered in a non-retryable streaming request.");
|
|
expect(md5(bytes)).toBe(command.input.ContentMD5);
|
|
return { ETag: `part-${number}` };
|
|
}
|
|
if (command instanceof AbortMultipartUploadCommand) throw new Error("must not abort a recovered upload");
|
|
return {};
|
|
});
|
|
try {
|
|
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
|
|
contentLength: source.size, body: Buffer.alloc(0), sourceFile: source.file })).resolves.toEqual({ integrity: "server_verified_md5" });
|
|
expect(attempts.get(2)).toBe(2);
|
|
expect(send.mock.calls.some(([command]) => command instanceof CompleteMultipartUploadCommand)).toBe(true);
|
|
} finally { send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
|
|
});
|
|
|
|
it("aborts the upload only after every in-flight part settles when a part keeps failing", async () => {
|
|
const source = await sourceFile(4);
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
let inFlight = 0, abortedWhileInFlight = false;
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any, options?: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
inFlight++;
|
|
try {
|
|
if (command.input.PartNumber === 1) throw new Error("storage refused part");
|
|
await new Promise<void>((_resolve, reject) => options?.abortSignal?.addEventListener("abort", () => reject(options.abortSignal!.reason), { once: true }));
|
|
return { ETag: "never" };
|
|
} finally { inFlight--; }
|
|
}
|
|
if (command instanceof AbortMultipartUploadCommand) { abortedWhileInFlight = inFlight > 0; return {}; }
|
|
throw new Error("must not complete a failed upload");
|
|
});
|
|
try {
|
|
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
|
|
contentLength: source.size, body: Buffer.alloc(0), sourceFile: source.file })).rejects.toThrow("storage refused part");
|
|
const partOneAttempts = send.mock.calls.filter(([command]) => command instanceof UploadPartCommand && command.input.PartNumber === 1).length;
|
|
expect(partOneAttempts).toBe(3);
|
|
expect(send.mock.calls.filter(([command]) => command instanceof AbortMultipartUploadCommand)).toHaveLength(1);
|
|
expect(abortedWhileInFlight).toBe(false);
|
|
} finally { send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
|
|
});
|
|
|
|
it("refuses a source file whose size differs from the declared length before starting an upload", async () => {
|
|
const source = await sourceFile(3);
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
const send = vi.spyOn(client, "send");
|
|
try {
|
|
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
|
|
contentLength: source.size + 1, body: Buffer.alloc(0), sourceFile: source.file })).rejects.toThrow("does not match its declared size");
|
|
expect(send).not.toHaveBeenCalled();
|
|
} finally { send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
|
|
});
|
|
|
|
it("sends a small file source as one PUT with Content-MD5 through the real S3 HTTP client", async () => {
|
|
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "pc-s3-small-file-"));
|
|
const file = path.join(dir, "small.bin"), bytes = Buffer.from("verified snapshot bytes\n".repeat(1000));
|
|
await fs.writeFile(file, bytes, { mode: 0o600 });
|
|
const requests: Array<{ method: string; md5: string | undefined; body: Buffer }> = [];
|
|
const server = createServer((request, response) => {
|
|
const chunks: Buffer[] = [];
|
|
request.on("data", chunk => chunks.push(chunk));
|
|
request.on("end", () => {
|
|
requests.push({ method: request.method!, md5: request.headers["content-md5"] as string | undefined, body: Buffer.concat(chunks) });
|
|
response.writeHead(200, { ETag: '"small"' }).end();
|
|
});
|
|
});
|
|
await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); });
|
|
vi.stubEnv("AWS_ACCESS_KEY_ID", "fixture"); vi.stubEnv("AWS_SECRET_ACCESS_KEY", "fixture");
|
|
const service = createStorageService(createS3StorageProvider({ bucket: "bucket", region: "us-east-1", forcePathStyle: true,
|
|
endpoint: `http://127.0.0.1:${(server.address() as AddressInfo).port}` }));
|
|
try {
|
|
const result = await service.putFile({ companyId: "company", objectKey: "company/workspace-snapshots/small.pcws", namespace: "workspace-snapshots",
|
|
originalFilename: null, contentType: "application/octet-stream", body: Readable.from([bytes]), byteSize: bytes.length,
|
|
sha256: createHash("sha256").update(bytes).digest("hex"), sourceFile: file });
|
|
expect(result.integrity).toBe("server_verified_md5");
|
|
expect(requests).toHaveLength(1);
|
|
expect(requests[0]).toMatchObject({ method: "PUT", md5: md5(bytes) });
|
|
expect(requests[0]!.body.equals(bytes)).toBe(true);
|
|
} finally {
|
|
vi.unstubAllEnvs();
|
|
server.closeAllConnections();
|
|
await new Promise<void>(resolve => server.close(() => resolve()));
|
|
await fs.rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|