Files
PaperClipAI/server/src/__tests__/storage-s3-multipart-throttle.test.ts
T
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00

174 lines
9.9 KiB
TypeScript

import { AbortMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, S3Client } from "@aws-sdk/client-s3";
import { Readable } from "node:stream";
import { setImmediate as nextEventLoopTurn } from "node:timers/promises";
import { createHash } from "node:crypto";
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { describe, expect, it, vi } from "vitest";
import { FILE_PART_CONCURRENCY, THROTTLED_PART_ATTEMPTS, putS3MultipartFile } from "../storage/s3-multipart.js";
import { createS3StorageProvider } from "../storage/s3-provider.js";
import { createStorageService } from "../storage/service.js";
import * as fileCrypto from "../storage/file-crypto-pool.js";
const partBytes = 8 * 1024 * 1024;
async function sourceFile(parts: number) {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "pc-s3-throttle-"));
const file = path.join(dir, "source.bin");
const handle = await fs.open(file, "wx", 0o600);
try {
for (let index = 0; index < parts; index++) await handle.write(Buffer.alloc(partBytes, index + 1));
} finally { await handle.close(); }
return { dir, file, size: parts * partBytes };
}
const md5 = (bytes: Buffer) => createHash("md5").update(bytes).digest("base64");
async function drain(body: unknown) {
const chunks: Buffer[] = [];
for await (const chunk of body as Readable) chunks.push(Buffer.from(chunk));
return Buffer.concat(chunks);
}
const slowDown = () => Object.assign(new Error("Please reduce your request rate."), { name: "SlowDown", $metadata: { httpStatusCode: 503 } });
describe("file-backed S3 multipart uploads under a request-rate limit", () => {
it("backs off and completes when the real S3 client receives SlowDown for one part", async () => {
const source = await sourceFile(3);
const attempts = new Map<number, number>();
let completed = "";
const server = createServer((request, response) => {
const url = new URL(request.url!, "http://localhost");
const chunks: Buffer[] = [];
request.on("data", chunk => chunks.push(chunk));
request.on("end", () => {
if (url.searchParams.has("uploads")) {
response.end("<InitiateMultipartUploadResult><UploadId>upload</UploadId></InitiateMultipartUploadResult>");
} else if (request.method === "PUT") {
const number = Number(url.searchParams.get("partNumber"));
attempts.set(number, (attempts.get(number) ?? 0) + 1);
if (number === 2 && attempts.get(number)! <= 3) {
response.writeHead(503, { "Content-Type": "application/xml" })
.end("<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><RequestId>throttled</RequestId></Error>");
} else {
expect(md5(Buffer.concat(chunks))).toBe(request.headers["content-md5"]);
response.writeHead(200, { ETag: `"part-${number}"` }).end();
}
} else if (request.method === "POST") {
completed = Buffer.concat(chunks).toString();
response.end("<CompleteMultipartUploadResult><ETag>complete</ETag></CompleteMultipartUploadResult>");
} else {
response.writeHead(500).end();
}
});
});
await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); });
vi.stubEnv("AWS_ACCESS_KEY_ID", "fixture"); vi.stubEnv("AWS_SECRET_ACCESS_KEY", "fixture");
const random = vi.spyOn(Math, "random").mockReturnValue(0);
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
const service = createStorageService(createS3StorageProvider({ bucket: "bucket", region: "us-east-1", forcePathStyle: true,
endpoint: `http://127.0.0.1:${(server.address() as AddressInfo).port}` }));
try {
const result = await service.putFile({ companyId: "company", objectKey: "company/workspace-snapshots/throttled.pcws", namespace: "workspace-snapshots",
originalFilename: null, contentType: "application/octet-stream", body: Readable.from([Buffer.alloc(0)]), byteSize: source.size,
sha256: createHash("sha256").update(await fs.readFile(source.file)).digest("hex"), sourceFile: source.file });
expect(result.integrity).toBe("server_verified_md5");
expect(attempts.get(2)).toBe(4);
expect([...completed.matchAll(/<PartNumber>(\d+)<\/PartNumber>/g)].map(match => Number(match[1]))).toEqual([1, 2, 3]);
} finally {
random.mockRestore(); warn.mockRestore(); vi.unstubAllEnvs();
server.closeAllConnections();
await new Promise<void>(resolve => server.close(() => resolve()));
await fs.rm(source.dir, { recursive: true, force: true });
}
});
it("halves new parts in flight once when a parallel burst of parts is throttled", async () => {
const source = await sourceFile(16);
const client = new S3Client({ region: "us-east-1" });
const attempts = new Map<number, number>();
let laterActive = 0, laterPeak = 0;
let firstRoundRetries = 0;
const firstRoundRetried = Promise.withResolvers<void>();
// Keep disk I/O out of the concurrency measurement. The real-client test checks MD5.
using digest = vi.spyOn(fileCrypto, "digestFileOffThread").mockResolvedValue({ hex: "0".repeat(32), bytes: partBytes });
const random = vi.spyOn(Math, "random").mockReturnValue(0);
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
const number = command.input.PartNumber!;
attempts.set(number, (attempts.get(number) ?? 0) + 1);
if (number <= FILE_PART_CONCURRENCY) {
if (attempts.get(number) === 1) throw slowDown();
// Finish the already-dispatched retries together so every eligible slot can take new work.
if (++firstRoundRetries === FILE_PART_CONCURRENCY) firstRoundRetried.resolve();
await firstRoundRetried.promise;
} else {
laterActive++; laterPeak = Math.max(laterPeak, laterActive);
await nextEventLoopTurn();
laterActive--;
}
return { ETag: `part-${number}` };
}
if (command instanceof AbortMultipartUploadCommand) throw new Error("must not abort a recovered upload");
return {};
});
try {
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
contentLength: source.size, body: Buffer.alloc(0), sourceFile: source.file })).resolves.toEqual({ integrity: "server_verified_md5" });
expect([...attempts.keys()].sort((a, b) => a - b)).toEqual([...Array(16).keys()].map(n => n + 1));
// Eight simultaneous throttles are one round: four slots remain, not one.
expect(laterPeak).toBe(FILE_PART_CONCURRENCY / 2);
} finally { random.mockRestore(); send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
});
it("stops after a bounded number of throttled attempts and aborts once", async () => {
const source = await sourceFile(3);
const client = new S3Client({ region: "us-east-1" });
const random = vi.spyOn(Math, "random").mockReturnValue(0);
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
await drain(command.input.Body);
if (command.input.PartNumber === 1) throw slowDown();
return { ETag: `part-${command.input.PartNumber}` };
}
if (command instanceof AbortMultipartUploadCommand) return {};
throw new Error("must not complete a failed upload");
});
try {
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
contentLength: source.size, body: Buffer.alloc(0), sourceFile: source.file })).rejects.toMatchObject({ name: "SlowDown" });
const partOne = send.mock.calls.filter(([command]) => command instanceof UploadPartCommand && command.input.PartNumber === 1);
expect(partOne).toHaveLength(THROTTLED_PART_ATTEMPTS);
expect(send.mock.calls.filter(([command]) => command instanceof AbortMultipartUploadCommand)).toHaveLength(1);
} finally { random.mockRestore(); send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
});
it("cancels promptly while a throttled part is backing off", async () => {
const source = await sourceFile(3);
const client = new S3Client({ region: "us-east-1" });
const cancellation = new AbortController();
const random = vi.spyOn(Math, "random").mockReturnValue(0.99);
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
if (command instanceof UploadPartCommand) {
await drain(command.input.Body);
if (command.input.PartNumber === 1) { setTimeout(() => cancellation.abort(new Error("run cancelled")), 10); throw slowDown(); }
return { ETag: `part-${command.input.PartNumber}` };
}
return {};
});
const started = Date.now();
try {
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
contentLength: source.size, body: Buffer.alloc(0), sourceFile: source.file, signal: cancellation.signal })).rejects.toThrow("run cancelled");
expect(Date.now() - started).toBeLessThan(900);
expect(send.mock.calls.filter(([command]) => command instanceof UploadPartCommand && command.input.PartNumber === 1)).toHaveLength(1);
} finally { random.mockRestore(); send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
});
});