mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path > - Paperclip delegates remote Runner execution to environment plugins. > - A plugin can move a session to another task allocation. > - The Core controller can lose an ownership reply or restart before the Runner starts. > - The source and target need a durable record of the same request. > - This change binds restart and attachment to that request and the original unlaunched task lease. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of plugin Runner sessions on replacement compute. **Current behavior** Core cannot replay a session ownership request from saved lease state during startup. **Proposed behavior** Record the operation on both leases and check the matched ownership receipt before attaching. Preserve the original pre-admission lease across controller restart. Managed environment configuration is in [#21065](https://github.com/paperclipai/paperclip/pull/21065). Related warm sandbox cleanup is in [#20065](https://github.com/paperclipai/paperclip/pull/20065). This layer preserves the same-sandbox fallback from [#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace placement follows in [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Add the optional `sessionHandoffVersion` capability and the ownership task operation. - Save matching ownership claims and fence source checkpoint access. Replay a pending request on the original target. - Restore a portable session using the durable claim and the current credentials for the same account. Start a fresh conversation when the provider reports an unavailable checkpoint, or when Core's persisted compatibility decision selects a different session after transfer. The latter requires an unlaunched target; subsequent recovery must match the fresh task's saved identity. Both paths retain the workspace transfer and source fencing. - Reuse the original unlaunched task lease after hard or graceful controller restart. After a move has attached, validate the surviving Runner through the saved claim rather than requiring its logical lease ID to equal the new physical lease. - Keep older retained leases without a saved account identity on the existing resume path. - Apply the existing supported MCP-only compatibility rule to warm recovery and predecessor accounting. Persist the capability discovered at admission and reuse it after restart without repeating discovery. - Keep predecessor-retirement fallback scoped to the same physical sandbox. A refusal after a portable move preserves the session instead of requesting a replacement that the plugin driver cannot enforce. ## Verification - On `1eb527be27`, server typecheck (including Runner build and SDK dependencies) passed. - Independent final verification passed 168 tests across portable source recovery, session resume, controller recovery, and task authority; two existing tests were skipped. The full portable source integration suite passed 64 tests. - The new integration follows confirmed ownership through an incompatible instruction change, persisted fresh admission, target-only prepare/auth/submit without the source checkpoint, and reattachment after controller replacement. It also rejects unpersisted session selection, ambiguous target leases, and incorrect saved identities. - Before this final change, independent verification passed 1,667 tests across executor, session resume, recovery usage, remote warm recovery, runtime selection, and runtime execution; the environment lifecycle suite passed 208 tests. - Existing regressions cover previous-release account metadata, supported and unsupported MCP changes, provider/instruction incompatibility, and admission capability reuse without another discovery call. - On `51d7eec373`, all four S3 multipart throttle tests passed independently. CI exposed an existing timing-sensitive test twice; its 20 ms overlap assumption now uses synchronized retries and an event-loop yield. Production upload code is unchanged. Temporary concurrency caps of one and eight both fail the retained exact-four assertion. - Exact SDK source-pin checks passed for both stack heads. The upper #20743 code patch remains unchanged; its documentation preserves both layers' behavior. ## Risks - The handoff protocol is opt-in. Existing plugin leases without portable ownership metadata keep the previous release's resume path. Live attachment still requires end-to-end qualification. An unverified claim blocks startup. Git workspaces prepared on a portable allocation also need the workspace placement layer. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
174 lines
9.9 KiB
TypeScript
174 lines
9.9 KiB
TypeScript
import { AbortMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, S3Client } from "@aws-sdk/client-s3";
|
|
import { Readable } from "node:stream";
|
|
import { setImmediate as nextEventLoopTurn } from "node:timers/promises";
|
|
import { createHash } from "node:crypto";
|
|
import { createServer } from "node:http";
|
|
import type { AddressInfo } from "node:net";
|
|
import fs from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { describe, expect, it, vi } from "vitest";
|
|
import { FILE_PART_CONCURRENCY, THROTTLED_PART_ATTEMPTS, putS3MultipartFile } from "../storage/s3-multipart.js";
|
|
import { createS3StorageProvider } from "../storage/s3-provider.js";
|
|
import { createStorageService } from "../storage/service.js";
|
|
import * as fileCrypto from "../storage/file-crypto-pool.js";
|
|
|
|
const partBytes = 8 * 1024 * 1024;
|
|
|
|
async function sourceFile(parts: number) {
|
|
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "pc-s3-throttle-"));
|
|
const file = path.join(dir, "source.bin");
|
|
const handle = await fs.open(file, "wx", 0o600);
|
|
try {
|
|
for (let index = 0; index < parts; index++) await handle.write(Buffer.alloc(partBytes, index + 1));
|
|
} finally { await handle.close(); }
|
|
return { dir, file, size: parts * partBytes };
|
|
}
|
|
|
|
const md5 = (bytes: Buffer) => createHash("md5").update(bytes).digest("base64");
|
|
|
|
async function drain(body: unknown) {
|
|
const chunks: Buffer[] = [];
|
|
for await (const chunk of body as Readable) chunks.push(Buffer.from(chunk));
|
|
return Buffer.concat(chunks);
|
|
}
|
|
|
|
const slowDown = () => Object.assign(new Error("Please reduce your request rate."), { name: "SlowDown", $metadata: { httpStatusCode: 503 } });
|
|
|
|
describe("file-backed S3 multipart uploads under a request-rate limit", () => {
|
|
it("backs off and completes when the real S3 client receives SlowDown for one part", async () => {
|
|
const source = await sourceFile(3);
|
|
const attempts = new Map<number, number>();
|
|
let completed = "";
|
|
const server = createServer((request, response) => {
|
|
const url = new URL(request.url!, "http://localhost");
|
|
const chunks: Buffer[] = [];
|
|
request.on("data", chunk => chunks.push(chunk));
|
|
request.on("end", () => {
|
|
if (url.searchParams.has("uploads")) {
|
|
response.end("<InitiateMultipartUploadResult><UploadId>upload</UploadId></InitiateMultipartUploadResult>");
|
|
} else if (request.method === "PUT") {
|
|
const number = Number(url.searchParams.get("partNumber"));
|
|
attempts.set(number, (attempts.get(number) ?? 0) + 1);
|
|
if (number === 2 && attempts.get(number)! <= 3) {
|
|
response.writeHead(503, { "Content-Type": "application/xml" })
|
|
.end("<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><RequestId>throttled</RequestId></Error>");
|
|
} else {
|
|
expect(md5(Buffer.concat(chunks))).toBe(request.headers["content-md5"]);
|
|
response.writeHead(200, { ETag: `"part-${number}"` }).end();
|
|
}
|
|
} else if (request.method === "POST") {
|
|
completed = Buffer.concat(chunks).toString();
|
|
response.end("<CompleteMultipartUploadResult><ETag>complete</ETag></CompleteMultipartUploadResult>");
|
|
} else {
|
|
response.writeHead(500).end();
|
|
}
|
|
});
|
|
});
|
|
await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); });
|
|
vi.stubEnv("AWS_ACCESS_KEY_ID", "fixture"); vi.stubEnv("AWS_SECRET_ACCESS_KEY", "fixture");
|
|
const random = vi.spyOn(Math, "random").mockReturnValue(0);
|
|
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
const service = createStorageService(createS3StorageProvider({ bucket: "bucket", region: "us-east-1", forcePathStyle: true,
|
|
endpoint: `http://127.0.0.1:${(server.address() as AddressInfo).port}` }));
|
|
try {
|
|
const result = await service.putFile({ companyId: "company", objectKey: "company/workspace-snapshots/throttled.pcws", namespace: "workspace-snapshots",
|
|
originalFilename: null, contentType: "application/octet-stream", body: Readable.from([Buffer.alloc(0)]), byteSize: source.size,
|
|
sha256: createHash("sha256").update(await fs.readFile(source.file)).digest("hex"), sourceFile: source.file });
|
|
expect(result.integrity).toBe("server_verified_md5");
|
|
expect(attempts.get(2)).toBe(4);
|
|
expect([...completed.matchAll(/<PartNumber>(\d+)<\/PartNumber>/g)].map(match => Number(match[1]))).toEqual([1, 2, 3]);
|
|
} finally {
|
|
random.mockRestore(); warn.mockRestore(); vi.unstubAllEnvs();
|
|
server.closeAllConnections();
|
|
await new Promise<void>(resolve => server.close(() => resolve()));
|
|
await fs.rm(source.dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it("halves new parts in flight once when a parallel burst of parts is throttled", async () => {
|
|
const source = await sourceFile(16);
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
const attempts = new Map<number, number>();
|
|
let laterActive = 0, laterPeak = 0;
|
|
let firstRoundRetries = 0;
|
|
const firstRoundRetried = Promise.withResolvers<void>();
|
|
// Keep disk I/O out of the concurrency measurement. The real-client test checks MD5.
|
|
using digest = vi.spyOn(fileCrypto, "digestFileOffThread").mockResolvedValue({ hex: "0".repeat(32), bytes: partBytes });
|
|
const random = vi.spyOn(Math, "random").mockReturnValue(0);
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
const number = command.input.PartNumber!;
|
|
attempts.set(number, (attempts.get(number) ?? 0) + 1);
|
|
if (number <= FILE_PART_CONCURRENCY) {
|
|
if (attempts.get(number) === 1) throw slowDown();
|
|
// Finish the already-dispatched retries together so every eligible slot can take new work.
|
|
if (++firstRoundRetries === FILE_PART_CONCURRENCY) firstRoundRetried.resolve();
|
|
await firstRoundRetried.promise;
|
|
} else {
|
|
laterActive++; laterPeak = Math.max(laterPeak, laterActive);
|
|
await nextEventLoopTurn();
|
|
laterActive--;
|
|
}
|
|
return { ETag: `part-${number}` };
|
|
}
|
|
if (command instanceof AbortMultipartUploadCommand) throw new Error("must not abort a recovered upload");
|
|
return {};
|
|
});
|
|
try {
|
|
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
|
|
contentLength: source.size, body: Buffer.alloc(0), sourceFile: source.file })).resolves.toEqual({ integrity: "server_verified_md5" });
|
|
expect([...attempts.keys()].sort((a, b) => a - b)).toEqual([...Array(16).keys()].map(n => n + 1));
|
|
// Eight simultaneous throttles are one round: four slots remain, not one.
|
|
expect(laterPeak).toBe(FILE_PART_CONCURRENCY / 2);
|
|
} finally { random.mockRestore(); send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
|
|
});
|
|
|
|
it("stops after a bounded number of throttled attempts and aborts once", async () => {
|
|
const source = await sourceFile(3);
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
const random = vi.spyOn(Math, "random").mockReturnValue(0);
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
await drain(command.input.Body);
|
|
if (command.input.PartNumber === 1) throw slowDown();
|
|
return { ETag: `part-${command.input.PartNumber}` };
|
|
}
|
|
if (command instanceof AbortMultipartUploadCommand) return {};
|
|
throw new Error("must not complete a failed upload");
|
|
});
|
|
try {
|
|
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
|
|
contentLength: source.size, body: Buffer.alloc(0), sourceFile: source.file })).rejects.toMatchObject({ name: "SlowDown" });
|
|
const partOne = send.mock.calls.filter(([command]) => command instanceof UploadPartCommand && command.input.PartNumber === 1);
|
|
expect(partOne).toHaveLength(THROTTLED_PART_ATTEMPTS);
|
|
expect(send.mock.calls.filter(([command]) => command instanceof AbortMultipartUploadCommand)).toHaveLength(1);
|
|
} finally { random.mockRestore(); send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
|
|
});
|
|
|
|
it("cancels promptly while a throttled part is backing off", async () => {
|
|
const source = await sourceFile(3);
|
|
const client = new S3Client({ region: "us-east-1" });
|
|
const cancellation = new AbortController();
|
|
const random = vi.spyOn(Math, "random").mockReturnValue(0.99);
|
|
const send = vi.spyOn(client, "send").mockImplementation(async (command: any) => {
|
|
if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" };
|
|
if (command instanceof UploadPartCommand) {
|
|
await drain(command.input.Body);
|
|
if (command.input.PartNumber === 1) { setTimeout(() => cancellation.abort(new Error("run cancelled")), 10); throw slowDown(); }
|
|
return { ETag: `part-${command.input.PartNumber}` };
|
|
}
|
|
return {};
|
|
});
|
|
const started = Date.now();
|
|
try {
|
|
await expect(putS3MultipartFile(client, "bucket", "key", { objectKey: "key", contentType: "application/octet-stream",
|
|
contentLength: source.size, body: Buffer.alloc(0), sourceFile: source.file, signal: cancellation.signal })).rejects.toThrow("run cancelled");
|
|
expect(Date.now() - started).toBeLessThan(900);
|
|
expect(send.mock.calls.filter(([command]) => command instanceof UploadPartCommand && command.input.PartNumber === 1)).toHaveLength(1);
|
|
} finally { random.mockRestore(); send.mockRestore(); client.destroy(); await fs.rm(source.dir, { recursive: true, force: true }); }
|
|
});
|
|
});
|