mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane must run 1,000 agents at once on one workspace (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) > - Above about 50 agents, the evidence is a load model only. Nobody has run 200 or 1,000 real agents > - Phase 3 of the plan proves the finish line with real runs on hosted instances, in a fixed order > - This pull request adds the harness for those runs. It does not run it against any real instance > - The benefit is that each proof step is one command with a written cost estimate, explicit pass/fail criteria, and a guaranteed cleanup ## Linked Issues or Issue Description **What problem does this solve?** The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents. No tool exists that creates real agents and tasks on a hosted instance, ramps concurrency, measures the control plane, and judges the plan's finish line. **Proposed solution** A script under `scripts/` that ramps real agents through the product API, stops at the first failed criterion, writes a machine-readable result, and always cleans up what it created. **Alternatives considered** Extend the restart-chaos hosted mode. That mode grades release swaps at up to 30 concurrent tasks. The ramp reuses its target guard and its cleanup order, but it has a different workload, different measures and a different stop rule. ## What Changed - `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness. It reads the base URL from a flag or the environment and the credentials from the environment only. It creates or reuses a company, then creates agents, tasks, a project and an optional provider secret. It ramps concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps each agent on one small deterministic task at a time. - It measures per step: tasks started, finished, failed and needing Retry; failed runs; peak running agents; queue wait; time to first token and first activity (from sampled run events); run duration; board GET latency. An optional metrics endpoint supplies event-loop, statement and pool figures. A missing source is reported as "not measured", never as zero. - It evaluates the criteria from the plan's finish line plus harness criteria, writes them into `result.json` and `SUMMARY.md`, and stops at the first failure. - Cleanup runs in a `finally`: pause agents, cancel live runs, archive tasks, terminate and delete agents, delete the stored secret, archive the project and the created company, and wait for sandbox leases. `--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run` prints the plan and the estimate and makes no request. - Target guard: `--target-kind` is required and must match the host. Shared staging and production hosts need a second explicit flag. The script asks for the host name before it starts unless `--yes` is given. A run above the spend threshold needs `--confirm-spend <max USD>`. - `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an in-process fake API with a virtual clock. They cover ramp logic, stop on failure, criteria evaluation, cleanup after a mid-run crash and an interrupt, `--cleanup-only` with and without a state file, dry run, the production-host guard, the spend gate, and that credentials never appear in output or files. - `scripts/scale-ramp/tsconfig.json` is added to the test programs that CI typechecks. The test file joins the `test:release-registry` node test list, which CI runs. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3 runbook with commands, cost, criteria, proof order, target prerequisites and open risks. ## Verification - `node scripts/ci-profile/node-test.mjs scripts/scale-ramp/scale-ramp.test.ts scripts/typecheck-test-programs.test.mjs`: 47 tests pass. - `node node_modules/typescript/bin/tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts --base-url https://scale1.test.paperclip.app --target-kind test-stack --steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate and the criteria, and makes no request. - The harness was not run against any real instance. That needs the owner's go-ahead. ## Risks - Low risk for the product: the change adds a script, tests and docs only. No server code changes. - A real run creates real agents, tasks and sandboxes and spends money. The guard, the typed confirmation, the spend gate, the task cap and the recorded-spend stop limit this. The runbook lists the open risks. - The cost defaults are assumptions until a first real run records actual spend. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
165 lines
5.8 KiB
TypeScript
165 lines
5.8 KiB
TypeScript
import { assertSameOrigin, type Target } from "./guard.ts";
|
|
|
|
export type Fetcher = (url: string, init: RequestInit) => Promise<Response>;
|
|
|
|
export interface Clock {
|
|
now(): number;
|
|
sleep(ms: number): Promise<void>;
|
|
}
|
|
|
|
export const realClock: Clock = {
|
|
now: () => Date.now(),
|
|
sleep: ms => new Promise(resolve => setTimeout(resolve, ms)),
|
|
};
|
|
|
|
export interface RequestRecord {
|
|
at: number;
|
|
method: string;
|
|
path: string;
|
|
status: number | null;
|
|
ms: number;
|
|
ok: boolean;
|
|
kind: "board" | "harness";
|
|
}
|
|
|
|
export class ApiError extends Error {
|
|
readonly status: number | null;
|
|
readonly code: string | null;
|
|
|
|
constructor(status: number | null, message: string, code: string | null = null) {
|
|
super(message);
|
|
this.name = "ApiError";
|
|
this.status = status;
|
|
this.code = code;
|
|
}
|
|
}
|
|
|
|
export class Redactor {
|
|
private readonly values: string[];
|
|
|
|
constructor(values: ReadonlyArray<string | null | undefined>) {
|
|
this.values = values.filter((value): value is string => typeof value === "string" && value.length >= 4)
|
|
.sort((a, b) => b.length - a.length);
|
|
}
|
|
|
|
text(input: string): string {
|
|
let out = input;
|
|
for (const value of this.values) out = out.split(value).join("[redacted]");
|
|
return out.replace(/(bearer\s+)[A-Za-z0-9._~+/=-]{8,}/gi, "$1[redacted]");
|
|
}
|
|
|
|
json<T>(value: T): T {
|
|
return JSON.parse(this.text(JSON.stringify(value))) as T;
|
|
}
|
|
}
|
|
|
|
const UUID = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi;
|
|
const TRANSIENT = new Set([502, 503, 504]);
|
|
|
|
export function pathClass(route: string): string {
|
|
return route.split("?")[0]!.replace(UUID, ":id");
|
|
}
|
|
|
|
export class PaperclipClient {
|
|
readonly records: RequestRecord[] = [];
|
|
private readonly target: Target;
|
|
private readonly token: string;
|
|
private readonly fetcher: Fetcher;
|
|
private readonly clock: Clock;
|
|
private readonly redactor: Redactor;
|
|
private readonly timeoutMs: number;
|
|
|
|
constructor(target: Target, token: string, fetcher: Fetcher, clock: Clock, redactor: Redactor, timeoutMs = 30_000) {
|
|
this.target = target;
|
|
this.token = token;
|
|
this.fetcher = fetcher;
|
|
this.clock = clock;
|
|
this.redactor = redactor;
|
|
this.timeoutMs = timeoutMs;
|
|
}
|
|
|
|
private async once(method: string, route: string, body: unknown, kind: RequestRecord["kind"]) {
|
|
const url = `${this.target.baseUrl}${route}`;
|
|
assertSameOrigin(this.target, url);
|
|
const started = this.clock.now();
|
|
try {
|
|
const response = await this.fetcher(url, {
|
|
method,
|
|
cache: "no-store",
|
|
signal: AbortSignal.timeout(this.timeoutMs),
|
|
headers: {
|
|
authorization: `Bearer ${this.token}`,
|
|
accept: "application/json",
|
|
origin: this.target.baseUrl,
|
|
...(body === undefined ? {} : { "content-type": "application/json" }),
|
|
},
|
|
body: body === undefined ? undefined : JSON.stringify(body),
|
|
});
|
|
const text = await response.text();
|
|
const ms = this.clock.now() - started;
|
|
const answered = response.ok || (kind === "harness" && [400, 404, 409, 422].includes(response.status));
|
|
this.records.push({ at: started, method, path: pathClass(route), status: response.status, ms, ok: answered, kind });
|
|
return { status: response.status, ok: response.ok, text };
|
|
} catch (error) {
|
|
this.records.push({ at: started, method, path: pathClass(route), status: null, ms: this.clock.now() - started, ok: false, kind });
|
|
return { status: null, ok: false, text: error instanceof Error ? error.message : String(error) };
|
|
}
|
|
}
|
|
|
|
async request<T>(method: string, route: string, body?: unknown, kind: RequestRecord["kind"] = "harness"): Promise<T> {
|
|
for (let attempt = 1; ; attempt += 1) {
|
|
const result = await this.once(method, route, body, kind);
|
|
if (result.ok) return (result.text ? JSON.parse(result.text) : null) as T;
|
|
const transient = result.status === null || TRANSIENT.has(result.status) || result.status === 429;
|
|
if (!transient || (method !== "GET" && result.status !== 429) || attempt >= 3) {
|
|
let code: string | null = null;
|
|
try {
|
|
const parsed = JSON.parse(result.text) as { code?: unknown; details?: { code?: unknown } };
|
|
code = typeof parsed.code === "string" ? parsed.code : typeof parsed.details?.code === "string" ? parsed.details.code : null;
|
|
} catch {
|
|
code = null;
|
|
}
|
|
const detail = this.redactor.text(result.text.slice(0, 300));
|
|
throw new ApiError(result.status, `${method} ${pathClass(route)} ${result.status === null ? "got no answer" : `returned ${result.status}`}: ${detail}`, code);
|
|
}
|
|
await this.clock.sleep(1_000 * attempt);
|
|
}
|
|
}
|
|
|
|
get<T>(route: string, kind: RequestRecord["kind"] = "harness") {
|
|
return this.request<T>("GET", route, undefined, kind);
|
|
}
|
|
|
|
post<T>(route: string, body: unknown = {}) {
|
|
return this.request<T>("POST", route, body);
|
|
}
|
|
|
|
patch<T>(route: string, body: unknown = {}) {
|
|
return this.request<T>("PATCH", route, body);
|
|
}
|
|
|
|
async delete(route: string, allowNotFound = true): Promise<void> {
|
|
try {
|
|
await this.request<unknown>("DELETE", route);
|
|
} catch (error) {
|
|
if (!(allowNotFound && error instanceof ApiError && error.status === 404)) throw error;
|
|
}
|
|
}
|
|
}
|
|
|
|
export async function readMetrics(fetcher: Fetcher, url: string, token: string | null, clock: Clock): Promise<Record<string, unknown> | null> {
|
|
try {
|
|
const response = await fetcher(url, {
|
|
method: "GET",
|
|
cache: "no-store",
|
|
signal: AbortSignal.timeout(10_000),
|
|
headers: { accept: "application/json", ...(token ? { authorization: `Bearer ${token}` } : {}) },
|
|
});
|
|
if (!response.ok) return null;
|
|
const body = await response.json() as unknown;
|
|
return body && typeof body === "object" && !Array.isArray(body) ? { ...(body as Record<string, unknown>), sampledAt: clock.now() } : null;
|
|
} catch {
|
|
return null;
|
|
}
|
|
}
|