mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane must run 1,000 agents at once on one workspace (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) > - Above about 50 agents, the evidence is a load model only. Nobody has run 200 or 1,000 real agents > - Phase 3 of the plan proves the finish line with real runs on hosted instances, in a fixed order > - This pull request adds the harness for those runs. It does not run it against any real instance > - The benefit is that each proof step is one command with a written cost estimate, explicit pass/fail criteria, and a guaranteed cleanup ## Linked Issues or Issue Description **What problem does this solve?** The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents. No tool exists that creates real agents and tasks on a hosted instance, ramps concurrency, measures the control plane, and judges the plan's finish line. **Proposed solution** A script under `scripts/` that ramps real agents through the product API, stops at the first failed criterion, writes a machine-readable result, and always cleans up what it created. **Alternatives considered** Extend the restart-chaos hosted mode. That mode grades release swaps at up to 30 concurrent tasks. The ramp reuses its target guard and its cleanup order, but it has a different workload, different measures and a different stop rule. ## What Changed - `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness. It reads the base URL from a flag or the environment and the credentials from the environment only. It creates or reuses a company, then creates agents, tasks, a project and an optional provider secret. It ramps concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps each agent on one small deterministic task at a time. - It measures per step: tasks started, finished, failed and needing Retry; failed runs; peak running agents; queue wait; time to first token and first activity (from sampled run events); run duration; board GET latency. An optional metrics endpoint supplies event-loop, statement and pool figures. A missing source is reported as "not measured", never as zero. - It evaluates the criteria from the plan's finish line plus harness criteria, writes them into `result.json` and `SUMMARY.md`, and stops at the first failure. - Cleanup runs in a `finally`: pause agents, cancel live runs, archive tasks, terminate and delete agents, delete the stored secret, archive the project and the created company, and wait for sandbox leases. `--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run` prints the plan and the estimate and makes no request. - Target guard: `--target-kind` is required and must match the host. Shared staging and production hosts need a second explicit flag. The script asks for the host name before it starts unless `--yes` is given. A run above the spend threshold needs `--confirm-spend <max USD>`. - `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an in-process fake API with a virtual clock. They cover ramp logic, stop on failure, criteria evaluation, cleanup after a mid-run crash and an interrupt, `--cleanup-only` with and without a state file, dry run, the production-host guard, the spend gate, and that credentials never appear in output or files. - `scripts/scale-ramp/tsconfig.json` is added to the test programs that CI typechecks. The test file joins the `test:release-registry` node test list, which CI runs. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3 runbook with commands, cost, criteria, proof order, target prerequisites and open risks. ## Verification - `node scripts/ci-profile/node-test.mjs scripts/scale-ramp/scale-ramp.test.ts scripts/typecheck-test-programs.test.mjs`: 47 tests pass. - `node node_modules/typescript/bin/tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts --base-url https://scale1.test.paperclip.app --target-kind test-stack --steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate and the criteria, and makes no request. - The harness was not run against any real instance. That needs the owner's go-ahead. ## Risks - Low risk for the product: the change adds a script, tests and docs only. No server code changes. - A real run creates real agents, tasks and sandboxes and spends money. The guard, the typed confirmation, the spend gate, the task cap and the recorded-spend stop limit this. The runbook lists the open risks. - The cost defaults are assumptions until a first real run records actual spend. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
181 lines
8.7 KiB
TypeScript
181 lines
8.7 KiB
TypeScript
import { mkdir, writeFile } from "node:fs/promises";
|
|
import path from "node:path";
|
|
import { createInterface } from "node:readline/promises";
|
|
import { cleanupRun, discoverState, type CleanupReport } from "./cleanup.ts";
|
|
import { PaperclipClient, Redactor, realClock, type Clock, type Fetcher } from "./client.ts";
|
|
import { FALLBACK_TOKEN_ENV, parseOptions, readSecrets, TOKEN_ENV, type Options } from "./config.ts";
|
|
import { criteriaTable } from "./criteria.ts";
|
|
import { checkSpend, estimateSpend, type SpendEstimate } from "./estimate.ts";
|
|
import { Ramp, type RampOutcome } from "./ramp.ts";
|
|
import { buildResult, summaryLines, verdictOf } from "./report.ts";
|
|
import { prepareCompany } from "./setup.ts";
|
|
import { emptyState, fileStateStore, type RunState } from "./state.ts";
|
|
|
|
export interface MainDeps {
|
|
env: NodeJS.ProcessEnv;
|
|
fetcher: Fetcher;
|
|
clock: Clock;
|
|
out: (line: string) => void;
|
|
err: (line: string) => void;
|
|
confirm: (question: string) => Promise<string>;
|
|
onInterrupt: (handler: () => void) => () => void;
|
|
random?: () => number;
|
|
}
|
|
|
|
export const EXIT = { pass: 0, fail: 1, usage: 2, incomplete: 3, error: 4 } as const;
|
|
|
|
export function planLines(options: Options, estimate: SpendEstimate): string[] {
|
|
const steps = options.steps.map(step => `${step.concurrency}`).join(" -> ");
|
|
const credential = options.profile.credentialKey === null ? "none"
|
|
: options.profile.providerSecretId ? `existing secret for ${options.profile.credentialKey}`
|
|
: `${options.profile.credentialKey} from the environment, stored as a company secret and deleted at cleanup`;
|
|
return [
|
|
`Target: ${options.target.baseUrl} (${options.target.kind})`,
|
|
`Run id: ${options.label}`,
|
|
`Company: ${options.companyId ?? "a new company named for the run id"}`,
|
|
`Ramp: ${steps} agents, ${options.steps[0]!.holdMs / 60_000} min per step, drain ${options.drainMs / 60_000} min, stop on the first failed criterion`,
|
|
`Agents: up to ${estimate.agents}, ${options.profile.provider} ${options.profile.model}; credential: ${credential}`,
|
|
`Tasks: about ${estimate.tasks}, hard cap ${estimate.maxTasks}`,
|
|
`Estimated spend: $${estimate.totalUsd} (model $${estimate.modelUsd}, sandboxes $${estimate.sandboxUsd} for ${estimate.sandboxHours} sandbox-hours), about ${estimate.wallMinutes} min`,
|
|
...estimate.assumptions.map(line => ` assumption: ${line}`),
|
|
"Criteria:",
|
|
...criteriaTable(options.thresholds).map(criterion => ` ${criterion.id} [${criterion.source}] ${criterion.description}: ${criterion.threshold}`),
|
|
];
|
|
}
|
|
|
|
async function writeOutputs(dir: string, name: string, body: unknown, summary: string[] | null, redactor: Redactor): Promise<void> {
|
|
await mkdir(dir, { recursive: true, mode: 0o700 });
|
|
await writeFile(path.join(dir, name), `${redactor.text(JSON.stringify(body, null, 2))}\n`, { mode: 0o600 });
|
|
if (summary) await writeFile(path.join(dir, "SUMMARY.md"), `${redactor.text(summary.join("\n"))}\n`, { mode: 0o600 });
|
|
}
|
|
|
|
export async function main(argv: readonly string[], deps: MainDeps): Promise<number> {
|
|
let options: Options;
|
|
try {
|
|
options = parseOptions(argv, deps.env, deps.clock.now(), deps.random);
|
|
} catch (error) {
|
|
deps.err(`scale ramp: ${error instanceof Error ? error.message : String(error)}`);
|
|
return EXIT.usage;
|
|
}
|
|
const secrets = readSecrets(deps.env, options.profile.providerSecretId ? null : options.profile.credentialKey);
|
|
const redactor = new Redactor([secrets.token, secrets.credential, secrets.metricsToken]);
|
|
const out = (line: string) => deps.out(redactor.text(line));
|
|
const err = (line: string) => deps.err(redactor.text(line));
|
|
const estimate = estimateSpend(options);
|
|
|
|
if (options.mode !== "cleanup") for (const line of planLines(options, estimate)) out(line);
|
|
if (options.mode === "dry-run") {
|
|
out("Dry run: nothing was created or contacted.");
|
|
return EXIT.pass;
|
|
}
|
|
if (options.mode === "run") {
|
|
const spend = checkSpend(estimate, options.spendThresholdUsd, options.confirmSpendUsd);
|
|
if (!spend.ok) {
|
|
err(spend.reason);
|
|
return EXIT.usage;
|
|
}
|
|
}
|
|
if (!secrets.token) {
|
|
err(`Set ${TOKEN_ENV} (or ${FALLBACK_TOKEN_ENV}) to a board token for ${options.target.host}`);
|
|
return EXIT.usage;
|
|
}
|
|
if (!options.yes) {
|
|
const action = options.mode === "cleanup" ? `clean up ${options.label} on` : "start creating agents and tasks on";
|
|
const answer = await deps.confirm(`Type ${options.target.host} to ${action} it: `);
|
|
if (answer.trim().toLowerCase() !== options.target.host) {
|
|
err("Confirmation did not match the target host; nothing was done.");
|
|
return EXIT.usage;
|
|
}
|
|
}
|
|
|
|
const api = new PaperclipClient(options.target, secrets.token, deps.fetcher, deps.clock, redactor);
|
|
const store = fileStateStore(options.outDir);
|
|
|
|
if (options.mode === "cleanup") {
|
|
let report: CleanupReport | null = null;
|
|
let error: string | null = null;
|
|
try {
|
|
const state = await store.load() ?? await discoverState(api, options.label, options.companyId, options.target.baseUrl, deps.clock.now());
|
|
report = await cleanupRun({ api, clock: deps.clock, label: options.label, state, log: out });
|
|
} catch (caught) {
|
|
error = caught instanceof Error ? caught.message : String(caught);
|
|
}
|
|
await writeOutputs(options.outDir, `cleanup-${deps.clock.now()}.json`, { runId: options.label, report, error }, null, redactor);
|
|
if (error) err(`Cleanup failed: ${error}`);
|
|
if (report) out(`Cleanup ${options.label}: ${report.runsCancelled} runs cancelled, ${report.tasksArchived} tasks archived, ${report.agentsDeleted} agents deleted, ${report.errors.length} errors`);
|
|
return error || (report?.errors.length ?? 0) > 0 ? EXIT.fail : EXIT.pass;
|
|
}
|
|
|
|
let interrupted = false;
|
|
const release = deps.onInterrupt(() => {
|
|
interrupted = true;
|
|
err("Interrupt received: stopping the ramp and cleaning up");
|
|
});
|
|
const startedAt = deps.clock.now();
|
|
const state: RunState = emptyState(options.label, options.target.baseUrl, startedAt);
|
|
let outcome: RampOutcome | null = null;
|
|
let cleanup: CleanupReport | null = null;
|
|
let error: string | null = null;
|
|
try {
|
|
await store.save(state);
|
|
await prepareCompany({ api, options, state, store, credential: secrets.credential, log: out });
|
|
const ramp = new Ramp({
|
|
api, clock: deps.clock, fetcher: deps.fetcher, options, state, store, maxTasks: estimate.maxTasks,
|
|
spendCapUsd: options.confirmSpendUsd ?? Math.max(options.spendThresholdUsd, estimate.totalUsd), metricsToken: secrets.metricsToken,
|
|
log: out, shouldStop: () => interrupted,
|
|
});
|
|
outcome = await ramp.run();
|
|
} catch (caught) {
|
|
error = caught instanceof Error ? caught.message : String(caught);
|
|
err(`Ramp stopped by an error: ${error}`);
|
|
} finally {
|
|
out(`Cleaning up ${options.label}`);
|
|
try {
|
|
cleanup = await cleanupRun({ api, clock: deps.clock, label: options.label, state, log: out });
|
|
} catch (caught) {
|
|
err(`Cleanup failed; finish it with --cleanup-only ${options.label}: ${caught instanceof Error ? caught.message : String(caught)}`);
|
|
}
|
|
release();
|
|
}
|
|
const failedRequests = api.records.filter(record => !record.ok).length;
|
|
const result = buildResult({
|
|
options, estimate, startedAt, finishedAt: deps.clock.now(), outcome, cleanup, error,
|
|
requests: { total: api.records.length, failed: failedRequests },
|
|
credentialSource: options.profile.credentialKey === null ? "none" : options.profile.providerSecretId ? "existing secret" : "stored for the run and deleted at cleanup",
|
|
});
|
|
const summary = summaryLines(result);
|
|
await writeOutputs(options.outDir, "result.json", result, summary, redactor);
|
|
for (const line of summary) out(line);
|
|
out(`Result: ${path.join(options.outDir, "result.json")}`);
|
|
if (cleanup && cleanup.errors.length) err(`Cleanup left ${cleanup.errors.length} errors; finish it with --cleanup-only ${options.label}`);
|
|
const verdict = verdictOf(outcome, error);
|
|
return verdict === "pass" ? EXIT.pass : verdict === "fail" ? EXIT.fail : verdict === "incomplete" ? EXIT.incomplete : EXIT.error;
|
|
}
|
|
|
|
export function processDeps(): MainDeps {
|
|
return {
|
|
env: process.env,
|
|
fetcher: (url, init) => fetch(url, init),
|
|
clock: realClock,
|
|
out: line => process.stdout.write(`${line}\n`),
|
|
err: line => process.stderr.write(`${line}\n`),
|
|
confirm: async question => {
|
|
const rl = createInterface({ input: process.stdin, output: process.stderr });
|
|
try {
|
|
return await rl.question(question);
|
|
} finally {
|
|
rl.close();
|
|
}
|
|
},
|
|
onInterrupt: handler => {
|
|
const listener = () => handler();
|
|
process.once("SIGINT", listener);
|
|
process.once("SIGTERM", listener);
|
|
return () => {
|
|
process.off("SIGINT", listener);
|
|
process.off("SIGTERM", listener);
|
|
};
|
|
},
|
|
};
|
|
}
|