mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane must run 1,000 agents at once on one workspace (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) > - Above about 50 agents, the evidence is a load model only. Nobody has run 200 or 1,000 real agents > - Phase 3 of the plan proves the finish line with real runs on hosted instances, in a fixed order > - This pull request adds the harness for those runs. It does not run it against any real instance > - The benefit is that each proof step is one command with a written cost estimate, explicit pass/fail criteria, and a guaranteed cleanup ## Linked Issues or Issue Description **What problem does this solve?** The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents. No tool exists that creates real agents and tasks on a hosted instance, ramps concurrency, measures the control plane, and judges the plan's finish line. **Proposed solution** A script under `scripts/` that ramps real agents through the product API, stops at the first failed criterion, writes a machine-readable result, and always cleans up what it created. **Alternatives considered** Extend the restart-chaos hosted mode. That mode grades release swaps at up to 30 concurrent tasks. The ramp reuses its target guard and its cleanup order, but it has a different workload, different measures and a different stop rule. ## What Changed - `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness. It reads the base URL from a flag or the environment and the credentials from the environment only. It creates or reuses a company, then creates agents, tasks, a project and an optional provider secret. It ramps concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps each agent on one small deterministic task at a time. - It measures per step: tasks started, finished, failed and needing Retry; failed runs; peak running agents; queue wait; time to first token and first activity (from sampled run events); run duration; board GET latency. An optional metrics endpoint supplies event-loop, statement and pool figures. A missing source is reported as "not measured", never as zero. - It evaluates the criteria from the plan's finish line plus harness criteria, writes them into `result.json` and `SUMMARY.md`, and stops at the first failure. - Cleanup runs in a `finally`: pause agents, cancel live runs, archive tasks, terminate and delete agents, delete the stored secret, archive the project and the created company, and wait for sandbox leases. `--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run` prints the plan and the estimate and makes no request. - Target guard: `--target-kind` is required and must match the host. Shared staging and production hosts need a second explicit flag. The script asks for the host name before it starts unless `--yes` is given. A run above the spend threshold needs `--confirm-spend <max USD>`. - `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an in-process fake API with a virtual clock. They cover ramp logic, stop on failure, criteria evaluation, cleanup after a mid-run crash and an interrupt, `--cleanup-only` with and without a state file, dry run, the production-host guard, the spend gate, and that credentials never appear in output or files. - `scripts/scale-ramp/tsconfig.json` is added to the test programs that CI typechecks. The test file joins the `test:release-registry` node test list, which CI runs. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3 runbook with commands, cost, criteria, proof order, target prerequisites and open risks. ## Verification - `node scripts/ci-profile/node-test.mjs scripts/scale-ramp/scale-ramp.test.ts scripts/typecheck-test-programs.test.mjs`: 47 tests pass. - `node node_modules/typescript/bin/tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts --base-url https://scale1.test.paperclip.app --target-kind test-stack --steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate and the criteria, and makes no request. - The harness was not run against any real instance. That needs the owner's go-ahead. ## Risks - Low risk for the product: the change adds a script, tests and docs only. No server code changes. - A real run creates real agents, tasks and sandboxes and spends money. The guard, the typed confirmation, the spend gate, the task cap and the recorded-spend stop limit this. The runbook lists the open risks. - The cost defaults are assumptions until a first real run records actual spend. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
161 lines
6.7 KiB
TypeScript
161 lines
6.7 KiB
TypeScript
import type { Step, Thresholds } from "./config.ts";
|
|
import type { Distribution } from "./stats.ts";
|
|
|
|
export interface ServerMetrics {
|
|
mainThreadCpuMax: number | null;
|
|
eventLoopUtilizationMax: number | null;
|
|
eventLoopDelayP99MsMax: number | null;
|
|
statementsPerSecondMax: number | null;
|
|
poolWaitP99MsMax: number | null;
|
|
runEventLatenessP99MsMax: number | null;
|
|
agentApiPreHandlerP99MsMax: number | null;
|
|
leaseRenewalsMissed: number | null;
|
|
samples: number;
|
|
}
|
|
|
|
export interface StepMetrics {
|
|
step: Step;
|
|
startedAt: number;
|
|
endedAt: number | null;
|
|
tasksCreated: number;
|
|
tasksStarted: number;
|
|
tasksFinished: number;
|
|
tasksFailed: number;
|
|
tasksNeedingRetry: number;
|
|
runsFailed: number;
|
|
runsInterrupted: number;
|
|
runsCancelled: number;
|
|
peakRunning: number;
|
|
meanRunning: number | null;
|
|
queueWaitMs: Distribution;
|
|
taskToStartMs: Distribution;
|
|
ttftMs: Distribution;
|
|
firstActivityMs: Distribution;
|
|
runDurationMs: Distribution;
|
|
board: { requests: number; failures: number; ms: Distribution };
|
|
server: ServerMetrics;
|
|
recordedSpendUsd: number;
|
|
}
|
|
|
|
export type CriterionStatus = "pass" | "fail" | "not_measured";
|
|
|
|
export interface CriterionResult {
|
|
id: string;
|
|
source: "finish-line" | "harness";
|
|
description: string;
|
|
threshold: string;
|
|
status: CriterionStatus;
|
|
observed: number | string | null;
|
|
}
|
|
|
|
interface Criterion {
|
|
id: string;
|
|
source: CriterionResult["source"];
|
|
description: string;
|
|
immediate: boolean;
|
|
threshold: (t: Thresholds) => string;
|
|
evaluate: (m: StepMetrics, t: Thresholds) => { status: CriterionStatus; observed: number | string | null };
|
|
}
|
|
|
|
const below = (value: number | null, limit: number, strict = true) =>
|
|
value === null ? { status: "not_measured" as const, observed: null }
|
|
: { status: (strict ? value < limit : value <= limit) ? "pass" as const : "fail" as const, observed: value };
|
|
|
|
export const CRITERIA: readonly Criterion[] = [
|
|
{
|
|
id: "retry_needed_zero", source: "finish-line", immediate: true,
|
|
description: "Runs that need Retry (a task offered the Retry action)",
|
|
threshold: () => "= 0",
|
|
evaluate: m => ({ status: m.tasksNeedingRetry === 0 ? "pass" : "fail", observed: m.tasksNeedingRetry }),
|
|
},
|
|
{
|
|
id: "failed_runs_zero", source: "harness", immediate: true,
|
|
description: "Runs that ended failed or timed out",
|
|
threshold: () => "= 0",
|
|
evaluate: m => ({ status: m.runsFailed === 0 ? "pass" : "fail", observed: m.runsFailed }),
|
|
},
|
|
{
|
|
id: "tasks_rested_not_done_zero", source: "harness", immediate: false,
|
|
description: "Tasks that came to rest without being done (blocked, in review or cancelled)",
|
|
threshold: () => "= 0",
|
|
evaluate: m => ({ status: m.tasksFailed === 0 ? "pass" : "fail", observed: m.tasksFailed }),
|
|
},
|
|
{
|
|
id: "concurrency_reached", source: "harness", immediate: false,
|
|
description: "Peak running agents against the step target",
|
|
threshold: t => `>= ${Math.round(t.minConcurrencyRatio * 100)}% of the step`,
|
|
evaluate: (m, t) => ({ status: m.peakRunning >= Math.ceil(m.step.concurrency * t.minConcurrencyRatio) ? "pass" : "fail", observed: m.peakRunning }),
|
|
},
|
|
{
|
|
id: "board_api_p90", source: "finish-line", immediate: false,
|
|
description: "Board page API requests, p90 (ms)",
|
|
threshold: t => `< ${t.boardP90Ms} ms`,
|
|
evaluate: (m, t) => m.board.ms.n < 3 ? { status: "not_measured", observed: null } : below(m.board.ms.p90, t.boardP90Ms),
|
|
},
|
|
{
|
|
id: "board_api_errors_zero", source: "harness", immediate: false,
|
|
description: "Board API requests that failed (no answer, 4xx or 5xx)",
|
|
threshold: () => "= 0",
|
|
evaluate: m => m.board.requests === 0 ? { status: "not_measured", observed: null } : { status: m.board.failures === 0 ? "pass" : "fail", observed: m.board.failures },
|
|
},
|
|
{
|
|
id: "queue_wait_p95", source: "harness", immediate: false,
|
|
description: "Queue wait before a run starts, p95 (ms)",
|
|
threshold: t => `<= ${t.queueWaitP95Ms} ms`,
|
|
evaluate: (m, t) => below(m.queueWaitMs.p95, t.queueWaitP95Ms, false),
|
|
},
|
|
{
|
|
id: "first_activity_p95", source: "harness", immediate: false,
|
|
description: "First visible agent activity from run creation, p95 (ms), from sampled run events",
|
|
threshold: t => `<= ${t.firstActivityP95Ms} ms`,
|
|
evaluate: (m, t) => below(m.firstActivityMs.p95, t.firstActivityP95Ms, false),
|
|
},
|
|
{
|
|
id: "main_thread_cpu", source: "finish-line", immediate: false,
|
|
description: "Main-thread CPU (or event-loop utilization) of every server process, max",
|
|
threshold: t => `< ${t.maxEventLoopUtilization}`,
|
|
evaluate: (m, t) => below(m.server.mainThreadCpuMax ?? m.server.eventLoopUtilizationMax, t.maxEventLoopUtilization),
|
|
},
|
|
{
|
|
id: "run_event_lateness_p99", source: "finish-line", immediate: false,
|
|
description: "Run event lateness (runner emits to row committed), p99 (ms)",
|
|
threshold: t => `< ${t.runEventLatenessP99Ms} ms`,
|
|
evaluate: (m, t) => below(m.server.runEventLatenessP99MsMax, t.runEventLatenessP99Ms),
|
|
},
|
|
{
|
|
id: "agent_api_pre_handler_p99", source: "finish-line", immediate: false,
|
|
description: "Agent API request time before the handler, p99 (ms)",
|
|
threshold: t => `< ${t.agentApiPreHandlerP99Ms} ms`,
|
|
evaluate: (m, t) => below(m.server.agentApiPreHandlerP99MsMax, t.agentApiPreHandlerP99Ms),
|
|
},
|
|
{
|
|
id: "lease_renewals_missed_zero", source: "finish-line", immediate: false,
|
|
description: "Lease renewals that missed their deadline",
|
|
threshold: () => "= 0",
|
|
evaluate: m => m.server.leaseRenewalsMissed === null ? { status: "not_measured", observed: null }
|
|
: { status: m.server.leaseRenewalsMissed === 0 ? "pass" : "fail", observed: m.server.leaseRenewalsMissed },
|
|
},
|
|
{
|
|
id: "kill_one_process_resumes", source: "finish-line", immediate: false,
|
|
description: "Killing one server process: its runs resume on another, no Retry",
|
|
threshold: () => "resume, no Retry",
|
|
evaluate: () => ({ status: "not_measured", observed: "this harness does not kill server processes" }),
|
|
},
|
|
];
|
|
|
|
export function evaluate(metrics: StepMetrics, thresholds: Thresholds, onlyImmediate = false): CriterionResult[] {
|
|
return CRITERIA.filter(criterion => !onlyImmediate || criterion.immediate).map(criterion => ({
|
|
id: criterion.id,
|
|
source: criterion.source,
|
|
description: criterion.description,
|
|
threshold: criterion.threshold(thresholds),
|
|
...criterion.evaluate(metrics, thresholds),
|
|
}));
|
|
}
|
|
|
|
export function criteriaTable(thresholds: Thresholds): Array<Pick<CriterionResult, "id" | "source" | "description" | "threshold">> {
|
|
return CRITERIA.map(criterion => ({ id: criterion.id, source: criterion.source, description: criterion.description, threshold: criterion.threshold(thresholds) }));
|
|
}
|
|
|
|
export const firstFailure = (results: readonly CriterionResult[]) => results.find(result => result.status === "fail") ?? null;
|