mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane must run 1,000 agents at once on one workspace (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) > - Above about 50 agents, the evidence is a load model only. Nobody has run 200 or 1,000 real agents > - Phase 3 of the plan proves the finish line with real runs on hosted instances, in a fixed order > - This pull request adds the harness for those runs. It does not run it against any real instance > - The benefit is that each proof step is one command with a written cost estimate, explicit pass/fail criteria, and a guaranteed cleanup ## Linked Issues or Issue Description **What problem does this solve?** The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents. No tool exists that creates real agents and tasks on a hosted instance, ramps concurrency, measures the control plane, and judges the plan's finish line. **Proposed solution** A script under `scripts/` that ramps real agents through the product API, stops at the first failed criterion, writes a machine-readable result, and always cleans up what it created. **Alternatives considered** Extend the restart-chaos hosted mode. That mode grades release swaps at up to 30 concurrent tasks. The ramp reuses its target guard and its cleanup order, but it has a different workload, different measures and a different stop rule. ## What Changed - `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness. It reads the base URL from a flag or the environment and the credentials from the environment only. It creates or reuses a company, then creates agents, tasks, a project and an optional provider secret. It ramps concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps each agent on one small deterministic task at a time. - It measures per step: tasks started, finished, failed and needing Retry; failed runs; peak running agents; queue wait; time to first token and first activity (from sampled run events); run duration; board GET latency. An optional metrics endpoint supplies event-loop, statement and pool figures. A missing source is reported as "not measured", never as zero. - It evaluates the criteria from the plan's finish line plus harness criteria, writes them into `result.json` and `SUMMARY.md`, and stops at the first failure. - Cleanup runs in a `finally`: pause agents, cancel live runs, archive tasks, terminate and delete agents, delete the stored secret, archive the project and the created company, and wait for sandbox leases. `--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run` prints the plan and the estimate and makes no request. - Target guard: `--target-kind` is required and must match the host. Shared staging and production hosts need a second explicit flag. The script asks for the host name before it starts unless `--yes` is given. A run above the spend threshold needs `--confirm-spend <max USD>`. - `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an in-process fake API with a virtual clock. They cover ramp logic, stop on failure, criteria evaluation, cleanup after a mid-run crash and an interrupt, `--cleanup-only` with and without a state file, dry run, the production-host guard, the spend gate, and that credentials never appear in output or files. - `scripts/scale-ramp/tsconfig.json` is added to the test programs that CI typechecks. The test file joins the `test:release-registry` node test list, which CI runs. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3 runbook with commands, cost, criteria, proof order, target prerequisites and open risks. ## Verification - `node scripts/ci-profile/node-test.mjs scripts/scale-ramp/scale-ramp.test.ts scripts/typecheck-test-programs.test.mjs`: 47 tests pass. - `node node_modules/typescript/bin/tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts --base-url https://scale1.test.paperclip.app --target-kind test-stack --steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate and the criteria, and makes no request. - The harness was not run against any real instance. That needs the owner's go-ahead. ## Risks - Low risk for the product: the change adds a script, tests and docs only. No server code changes. - A real run creates real agents, tasks and sandboxes and spends money. The guard, the typed confirmation, the spend gate, the task cap and the recorded-spend stop limit this. The runbook lists the open risks. - The cost defaults are assumptions until a first real run records actual spend. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
165 lines
8.2 KiB
TypeScript
165 lines
8.2 KiB
TypeScript
import type { CleanupReport } from "./cleanup.ts";
|
|
import type { Options } from "./config.ts";
|
|
import { criteriaTable, type CriterionResult } from "./criteria.ts";
|
|
import type { SpendEstimate } from "./estimate.ts";
|
|
import type { RampOutcome } from "./ramp.ts";
|
|
import type { Distribution } from "./stats.ts";
|
|
|
|
export type Verdict = "pass" | "fail" | "incomplete" | "interrupted" | "error";
|
|
|
|
export interface RampResult {
|
|
schema: "paperclip.scale-ramp.result/v1";
|
|
runId: string;
|
|
target: { baseUrl: string; host: string; kind: string };
|
|
startedAt: string;
|
|
finishedAt: string;
|
|
plan: {
|
|
steps: Array<{ concurrency: number; holdMinutes: number }>;
|
|
drainMinutes: number;
|
|
pollSeconds: number;
|
|
boardProbeSeconds: number;
|
|
agent: { provider: string; model: string; credential: string };
|
|
estimate: SpendEstimate;
|
|
};
|
|
criteria: ReturnType<typeof criteriaTable>;
|
|
verdict: Verdict;
|
|
stop: RampOutcome["stop"];
|
|
steps: RampOutcome["steps"];
|
|
notMeasured: string[];
|
|
incompleteReasons: string[];
|
|
drained: boolean;
|
|
openTasksAtEnd: number;
|
|
tasksCreated: number;
|
|
taskCapReached: boolean;
|
|
recordedSpendUsd: number | null;
|
|
sources: Record<string, string>;
|
|
cleanup: CleanupReport | null;
|
|
requests: { total: number; failed: number };
|
|
error: string | null;
|
|
}
|
|
|
|
export function verdictOf(outcome: RampOutcome | null, error: string | null): Verdict {
|
|
if (error || !outcome) return "error";
|
|
if (outcome.stop && outcome.stop.criterion) return "fail";
|
|
if (outcome.stop) return "interrupted";
|
|
if (outcome.steps.some(step => step.criteriaFinal.some(result => result.status === "fail"))) return "fail";
|
|
if (incompleteReasons(outcome).length) return "incomplete";
|
|
return "pass";
|
|
}
|
|
|
|
export function incompleteReasons(outcome: RampOutcome | null): string[] {
|
|
if (!outcome) return [];
|
|
const reasons: string[] = [];
|
|
const missing = notMeasured(outcome);
|
|
if (missing.length) reasons.push(`criteria not measured: ${missing.join(", ")}`);
|
|
if (outcome.taskCapReached) reasons.push("the task cap was reached, so a step may have run below its concurrency");
|
|
if (outcome.openTasksAtEnd > 0) reasons.push(`${outcome.openTasksAtEnd} tasks were still open when the drain ended`);
|
|
return reasons;
|
|
}
|
|
|
|
export function notMeasured(outcome: RampOutcome | null): string[] {
|
|
const ids = new Set<string>();
|
|
for (const step of outcome?.steps ?? []) for (const result of step.criteriaFinal) if (result.status === "not_measured") ids.add(result.id);
|
|
return [...ids];
|
|
}
|
|
|
|
export function buildResult(input: {
|
|
options: Options;
|
|
estimate: SpendEstimate;
|
|
startedAt: number;
|
|
finishedAt: number;
|
|
outcome: RampOutcome | null;
|
|
cleanup: CleanupReport | null;
|
|
error: string | null;
|
|
requests: { total: number; failed: number };
|
|
credentialSource: string;
|
|
}): RampResult {
|
|
const { options, outcome } = input;
|
|
return {
|
|
schema: "paperclip.scale-ramp.result/v1",
|
|
runId: options.label,
|
|
target: { baseUrl: options.target.baseUrl, host: options.target.host, kind: options.target.kind },
|
|
startedAt: new Date(input.startedAt).toISOString(),
|
|
finishedAt: new Date(input.finishedAt).toISOString(),
|
|
plan: {
|
|
steps: options.steps.map(step => ({ concurrency: step.concurrency, holdMinutes: step.holdMs / 60_000 })),
|
|
drainMinutes: options.drainMs / 60_000,
|
|
pollSeconds: options.pollMs / 1_000,
|
|
boardProbeSeconds: options.boardProbeMs / 1_000,
|
|
agent: { provider: options.profile.provider, model: options.profile.model, credential: input.credentialSource },
|
|
estimate: input.estimate,
|
|
},
|
|
criteria: criteriaTable(options.thresholds),
|
|
verdict: verdictOf(outcome, input.error),
|
|
stop: outcome?.stop ?? null,
|
|
steps: outcome?.steps ?? [],
|
|
notMeasured: notMeasured(outcome),
|
|
incompleteReasons: verdictOf(outcome, input.error) === "incomplete" ? incompleteReasons(outcome) : [],
|
|
drained: outcome?.drained ?? false,
|
|
openTasksAtEnd: outcome?.openTasksAtEnd ?? 0,
|
|
tasksCreated: outcome?.tasksCreated ?? 0,
|
|
taskCapReached: outcome?.taskCapReached ?? false,
|
|
recordedSpendUsd: outcome ? outcome.recordedSpendUsd : null,
|
|
sources: {
|
|
runsAndTasks: outcome?.runsListTruncated ? "API (company run list was full; open tasks without a listed run were read one by one)" : "API",
|
|
boardLatency: "harness GET mix (dashboard, issues, agents, live runs, sidebar badges)",
|
|
firstTokenAndActivity: `run events of up to ${options.eventSamplePerStep} succeeded runs per step`,
|
|
serverMetrics: outcome?.metricsSource === "endpoint" ? "PAPERCLIP_SCALE_METRICS_URL"
|
|
: outcome?.metricsSource === "unavailable" ? "PAPERCLIP_SCALE_METRICS_URL gave no readable sample: not measured" : "not provided: not measured",
|
|
database: "not read by this harness: not measured",
|
|
},
|
|
cleanup: input.cleanup,
|
|
requests: input.requests,
|
|
error: input.error,
|
|
};
|
|
}
|
|
|
|
const seconds = (value: number | null) => value === null ? "not measured" : `${(value / 1_000).toFixed(1)} s`;
|
|
const millis = (value: number | null) => value === null ? "not measured" : `${Math.round(value)} ms`;
|
|
const dist = (d: Distribution, format: (value: number | null) => string) => d.n === 0 ? "not measured" : `${format(d.p50)} / ${format(d.p95)} / ${format(d.p99)} (n=${d.n})`;
|
|
|
|
function criterionLine(result: CriterionResult): string {
|
|
const observed = result.observed === null ? "not measured" : String(result.observed);
|
|
return ` - ${result.status === "pass" ? "PASS" : result.status === "fail" ? "FAIL" : "not measured"} ${result.id}: ${observed} (needs ${result.threshold})`;
|
|
}
|
|
|
|
export function summaryLines(result: RampResult): string[] {
|
|
const lines = [
|
|
`# Scale ramp ${result.runId}`,
|
|
"",
|
|
`Verdict: **${result.verdict}**. Target ${result.target.baseUrl} (${result.target.kind}).`,
|
|
result.stop ? `Stopped at step ${result.stop.stepIndex + 1}: ${result.stop.reason}.` : "No stop.",
|
|
`Tasks created: ${result.tasksCreated}${result.taskCapReached ? " (task cap reached)" : ""}. Recorded model spend: ${result.recordedSpendUsd === null ? "not measured" : `$${result.recordedSpendUsd}`}; estimate was $${result.plan.estimate.totalUsd}.`,
|
|
"",
|
|
];
|
|
for (const step of result.steps) {
|
|
const m = step.metrics;
|
|
lines.push(
|
|
`## Step ${step.index + 1}: ${step.concurrency} agents, ${step.holdMs / 60_000} min`,
|
|
"",
|
|
`- Tasks created / started / finished / failed / needed Retry: ${m.tasksCreated} / ${m.tasksStarted} / ${m.tasksFinished} / ${m.tasksFailed} / ${m.tasksNeedingRetry}`,
|
|
`- Runs failed / interrupted / cancelled: ${m.runsFailed} / ${m.runsInterrupted} / ${m.runsCancelled}`,
|
|
`- Running agents peak / mean: ${m.peakRunning} / ${m.meanRunning ?? "not measured"}`,
|
|
`- Queue wait p50 / p95 / p99: ${dist(m.queueWaitMs, seconds)}`,
|
|
`- Time to first token p50 / p95 / p99: ${dist(m.ttftMs, seconds)}`,
|
|
`- First activity p50 / p95 / p99: ${dist(m.firstActivityMs, seconds)}`,
|
|
`- Run duration p50 / p95 / p99: ${dist(m.runDurationMs, seconds)}`,
|
|
`- Board API p50 / p95 / p99: ${dist(m.board.ms, millis)}; failed ${m.board.failures} of ${m.board.requests}`,
|
|
`- Event-loop utilization max: ${m.server.eventLoopUtilizationMax ?? "not measured"}; delay p99 max: ${millis(m.server.eventLoopDelayP99MsMax)}; statements/s max: ${m.server.statementsPerSecondMax ?? "not measured"}; pool wait p99 max: ${millis(m.server.poolWaitP99MsMax)}`,
|
|
"- Criteria (final):",
|
|
...step.criteriaFinal.map(criterionLine),
|
|
"",
|
|
);
|
|
}
|
|
if (result.notMeasured.length) lines.push(`Not measured: ${result.notMeasured.join(", ")}.`, "");
|
|
if (result.incompleteReasons.length) lines.push(`Incomplete because: ${result.incompleteReasons.join("; ")}.`, "");
|
|
lines.push(`Drain: ${result.drained ? "every task finished" : `${result.openTasksAtEnd} tasks still open`}.`, "");
|
|
if (result.cleanup) {
|
|
const c = result.cleanup;
|
|
lines.push(`Cleanup in ${seconds(c.durationMs)}: ${c.runsCancelled} runs cancelled, ${c.tasksArchived} tasks archived, ${c.agentsDeleted} agents deleted`
|
|
+ ` (${c.agentsKeptForCostHistory} kept terminated for cost history), sandbox leases still active: ${c.activeSandboxLeases ?? "not readable"}, errors: ${c.errors.length}.`);
|
|
}
|
|
if (result.error) lines.push(`Error: ${result.error}`);
|
|
return lines;
|
|
}
|