mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane must run 1,000 agents at once on one workspace (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) > - Above about 50 agents, the evidence is a load model only. Nobody has run 200 or 1,000 real agents > - Phase 3 of the plan proves the finish line with real runs on hosted instances, in a fixed order > - This pull request adds the harness for those runs. It does not run it against any real instance > - The benefit is that each proof step is one command with a written cost estimate, explicit pass/fail criteria, and a guaranteed cleanup ## Linked Issues or Issue Description **What problem does this solve?** The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents. No tool exists that creates real agents and tasks on a hosted instance, ramps concurrency, measures the control plane, and judges the plan's finish line. **Proposed solution** A script under `scripts/` that ramps real agents through the product API, stops at the first failed criterion, writes a machine-readable result, and always cleans up what it created. **Alternatives considered** Extend the restart-chaos hosted mode. That mode grades release swaps at up to 30 concurrent tasks. The ramp reuses its target guard and its cleanup order, but it has a different workload, different measures and a different stop rule. ## What Changed - `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness. It reads the base URL from a flag or the environment and the credentials from the environment only. It creates or reuses a company, then creates agents, tasks, a project and an optional provider secret. It ramps concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps each agent on one small deterministic task at a time. - It measures per step: tasks started, finished, failed and needing Retry; failed runs; peak running agents; queue wait; time to first token and first activity (from sampled run events); run duration; board GET latency. An optional metrics endpoint supplies event-loop, statement and pool figures. A missing source is reported as "not measured", never as zero. - It evaluates the criteria from the plan's finish line plus harness criteria, writes them into `result.json` and `SUMMARY.md`, and stops at the first failure. - Cleanup runs in a `finally`: pause agents, cancel live runs, archive tasks, terminate and delete agents, delete the stored secret, archive the project and the created company, and wait for sandbox leases. `--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run` prints the plan and the estimate and makes no request. - Target guard: `--target-kind` is required and must match the host. Shared staging and production hosts need a second explicit flag. The script asks for the host name before it starts unless `--yes` is given. A run above the spend threshold needs `--confirm-spend <max USD>`. - `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an in-process fake API with a virtual clock. They cover ramp logic, stop on failure, criteria evaluation, cleanup after a mid-run crash and an interrupt, `--cleanup-only` with and without a state file, dry run, the production-host guard, the spend gate, and that credentials never appear in output or files. - `scripts/scale-ramp/tsconfig.json` is added to the test programs that CI typechecks. The test file joins the `test:release-registry` node test list, which CI runs. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3 runbook with commands, cost, criteria, proof order, target prerequisites and open risks. ## Verification - `node scripts/ci-profile/node-test.mjs scripts/scale-ramp/scale-ramp.test.ts scripts/typecheck-test-programs.test.mjs`: 47 tests pass. - `node node_modules/typescript/bin/tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts --base-url https://scale1.test.paperclip.app --target-kind test-stack --steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate and the criteria, and makes no request. - The harness was not run against any real instance. That needs the owner's go-ahead. ## Risks - Low risk for the product: the change adds a script, tests and docs only. No server code changes. - A real run creates real agents, tasks and sandboxes and spends money. The guard, the typed confirmation, the spend gate, the task cap and the recorded-spend stop limit this. The runbook lists the open risks. - The cost defaults are assumptions until a first real run records actual spend. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
181 lines
8.5 KiB
TypeScript
181 lines
8.5 KiB
TypeScript
import { ApiError, type Clock, type PaperclipClient } from "./client.ts";
|
|
import { agentPrefix, companyName, mapLimit, projectName, secretPrefix, type RunState } from "./state.ts";
|
|
|
|
type Json = Record<string, any>;
|
|
|
|
export interface CleanupReport {
|
|
agentsPaused: number;
|
|
runsCancelled: number;
|
|
tasksArchived: number;
|
|
agentsTerminated: number;
|
|
agentsDeleted: number;
|
|
agentsKeptForCostHistory: number;
|
|
secretDeleted: boolean;
|
|
projectArchived: boolean;
|
|
companyArchived: boolean;
|
|
activeSandboxLeases: number | null;
|
|
durationMs: number;
|
|
errors: string[];
|
|
}
|
|
|
|
export const LIVE_RUN_STATUSES = new Set(["queued", "running", "scheduled_retry"]);
|
|
const PARALLELISM = 8;
|
|
|
|
export interface CleanupInput {
|
|
api: PaperclipClient;
|
|
clock: Clock;
|
|
label: string;
|
|
state: RunState;
|
|
terminateWaitMs?: number;
|
|
leaseWaitMs?: number;
|
|
pollMs?: number;
|
|
log?: (line: string) => void;
|
|
}
|
|
|
|
export async function discoverState(api: PaperclipClient, label: string, companyId: string | null, baseUrl: string, now: number): Promise<RunState> {
|
|
let resolved = companyId;
|
|
let created = false;
|
|
if (!resolved) {
|
|
const companies = await api.get<Json[]>("/api/companies");
|
|
const match = companies.find(company => company.name === companyName(label));
|
|
if (!match) throw new Error(`No state file for ${label} and no company named "${companyName(label)}"; pass --company <id>`);
|
|
resolved = String(match.id);
|
|
created = true;
|
|
}
|
|
return {
|
|
schema: "paperclip.scale-ramp.state/v1", label, baseUrl, companyId: resolved, companyCreated: created, environmentId: null,
|
|
projectId: null, secretId: null, agentIds: [], taskIds: [], createdAt: new Date(now).toISOString(),
|
|
};
|
|
}
|
|
|
|
export async function cleanupRun(input: CleanupInput): Promise<CleanupReport> {
|
|
const { api, clock, label, state } = input;
|
|
const started = clock.now();
|
|
const pollMs = input.pollMs ?? 3_000;
|
|
const report: CleanupReport = {
|
|
agentsPaused: 0, runsCancelled: 0, tasksArchived: 0, agentsTerminated: 0, agentsDeleted: 0, agentsKeptForCostHistory: 0,
|
|
secretDeleted: false, projectArchived: false, companyArchived: false, activeSandboxLeases: null, durationMs: 0, errors: [],
|
|
};
|
|
const attempt = async (what: string, action: () => Promise<unknown>): Promise<boolean> => {
|
|
try {
|
|
await action();
|
|
return true;
|
|
} catch (error) {
|
|
report.errors.push(`${what}: ${error instanceof Error ? error.message : String(error)}`.slice(0, 300));
|
|
return false;
|
|
}
|
|
};
|
|
const read = async <T>(what: string, route: string, fallback: T): Promise<T> => {
|
|
try {
|
|
return await api.get<T>(route);
|
|
} catch (error) {
|
|
report.errors.push(`${what}: ${error instanceof Error ? error.message : String(error)}`.slice(0, 300));
|
|
return fallback;
|
|
}
|
|
};
|
|
const companyId = state.companyId;
|
|
if (!companyId) {
|
|
report.durationMs = clock.now() - started;
|
|
return report;
|
|
}
|
|
const prefix = agentPrefix(label);
|
|
const listed = await read<Json[]>("list agents", `/api/companies/${companyId}/agents`, []);
|
|
const agents = listed.filter(agent => String(agent.name ?? "").startsWith(prefix));
|
|
const agentIds = new Set(agents.map(agent => String(agent.id)));
|
|
input.log?.(`Cleanup ${label}: ${agents.length} agents, ${state.taskIds.length} recorded tasks`);
|
|
|
|
await mapLimit(agents.filter(agent => agent.status !== "terminated" && agent.status !== "paused"), PARALLELISM, async agent => {
|
|
if (await attempt(`pause agent ${agent.id}`, () => api.patch(`/api/agents/${agent.id}`, { status: "paused" }))) report.agentsPaused += 1;
|
|
});
|
|
|
|
const runs = await read<Json[]>("list runs", `/api/companies/${companyId}/heartbeat-runs?summary=true&limit=1000`, []);
|
|
const live = new Map<string, Json>();
|
|
for (const run of runs) if (agentIds.has(String(run.agentId)) && LIVE_RUN_STATUSES.has(String(run.status))) live.set(String(run.id), run);
|
|
if (runs.length >= 1_000) {
|
|
await mapLimit([...agentIds], PARALLELISM, async agentId => {
|
|
const own = await read<Json[]>(`list runs of ${agentId}`, `/api/companies/${companyId}/heartbeat-runs?agentId=${agentId}&summary=true&limit=50`, []);
|
|
for (const run of own) if (LIVE_RUN_STATUSES.has(String(run.status))) live.set(String(run.id), run);
|
|
});
|
|
}
|
|
await mapLimit([...live.keys()], PARALLELISM, async runId => {
|
|
if (await attempt(`cancel run ${runId}`, () => api.post(`/api/heartbeat-runs/${runId}/cancel`))) report.runsCancelled += 1;
|
|
});
|
|
|
|
const taskIds = new Set(state.taskIds);
|
|
await mapLimit([...agentIds], PARALLELISM, async agentId => {
|
|
const issues = await read<Json[]>(`list tasks of ${agentId}`, `/api/companies/${companyId}/issues?assigneeAgentId=${agentId}&limit=500`, []);
|
|
for (const issue of issues) if (issue.assigneeAgentId === agentId && !issue.hiddenAt) taskIds.add(String(issue.id));
|
|
});
|
|
await mapLimit([...taskIds], PARALLELISM, async taskId => {
|
|
const archived = await attempt(`archive task ${taskId}`, async () => {
|
|
try {
|
|
await api.patch(`/api/issues/${taskId}`, { hiddenAt: new Date(clock.now()).toISOString() });
|
|
} catch (error) {
|
|
if (!(error instanceof ApiError && error.status === 404)) throw error;
|
|
}
|
|
});
|
|
if (archived) report.tasksArchived += 1;
|
|
});
|
|
|
|
await mapLimit(agents.filter(agent => agent.status !== "terminated"), PARALLELISM, async agent => {
|
|
if (await attempt(`terminate agent ${agent.id}`, () => api.post(`/api/agents/${agent.id}/terminate`))) report.agentsTerminated += 1;
|
|
});
|
|
const pending = new Set(agentIds);
|
|
for (const until = clock.now() + (input.terminateWaitMs ?? 300_000); pending.size && clock.now() < until;) {
|
|
const current = await read<Json[]>("list agents", `/api/companies/${companyId}/agents`, []);
|
|
for (const agent of current) {
|
|
if (!pending.has(String(agent.id))) continue;
|
|
if (agent.status === "terminated" && (agent.lifecycleState == null || agent.lifecycleState === "terminated")) pending.delete(String(agent.id));
|
|
}
|
|
for (const id of [...pending]) if (!current.some(agent => String(agent.id) === id)) pending.delete(id);
|
|
if (pending.size) await clock.sleep(pollMs);
|
|
}
|
|
if (pending.size) report.errors.push(`${pending.size} agents were not terminated in time; run --cleanup-only ${label} again`);
|
|
await mapLimit([...agentIds].filter(id => !pending.has(id)), PARALLELISM, async agentId => {
|
|
try {
|
|
await api.delete(`/api/agents/${agentId}`);
|
|
report.agentsDeleted += 1;
|
|
} catch (error) {
|
|
if (error instanceof ApiError && error.status === 409 && error.message.includes("agent_cost_history_retained")) report.agentsKeptForCostHistory += 1;
|
|
else report.errors.push(`delete agent ${agentId}: ${error instanceof Error ? error.message : String(error)}`.slice(0, 300));
|
|
}
|
|
});
|
|
|
|
const secrets = await read<Json[]>("list secrets", `/api/companies/${companyId}/secrets`, []);
|
|
for (const secret of secrets.filter(secret => String(secret.name ?? "").startsWith(secretPrefix(label)))) {
|
|
if (await attempt(`delete secret ${secret.id}`, () => api.delete(`/api/secrets/${secret.id}`))) report.secretDeleted = true;
|
|
}
|
|
|
|
const projects = await read<Json[]>("list projects", `/api/companies/${companyId}/projects`, []);
|
|
for (const project of projects.filter(project => project.name === projectName(label) && !project.archivedAt)) {
|
|
if (await attempt(`archive project ${project.id}`, () => api.patch(`/api/projects/${project.id}`, { archivedAt: new Date(clock.now()).toISOString() }))) {
|
|
report.projectArchived = true;
|
|
}
|
|
}
|
|
|
|
if (state.environmentId && taskIds.size) {
|
|
for (const until = clock.now() + (input.leaseWaitMs ?? 300_000); ;) {
|
|
let leases: Json[] | null = null;
|
|
try {
|
|
leases = await api.get<Json[]>(`/api/environments/${state.environmentId}/leases?status=active`);
|
|
} catch {
|
|
leases = null;
|
|
}
|
|
if (leases === null) break;
|
|
report.activeSandboxLeases = leases.filter(lease => taskIds.has(String(lease.issueId))).length;
|
|
if (report.activeSandboxLeases === 0 || clock.now() >= until) break;
|
|
await clock.sleep(pollMs);
|
|
}
|
|
if (report.activeSandboxLeases) report.errors.push(`${report.activeSandboxLeases} sandbox leases of this run are still active`);
|
|
}
|
|
|
|
if (state.companyCreated) {
|
|
const company = await read<Json | null>("read company", `/api/companies/${companyId}`, null);
|
|
if (company && company.name === companyName(label) && company.status !== "archived") {
|
|
if (await attempt(`archive company ${companyId}`, () => api.post(`/api/companies/${companyId}/archive`))) report.companyArchived = true;
|
|
}
|
|
}
|
|
report.durationMs = clock.now() - started;
|
|
return report;
|
|
}
|