mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane must run 1,000 agents at once on one workspace (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) > - Above about 50 agents, the evidence is a load model only. Nobody has run 200 or 1,000 real agents > - Phase 3 of the plan proves the finish line with real runs on hosted instances, in a fixed order > - This pull request adds the harness for those runs. It does not run it against any real instance > - The benefit is that each proof step is one command with a written cost estimate, explicit pass/fail criteria, and a guaranteed cleanup ## Linked Issues or Issue Description **What problem does this solve?** The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents. No tool exists that creates real agents and tasks on a hosted instance, ramps concurrency, measures the control plane, and judges the plan's finish line. **Proposed solution** A script under `scripts/` that ramps real agents through the product API, stops at the first failed criterion, writes a machine-readable result, and always cleans up what it created. **Alternatives considered** Extend the restart-chaos hosted mode. That mode grades release swaps at up to 30 concurrent tasks. The ramp reuses its target guard and its cleanup order, but it has a different workload, different measures and a different stop rule. ## What Changed - `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness. It reads the base URL from a flag or the environment and the credentials from the environment only. It creates or reuses a company, then creates agents, tasks, a project and an optional provider secret. It ramps concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps each agent on one small deterministic task at a time. - It measures per step: tasks started, finished, failed and needing Retry; failed runs; peak running agents; queue wait; time to first token and first activity (from sampled run events); run duration; board GET latency. An optional metrics endpoint supplies event-loop, statement and pool figures. A missing source is reported as "not measured", never as zero. - It evaluates the criteria from the plan's finish line plus harness criteria, writes them into `result.json` and `SUMMARY.md`, and stops at the first failure. - Cleanup runs in a `finally`: pause agents, cancel live runs, archive tasks, terminate and delete agents, delete the stored secret, archive the project and the created company, and wait for sandbox leases. `--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run` prints the plan and the estimate and makes no request. - Target guard: `--target-kind` is required and must match the host. Shared staging and production hosts need a second explicit flag. The script asks for the host name before it starts unless `--yes` is given. A run above the spend threshold needs `--confirm-spend <max USD>`. - `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an in-process fake API with a virtual clock. They cover ramp logic, stop on failure, criteria evaluation, cleanup after a mid-run crash and an interrupt, `--cleanup-only` with and without a state file, dry run, the production-host guard, the spend gate, and that credentials never appear in output or files. - `scripts/scale-ramp/tsconfig.json` is added to the test programs that CI typechecks. The test file joins the `test:release-registry` node test list, which CI runs. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3 runbook with commands, cost, criteria, proof order, target prerequisites and open risks. ## Verification - `node scripts/ci-profile/node-test.mjs scripts/scale-ramp/scale-ramp.test.ts scripts/typecheck-test-programs.test.mjs`: 47 tests pass. - `node node_modules/typescript/bin/tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts --base-url https://scale1.test.paperclip.app --target-kind test-stack --steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate and the criteria, and makes no request. - The harness was not run against any real instance. That needs the owner's go-ahead. ## Risks - Low risk for the product: the change adds a script, tests and docs only. No server code changes. - A real run creates real agents, tasks and sandboxes and spends money. The guard, the typed confirmation, the spend gate, the task cap and the recorded-spend stop limit this. The runbook lists the open risks. - The cost defaults are assumptions until a first real run records actual spend. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
141 lines
6.1 KiB
TypeScript
141 lines
6.1 KiB
TypeScript
import { ApiError, type PaperclipClient } from "./client.ts";
|
|
import type { AgentProfile, Options } from "./config.ts";
|
|
import { agentName, companyName, mapLimit, projectName, secretPrefix, type RunState, type StateStore } from "./state.ts";
|
|
|
|
type Json = Record<string, any>;
|
|
|
|
export class SetupError extends Error {
|
|
constructor(message: string) {
|
|
super(message);
|
|
this.name = "SetupError";
|
|
}
|
|
}
|
|
|
|
export async function resolveCompany(api: PaperclipClient, options: Pick<Options, "companyId" | "label">, state: RunState, store: StateStore): Promise<void> {
|
|
if (options.companyId) {
|
|
const me = await api.get<{ companyIds?: string[] }>("/api/cli-auth/me");
|
|
if (!(me.companyIds ?? []).includes(options.companyId)) throw new SetupError(`The signed-in board user is not a member of company ${options.companyId}`);
|
|
state.companyId = options.companyId;
|
|
await store.save(state);
|
|
return;
|
|
}
|
|
try {
|
|
const company = await api.post<{ id: string }>("/api/companies", {
|
|
name: companyName(options.label),
|
|
description: `Scale ramp ${options.label}. Created and archived by scripts/scale-agents-ramp.ts.`,
|
|
});
|
|
state.companyId = String(company.id);
|
|
state.companyCreated = true;
|
|
await store.save(state);
|
|
} catch (error) {
|
|
if (error instanceof ApiError && error.status === 403) {
|
|
throw new SetupError(`This instance does not let the harness create a company (${error.code ?? "forbidden"}); pass --company <id> or PAPERCLIP_SCALE_COMPANY_ID`);
|
|
}
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
export async function pickEnvironment(api: PaperclipClient, companyId: string, requested: string | null, allowLocal: boolean): Promise<string> {
|
|
const environments = await api.get<Json[]>(`/api/companies/${companyId}/environments`);
|
|
if (requested) {
|
|
if (!environments.some(environment => environment.id === requested)) throw new SetupError(`Environment ${requested} is not in the company`);
|
|
return requested;
|
|
}
|
|
const active = environments.filter(environment => environment.status !== "archived" && (allowLocal || environment.driver !== "local"));
|
|
const sandboxes = active.filter(environment => environment.driver === "sandbox");
|
|
const candidates = sandboxes.length ? sandboxes : active;
|
|
if (candidates.length !== 1) {
|
|
throw new SetupError(`The company has ${candidates.length} usable environments; pass --environment <id>`);
|
|
}
|
|
return String(candidates[0]!.id);
|
|
}
|
|
|
|
export const AGENT_INSTRUCTIONS = [
|
|
"You are a load-test worker for a Paperclip scale ramp.",
|
|
"Follow the assigned task literally and in order. Keep every reply short.",
|
|
"Use only the runtime's real tool-call channel for tools. Never print tool-call markup as text.",
|
|
"Never print, store or expose credential values. Never create other tasks, documents or agents.",
|
|
].join("\n");
|
|
|
|
export function agentPayload(input: { label: string; index: number; environmentId: string; profile: AgentProfile; secretId: string | null }): Json {
|
|
const { profile } = input;
|
|
const env = profile.credentialKey && input.secretId
|
|
? { [profile.credentialKey]: { type: "secret_ref", secretId: input.secretId, version: "latest" } }
|
|
: {};
|
|
return {
|
|
name: agentName(input.label, input.index),
|
|
role: "qa",
|
|
title: "Scale ramp worker",
|
|
capabilities: "Runs one small deterministic task at a time for a load test.",
|
|
adapterType: "paperclip_runner",
|
|
adapterConfig: {
|
|
provider: profile.provider,
|
|
model: profile.model,
|
|
lifecycleMode: "warm",
|
|
idleTimeoutMs: 300_000,
|
|
...(profile.provider === "opencode" ? { opencodePermissionMode: "allow" } : {}),
|
|
env,
|
|
...profile.adapterOverrides,
|
|
},
|
|
defaultEnvironmentId: input.environmentId,
|
|
budgetMonthlyCents: 0,
|
|
runtimeConfig: { heartbeat: { enabled: false, maxConcurrentRuns: 1 } },
|
|
instructionsBundle: { entryFile: "AGENTS.md", files: { "AGENTS.md": AGENT_INSTRUCTIONS } },
|
|
};
|
|
}
|
|
|
|
export async function prepareCompany(input: {
|
|
api: PaperclipClient;
|
|
options: Options;
|
|
state: RunState;
|
|
store: StateStore;
|
|
credential: string | null;
|
|
log: (line: string) => void;
|
|
}): Promise<void> {
|
|
const { api, options, state, store } = input;
|
|
await resolveCompany(api, options, state, store);
|
|
const companyId = state.companyId!;
|
|
state.environmentId = await pickEnvironment(api, companyId, options.environmentId, options.target.kind === "local");
|
|
await store.save(state);
|
|
const profile = options.profile;
|
|
if (profile.credentialKey && !profile.providerSecretId) {
|
|
if (!input.credential) throw new SetupError(`Set ${profile.credentialKey} in the environment, or PAPERCLIP_SCALE_PROVIDER_SECRET_ID to reuse a stored secret`);
|
|
const secret = await api.post<{ id: string }>(`/api/companies/${companyId}/secrets`, {
|
|
name: `${secretPrefix(options.label)}${profile.credentialKey}`,
|
|
value: input.credential,
|
|
description: `Scale ramp ${options.label}; deleted at cleanup`,
|
|
});
|
|
state.secretId = String(secret.id);
|
|
await store.save(state);
|
|
}
|
|
const project = await api.post<{ id: string }>(`/api/companies/${companyId}/projects`, {
|
|
name: projectName(options.label),
|
|
description: `Scale ramp ${options.label}. Created and archived by scripts/scale-agents-ramp.ts.`,
|
|
});
|
|
state.projectId = String(project.id);
|
|
await store.save(state);
|
|
input.log(`Prepared company ${companyId}: environment ${state.environmentId}, project ${state.projectId}`);
|
|
}
|
|
|
|
export async function ensureAgents(input: {
|
|
api: PaperclipClient;
|
|
options: Options;
|
|
state: RunState;
|
|
store: StateStore;
|
|
count: number;
|
|
}): Promise<void> {
|
|
const { api, options, state, store } = input;
|
|
const secretId = options.profile.providerSecretId ?? state.secretId;
|
|
const start = state.agentIds.length;
|
|
const indices = Array.from({ length: Math.max(0, input.count - start) }, (_, offset) => start + offset);
|
|
await mapLimit(indices, AGENT_CREATE_PARALLELISM, async index => {
|
|
const created = await api.post<{ id: string }>(`/api/companies/${state.companyId}/agents`, agentPayload({
|
|
label: options.label, index, environmentId: state.environmentId!, profile: options.profile, secretId,
|
|
}));
|
|
state.agentIds.push(String(created.id));
|
|
await store.save(state);
|
|
});
|
|
}
|
|
|
|
export const AGENT_CREATE_PARALLELISM = 8;
|