mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:48:15 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane must run 1,000 agents at once on one workspace (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) > - Above about 50 agents, the evidence is a load model only. Nobody has run 200 or 1,000 real agents > - Phase 3 of the plan proves the finish line with real runs on hosted instances, in a fixed order > - This pull request adds the harness for those runs. It does not run it against any real instance > - The benefit is that each proof step is one command with a written cost estimate, explicit pass/fail criteria, and a guaranteed cleanup ## Linked Issues or Issue Description **What problem does this solve?** The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents. No tool exists that creates real agents and tasks on a hosted instance, ramps concurrency, measures the control plane, and judges the plan's finish line. **Proposed solution** A script under `scripts/` that ramps real agents through the product API, stops at the first failed criterion, writes a machine-readable result, and always cleans up what it created. **Alternatives considered** Extend the restart-chaos hosted mode. That mode grades release swaps at up to 30 concurrent tasks. The ramp reuses its target guard and its cleanup order, but it has a different workload, different measures and a different stop rule. ## What Changed - `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness. It reads the base URL from a flag or the environment and the credentials from the environment only. It creates or reuses a company, then creates agents, tasks, a project and an optional provider secret. It ramps concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps each agent on one small deterministic task at a time. - It measures per step: tasks started, finished, failed and needing Retry; failed runs; peak running agents; queue wait; time to first token and first activity (from sampled run events); run duration; board GET latency. An optional metrics endpoint supplies event-loop, statement and pool figures. A missing source is reported as "not measured", never as zero. - It evaluates the criteria from the plan's finish line plus harness criteria, writes them into `result.json` and `SUMMARY.md`, and stops at the first failure. - Cleanup runs in a `finally`: pause agents, cancel live runs, archive tasks, terminate and delete agents, delete the stored secret, archive the project and the created company, and wait for sandbox leases. `--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run` prints the plan and the estimate and makes no request. - Target guard: `--target-kind` is required and must match the host. Shared staging and production hosts need a second explicit flag. The script asks for the host name before it starts unless `--yes` is given. A run above the spend threshold needs `--confirm-spend <max USD>`. - `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an in-process fake API with a virtual clock. They cover ramp logic, stop on failure, criteria evaluation, cleanup after a mid-run crash and an interrupt, `--cleanup-only` with and without a state file, dry run, the production-host guard, the spend gate, and that credentials never appear in output or files. - `scripts/scale-ramp/tsconfig.json` is added to the test programs that CI typechecks. The test file joins the `test:release-registry` node test list, which CI runs. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3 runbook with commands, cost, criteria, proof order, target prerequisites and open risks. ## Verification - `node scripts/ci-profile/node-test.mjs scripts/scale-ramp/scale-ramp.test.ts scripts/typecheck-test-programs.test.mjs`: 47 tests pass. - `node node_modules/typescript/bin/tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts --base-url https://scale1.test.paperclip.app --target-kind test-stack --steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate and the criteria, and makes no request. - The harness was not run against any real instance. That needs the owner's go-ahead. ## Risks - Low risk for the product: the change adds a script, tests and docs only. No server code changes. - A real run creates real agents, tasks and sandboxes and spends money. The guard, the typed confirmation, the spend gate, the task cap and the recorded-spend stop limit this. The runbook lists the open risks. - The cost defaults are assumptions until a first real run records actual spend. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
269 lines
11 KiB
JavaScript
269 lines
11 KiB
JavaScript
#!/usr/bin/env node
|
|
// Typechecks the test files that no build or package typecheck reads.
|
|
//
|
|
// Most package tsconfigs exclude tests so that `tsc` does not emit them, and
|
|
// Vitest strips types, so a type error in those tests fails nothing. Each program
|
|
// in PROGRAMS reads them without emitting. Sandbox providers are outside the
|
|
// workspace, so the root install has none of their dependencies. They install from
|
|
// typecheck-sandbox-providers/pnpm-lock.yaml here, because their SDK types are part
|
|
// of what a test checks. A provider with its own lockfile (LOCKED_PROVIDERS) installs
|
|
// from that lockfile instead, so each lockfile has one owner and one Dependabot entry.
|
|
//
|
|
// The script then fails if a tracked test file is in no program. tsc decides that,
|
|
// not a glob: `include` silently drops foo.test.tsx when foo.test.ts exists.
|
|
//
|
|
// Usage: node scripts/typecheck-test-programs.mjs [--low-memory] [tsconfig ...]
|
|
// Naming a tsconfig runs only that program and skips the coverage check.
|
|
|
|
import { spawnSync } from "node:child_process";
|
|
import { existsSync } from "node:fs";
|
|
import path from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
import { linkExcludedPlugins } from "./link-plugin-dev-sdk.mjs";
|
|
import { LOW_MEMORY_VARIABLE, runTsc } from "./run-tsc.mjs";
|
|
import { PROJECTS } from "./typecheck-test-projects.mjs";
|
|
|
|
export const PROVIDER_WORKSPACE = "scripts/typecheck-sandbox-providers";
|
|
|
|
export const SANDBOX_PROVIDERS = [
|
|
"packages/plugins/sandbox-providers/boxd",
|
|
"packages/plugins/sandbox-providers/cloudflare",
|
|
"packages/plugins/sandbox-providers/cloudflare/bridge-template",
|
|
"packages/plugins/sandbox-providers/createos",
|
|
"packages/plugins/sandbox-providers/daytona",
|
|
"packages/plugins/sandbox-providers/e2b",
|
|
"packages/plugins/sandbox-providers/exe-dev",
|
|
"packages/plugins/sandbox-providers/kubernetes",
|
|
"packages/plugins/sandbox-providers/modal",
|
|
"packages/plugins/sandbox-providers/novita",
|
|
];
|
|
|
|
// Providers that commit their own pnpm-lock.yaml. The release build and the Daytona
|
|
// end-to-end job install it frozen, so the shared workspace leaves them out.
|
|
export const LOCKED_PROVIDERS = ["packages/plugins/sandbox-providers/daytona"];
|
|
|
|
export const PROGRAMS = [
|
|
"packages/db/tsconfig.test.json",
|
|
"packages/kv-demo-mcp-server/tsconfig.test.json",
|
|
"packages/paperclip-runner/tsconfig.browser.test.json",
|
|
"packages/paperclip-runner/tsconfig.surfaces.test.json",
|
|
"packages/paperclip-runner/tsconfig.test.json",
|
|
"packages/plugins/examples/plugin-orchestration-smoke-example/tsconfig.test.json",
|
|
"packages/plugins/plugin-workspace-diff/tsconfig.test.json",
|
|
"packages/plugins/sdk/tsconfig.test.json",
|
|
...SANDBOX_PROVIDERS.map((directory) => `${directory}/tsconfig.test.json`),
|
|
"scripts/scale-ramp/tsconfig.json",
|
|
"tests/runner-acceptance/tsconfig.json",
|
|
"tests/tsconfig.playwright.json",
|
|
];
|
|
|
|
// server/scripts/typecheck-layers.mjs reads this tree. Everything else must be in a program.
|
|
export const CHECKED_ELSEWHERE = ["server/src/"];
|
|
|
|
const PREPARE_STEPS = [
|
|
["--filter", "@paperclipai/plugin-sdk", "ensure-build-deps"],
|
|
["--filter", "@paperclipai/paperclip-runner", "run", "ensure:eval-build-deps"],
|
|
];
|
|
// Replacing a provider's standalone node_modules would otherwise ask for confirmation.
|
|
const INSTALL_ARGS = ["install", "--frozen-lockfile", "--ignore-scripts", "--prefer-offline", "--config.confirmModulesPurge=false"];
|
|
const LOCKED_INSTALL_ARGS = ["install", "--ignore-workspace", ...INSTALL_ARGS.slice(1)];
|
|
// Contributors do not commit provider lockfiles: the lockfile refresh workflow regenerates them after
|
|
// merge, as it does the root pnpm-lock.yaml. Until then a changed manifest leaves the lockfile stale,
|
|
// so resolve it in place, like the root install in CI.
|
|
export const RESOLVE_ARGS = ["install", "--resolution-only", "--ignore-scripts", "--no-frozen-lockfile"];
|
|
const LOCKED_RESOLVE_ARGS = ["install", "--ignore-workspace", ...RESOLVE_ARGS.slice(1)];
|
|
|
|
export function findUncheckedTests({ trackedFiles, checkedFiles, checkedElsewhere = CHECKED_ELSEWHERE }) {
|
|
const checked = new Set(checkedFiles);
|
|
return trackedFiles.filter(
|
|
(file) =>
|
|
/\.(test|spec)\.tsx?$/.test(file) &&
|
|
!checked.has(file) &&
|
|
!checkedElsewhere.some((prefix) => file.startsWith(prefix)),
|
|
);
|
|
}
|
|
|
|
function runPnpm(args, { cwd, env }) {
|
|
const result = spawnSync("pnpm", args, {
|
|
cwd,
|
|
env,
|
|
stdio: "inherit",
|
|
// A bare `pnpm` on Windows is pnpm.cmd, which needs the shell to resolve.
|
|
shell: process.platform === "win32",
|
|
});
|
|
if (result.error) throw result.error;
|
|
return result.status ?? 1;
|
|
}
|
|
|
|
function listProgramFiles(config, { cwd, env }) {
|
|
const tsc = path.join(cwd, "node_modules", "typescript", "bin", "tsc");
|
|
const result = spawnSync(process.execPath, [tsc, "-p", config, "--listFilesOnly"], {
|
|
cwd,
|
|
env,
|
|
encoding: "utf8",
|
|
maxBuffer: 1 << 28,
|
|
});
|
|
if (result.error) throw result.error;
|
|
return {
|
|
status: result.status ?? 1,
|
|
output: result.stdout,
|
|
files: result.stdout
|
|
.split("\n")
|
|
.filter(Boolean)
|
|
.map((file) => path.relative(cwd, file).split(path.sep).join("/")),
|
|
};
|
|
}
|
|
|
|
function listWorkspaceConfigs(root) {
|
|
const result = spawnSync("pnpm", ["ls", "-r", "--depth", "-1", "--json"], {
|
|
cwd: root,
|
|
encoding: "utf8",
|
|
maxBuffer: 1 << 26,
|
|
shell: process.platform === "win32",
|
|
});
|
|
if (result.error) throw result.error;
|
|
if (result.status !== 0) throw new Error("Unable to list pnpm workspace packages.");
|
|
return JSON.parse(result.stdout)
|
|
.map((workspace) => path.relative(root, path.join(workspace.path, "tsconfig.json")).split(path.sep).join("/"))
|
|
.filter((config) => config !== "server/tsconfig.json" && existsSync(path.join(root, config)));
|
|
}
|
|
|
|
function listTrackedFiles(root) {
|
|
const result = spawnSync("git", ["ls-files", "-z"], { cwd: root, encoding: "utf8", maxBuffer: 1 << 28 });
|
|
if (result.error) throw result.error;
|
|
if (result.status !== 0) throw new Error("Unable to list tracked files.");
|
|
return result.stdout.split("\0").filter(Boolean);
|
|
}
|
|
|
|
/** Returns the test files in no program, or null if a program could not be listed. */
|
|
export function checkCoverage({
|
|
root,
|
|
env = process.env,
|
|
programs = PROGRAMS,
|
|
workspaceConfigs = listWorkspaceConfigs,
|
|
trackedFiles = listTrackedFiles,
|
|
listFiles = listProgramFiles,
|
|
logError = console.error,
|
|
}) {
|
|
const checkedFiles = [];
|
|
// `pnpm build` and `typecheck:build-gaps` check each workspace's tsconfig.json.
|
|
// scripts/typecheck-test-projects.mjs checks these same files through tsconfig.layers.json.
|
|
const projectConfigs = PROJECTS.map((project) => `${project}/tsconfig.json`);
|
|
for (const config of [...workspaceConfigs(root), ...projectConfigs, ...programs]) {
|
|
const listing = listFiles(config, { cwd: root, env });
|
|
if (listing.status !== 0) {
|
|
logError(`typecheck:test-programs: could not list the files of ${config}\n${listing.output ?? ""}`);
|
|
return null;
|
|
}
|
|
checkedFiles.push(...listing.files);
|
|
}
|
|
return findUncheckedTests({ trackedFiles: trackedFiles(root), checkedFiles });
|
|
}
|
|
|
|
export function prepareTestPrograms({
|
|
root,
|
|
env = process.env,
|
|
pnpm = runPnpm,
|
|
linkPlugins = linkExcludedPlugins,
|
|
log = console.log,
|
|
logError = console.error,
|
|
}) {
|
|
for (const args of PREPARE_STEPS) {
|
|
const status = pnpm(args, { cwd: root, env });
|
|
if (status !== 0) return status;
|
|
}
|
|
const install = (directory, frozenArgs, resolveArgs) => {
|
|
const cwd = path.join(root, directory);
|
|
let status = pnpm(frozenArgs, { cwd, env });
|
|
if (status !== 0) {
|
|
log(
|
|
`typecheck:test-programs: ${directory}/pnpm-lock.yaml does not match its manifests. ` +
|
|
`Resolving it in place. Do not commit it: the lockfile refresh workflow updates it after merge.`,
|
|
);
|
|
status = pnpm(resolveArgs, { cwd, env });
|
|
if (status === 0) status = pnpm(frozenArgs, { cwd, env });
|
|
}
|
|
if (status !== 0) logError(`typecheck:test-programs: could not install the sandbox providers in ${directory}`);
|
|
return status;
|
|
};
|
|
const installed = install(PROVIDER_WORKSPACE, INSTALL_ARGS, RESOLVE_ARGS);
|
|
if (installed !== 0) return installed;
|
|
for (const provider of LOCKED_PROVIDERS) {
|
|
const status = install(provider, LOCKED_INSTALL_ARGS, LOCKED_RESOLVE_ARGS);
|
|
if (status !== 0) return status;
|
|
}
|
|
// The installs remove the SDK link from each provider's node_modules.
|
|
linkPlugins();
|
|
return 0;
|
|
}
|
|
|
|
export function typecheckTestPrograms({
|
|
root,
|
|
env = process.env,
|
|
programs = PROGRAMS,
|
|
coverage = true,
|
|
prepare = true,
|
|
pnpm = runPnpm,
|
|
linkPlugins = linkExcludedPlugins,
|
|
run = runTsc,
|
|
checkFiles = checkCoverage,
|
|
log = console.log,
|
|
logError = console.error,
|
|
}) {
|
|
if (prepare) {
|
|
const prepared = prepareTestPrograms({ root, env, pnpm, linkPlugins, log, logError });
|
|
if (prepared !== 0) return prepared;
|
|
}
|
|
|
|
let exitCode = 0;
|
|
for (const config of programs) {
|
|
log(`typecheck:test-programs: ${config}`);
|
|
const { status, signal } = run(["-p", config], { cwd: root, env });
|
|
if (signal) return status || 1;
|
|
if (status !== 0) exitCode = status;
|
|
}
|
|
if (!coverage) return exitCode;
|
|
|
|
const unchecked = checkFiles({ root, env, logError });
|
|
if (unchecked === null) return exitCode || 1;
|
|
if (unchecked.length > 0) {
|
|
logError(
|
|
`typecheck:test-programs: ${unchecked.length} test files are in no typechecked program. ` +
|
|
`Add them to a program in scripts/typecheck-test-programs.mjs. A foo.test.tsx beside a foo.test.ts needs a new name.\n` +
|
|
unchecked.map((file) => ` ${file}`).join("\n"),
|
|
);
|
|
return exitCode || 1;
|
|
}
|
|
return exitCode;
|
|
}
|
|
|
|
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
|
const args = process.argv.slice(2);
|
|
const flags = args.filter((arg) => arg.startsWith("--"));
|
|
const programs = args.filter((arg) => !arg.startsWith("--"));
|
|
const unknownFlags = flags.filter((flag) => !["--low-memory", "--prepare-only", "--skip-prepare"].includes(flag));
|
|
const unknown = programs.filter((program) => !PROGRAMS.includes(program));
|
|
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
|
const env = flags.includes("--low-memory") ? { ...process.env, [LOW_MEMORY_VARIABLE]: "1" } : process.env;
|
|
if (unknownFlags.length > 0) {
|
|
console.error(`Unknown option: ${unknownFlags.join(", ")}`);
|
|
process.exitCode = 2;
|
|
} else if (flags.includes("--prepare-only") && (flags.includes("--skip-prepare") || programs.length > 0)) {
|
|
console.error("--prepare-only takes no programs and cannot be combined with --skip-prepare");
|
|
process.exitCode = 2;
|
|
} else if (unknown.length > 0) {
|
|
console.error(`Unknown program: ${unknown.join(", ")}. Known programs:\n ${PROGRAMS.join("\n ")}`);
|
|
process.exitCode = 2;
|
|
} else if (flags.includes("--prepare-only")) {
|
|
process.exitCode = prepareTestPrograms({ root, env });
|
|
} else {
|
|
process.exitCode = typecheckTestPrograms({
|
|
root,
|
|
env,
|
|
programs: programs.length > 0 ? programs : PROGRAMS,
|
|
coverage: programs.length === 0,
|
|
prepare: !flags.includes("--skip-prepare"),
|
|
});
|
|
}
|
|
}
|