Files
PaperClipAI/scripts/scale-ramp/estimate.ts
T
DottaandClaude Opus 5.5 328a954cc1 test(scaling): add a real-agent ramp harness for hosted instances (#21695)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane must run 1,000 agents at once on one workspace
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`)
> - Above about 50 agents, the evidence is a load model only. Nobody has
run 200 or 1,000 real agents
> - Phase 3 of the plan proves the finish line with real runs on hosted
instances, in a fixed order
> - This pull request adds the harness for those runs. It does not run
it against any real instance
> - The benefit is that each proof step is one command with a written
cost estimate, explicit pass/fail criteria, and a guaranteed cleanup

## Linked Issues or Issue Description

**What problem does this solve?**
The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents.
No tool exists that creates real agents and tasks on a hosted instance,
ramps concurrency, measures the control plane, and judges the plan's
finish line.

**Proposed solution**
A script under `scripts/` that ramps real agents through the product
API, stops at the first failed criterion, writes a machine-readable
result, and always cleans up what it created.

**Alternatives considered**
Extend the restart-chaos hosted mode. That mode grades release swaps at
up to 30 concurrent tasks. The ramp reuses its target guard and its
cleanup order, but it has a different workload, different measures and a
different stop rule.

## What Changed

- `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness.
It reads the base URL from a flag or the environment and the credentials
from the environment only. It creates or reuses a company, then creates
agents, tasks, a project and an optional provider secret. It ramps
concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps
each agent on one small deterministic task at a time.
- It measures per step: tasks started, finished, failed and needing
Retry; failed runs; peak running agents; queue wait; time to first token
and first activity (from sampled run events); run duration; board GET
latency. An optional metrics endpoint supplies event-loop, statement and
pool figures. A missing source is reported as "not measured", never as
zero.
- It evaluates the criteria from the plan's finish line plus harness
criteria, writes them into `result.json` and `SUMMARY.md`, and stops at
the first failure.
- Cleanup runs in a `finally`: pause agents, cancel live runs, archive
tasks, terminate and delete agents, delete the stored secret, archive
the project and the created company, and wait for sandbox leases.
`--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run`
prints the plan and the estimate and makes no request.
- Target guard: `--target-kind` is required and must match the host.
Shared staging and production hosts need a second explicit flag. The
script asks for the host name before it starts unless `--yes` is given.
A run above the spend threshold needs `--confirm-spend <max USD>`.
- `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an
in-process fake API with a virtual clock. They cover ramp logic, stop on
failure, criteria evaluation, cleanup after a mid-run crash and an
interrupt, `--cleanup-only` with and without a state file, dry run, the
production-host guard, the spend gate, and that credentials never appear
in output or files.
- `scripts/scale-ramp/tsconfig.json` is added to the test programs that
CI typechecks. The test file joins the `test:release-registry` node test
list, which CI runs.
- `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3
runbook with commands, cost, criteria, proof order, target prerequisites
and open risks.

## Verification

- `node scripts/ci-profile/node-test.mjs
scripts/scale-ramp/scale-ramp.test.ts
scripts/typecheck-test-programs.test.mjs`: 47 tests pass.
- `node node_modules/typescript/bin/tsc -p
scripts/scale-ramp/tsconfig.json`: no errors.
- `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts
--base-url https://scale1.test.paperclip.app --target-kind test-stack
--steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate
and the criteria, and makes no request.
- The harness was not run against any real instance. That needs the
owner's go-ahead.

## Risks

- Low risk for the product: the change adds a script, tests and docs
only. No server code changes.
- A real run creates real agents, tasks and sandboxes and spends money.
The guard, the typed confirmation, the spend gate, the task cap and the
recorded-spend stop limit this. The runbook lists the open risks.
- The cost defaults are assumptions until a first real run records
actual spend.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:10:51 -06:00

57 lines
2.7 KiB
TypeScript

import type { Options } from "./config.ts";
export interface SpendEstimate {
agents: number;
tasks: number;
maxTasks: number;
sandboxHours: number;
modelUsd: number;
sandboxUsd: number;
totalUsd: number;
wallMinutes: number;
assumptions: string[];
}
const round = (value: number, places = 2) => Math.round(value * 10 ** places) / 10 ** places;
export function estimateSpend(options: Pick<Options, "steps" | "drainMs" | "taskMinutes" | "maxTasks" | "pricing">): SpendEstimate {
const agents = Math.max(...options.steps.map(step => step.concurrency));
const holdMinutes = (step: { holdMs: number }) => step.holdMs / 60_000;
const tasks = options.steps.reduce((sum, step) => sum + step.concurrency * Math.ceil(holdMinutes(step) / options.taskMinutes), 0);
const maxTasks = options.maxTasks ?? Math.ceil(tasks * 1.25);
const drainMinutes = options.drainMs / 60_000;
const sandboxHours = options.steps.reduce((sum, step) => sum + step.concurrency * holdMinutes(step) / 60, 0)
+ agents * Math.min(drainMinutes, options.taskMinutes * 2) / 60;
const modelUsd = maxTasks * options.pricing.modelUsdPerTask;
const sandboxUsd = sandboxHours * options.pricing.sandboxUsdPerHour;
return {
agents,
tasks,
maxTasks,
sandboxHours: round(sandboxHours, 1),
modelUsd: round(modelUsd),
sandboxUsd: round(sandboxUsd),
totalUsd: round(modelUsd + sandboxUsd),
wallMinutes: Math.round(options.steps.reduce((sum, step) => sum + holdMinutes(step), 0) + drainMinutes),
assumptions: [
`each agent finishes one task every ${options.taskMinutes} min, so a step creates concurrency x ceil(hold / task minutes) tasks`,
`model spend is the task cap (${maxTasks}) x $${options.pricing.modelUsdPerTask} per task (PAPERCLIP_SCALE_MODEL_USD_PER_TASK)`,
`sandbox spend is agent-hours of holds plus the drain x $${options.pricing.sandboxUsdPerHour} per sandbox-hour (PAPERCLIP_SCALE_SANDBOX_USD_PER_HOUR)`,
"these are estimates; the result reports the model spend the runs recorded",
],
};
}
export type SpendDecision = { ok: true } | { ok: false; reason: string };
export function checkSpend(estimate: SpendEstimate, thresholdUsd: number, confirmedUsd: number | null): SpendDecision {
if (estimate.totalUsd <= thresholdUsd) return { ok: true };
if (confirmedUsd === null) {
return { ok: false, reason: `Estimated spend $${estimate.totalUsd} is above $${thresholdUsd}; rerun with --confirm-spend <max USD> of at least ${Math.ceil(estimate.totalUsd)}` };
}
if (confirmedUsd < estimate.totalUsd) {
return { ok: false, reason: `--confirm-spend ${confirmedUsd} is below the estimated $${estimate.totalUsd}` };
}
return { ok: true };
}