mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
test(scaling): add a real-agent ramp harness for hosted instances (#21695)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane must run 1,000 agents at once on one workspace (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) > - Above about 50 agents, the evidence is a load model only. Nobody has run 200 or 1,000 real agents > - Phase 3 of the plan proves the finish line with real runs on hosted instances, in a fixed order > - This pull request adds the harness for those runs. It does not run it against any real instance > - The benefit is that each proof step is one command with a written cost estimate, explicit pass/fail criteria, and a guaranteed cleanup ## Linked Issues or Issue Description **What problem does this solve?** The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents. No tool exists that creates real agents and tasks on a hosted instance, ramps concurrency, measures the control plane, and judges the plan's finish line. **Proposed solution** A script under `scripts/` that ramps real agents through the product API, stops at the first failed criterion, writes a machine-readable result, and always cleans up what it created. **Alternatives considered** Extend the restart-chaos hosted mode. That mode grades release swaps at up to 30 concurrent tasks. The ramp reuses its target guard and its cleanup order, but it has a different workload, different measures and a different stop rule. ## What Changed - `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness. It reads the base URL from a flag or the environment and the credentials from the environment only. It creates or reuses a company, then creates agents, tasks, a project and an optional provider secret. It ramps concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps each agent on one small deterministic task at a time. - It measures per step: tasks started, finished, failed and needing Retry; failed runs; peak running agents; queue wait; time to first token and first activity (from sampled run events); run duration; board GET latency. An optional metrics endpoint supplies event-loop, statement and pool figures. A missing source is reported as "not measured", never as zero. - It evaluates the criteria from the plan's finish line plus harness criteria, writes them into `result.json` and `SUMMARY.md`, and stops at the first failure. - Cleanup runs in a `finally`: pause agents, cancel live runs, archive tasks, terminate and delete agents, delete the stored secret, archive the project and the created company, and wait for sandbox leases. `--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run` prints the plan and the estimate and makes no request. - Target guard: `--target-kind` is required and must match the host. Shared staging and production hosts need a second explicit flag. The script asks for the host name before it starts unless `--yes` is given. A run above the spend threshold needs `--confirm-spend <max USD>`. - `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an in-process fake API with a virtual clock. They cover ramp logic, stop on failure, criteria evaluation, cleanup after a mid-run crash and an interrupt, `--cleanup-only` with and without a state file, dry run, the production-host guard, the spend gate, and that credentials never appear in output or files. - `scripts/scale-ramp/tsconfig.json` is added to the test programs that CI typechecks. The test file joins the `test:release-registry` node test list, which CI runs. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3 runbook with commands, cost, criteria, proof order, target prerequisites and open risks. ## Verification - `node scripts/ci-profile/node-test.mjs scripts/scale-ramp/scale-ramp.test.ts scripts/typecheck-test-programs.test.mjs`: 47 tests pass. - `node node_modules/typescript/bin/tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts --base-url https://scale1.test.paperclip.app --target-kind test-stack --steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate and the criteria, and makes no request. - The harness was not run against any real instance. That needs the owner's go-ahead. ## Risks - Low risk for the product: the change adds a script, tests and docs only. No server code changes. - A real run creates real agents, tasks and sandboxes and spends money. The guard, the typed confirmation, the spend gate, the task cap and the recorded-spend stop limit this. The runbook lists the open risks. - The cost defaults are assumptions until a first real run records actual spend. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d6acbf50e6
commit
328a954cc1
22 files changed
+2986
-2
No files matched your search
@@ -104,3 +104,4 @@ test-results/
|
||||
|
||||
# Content-addressed CI task results
|
||||
.turbo/
|
||||
scripts/scale-ramp/results/
|
||||
@@ -358,6 +358,139 @@ of other servers keep their records.
|
||||
3. `paperclip.staging` only after the owner's explicit go-ahead.
|
||||
4. Repeat the 1,000 step with real model calls for a short window.
|
||||
|
||||
### Phase 3 runbook: real-agent ramp
|
||||
|
||||
`scripts/scale-agents-ramp.ts` runs real agents on a hosted instance through
|
||||
the product API only. It ramps concurrency in steps, holds each step, stops at
|
||||
the first failed criterion, and always cleans up. It has run only against the
|
||||
in-process fake API of its tests (`scripts/scale-ramp/scale-ramp.test.ts`).
|
||||
A run on a real instance needs the owner's go-ahead.
|
||||
|
||||
**What one run does.** It creates a company named `Scale ramp <run id>`, or
|
||||
uses `--company <id>` (a Cloud-managed instance refuses company creation, so
|
||||
pass the company there). It stores the provider key as a company secret, or
|
||||
reuses `PAPERCLIP_SCALE_PROVIDER_SECRET_ID`. It creates one project, and one
|
||||
agent per concurrent slot (`maxConcurrentRuns` 1, heartbeat off, warm runner,
|
||||
the company's single sandbox environment). Each agent works one task at a
|
||||
time. When a task ends, the agent gets a new one, so the step keeps its
|
||||
concurrency. Each task is the same small job: stream a sentence, run `pwd`
|
||||
and `ls`, write `scale-<nonce>.txt` in the workspace, read it back, post a
|
||||
`CHECKPOINT` comment, post a `DONE` comment and mark the task done.
|
||||
|
||||
**Run it.**
|
||||
|
||||
```sh
|
||||
export PAPERCLIP_SCALE_TOKEN=<board token for the host> # never a flag
|
||||
export OPENROUTER_API_KEY=<key> # or PAPERCLIP_SCALE_PROVIDER_SECRET_ID
|
||||
H=<name>.test.paperclip.app
|
||||
S="node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts --base-url https://$H --target-kind test-stack"
|
||||
$S --steps 10,50 --hold-minutes 20 --dry-run # plan and estimate; no request
|
||||
$S --steps 10,50 --hold-minutes 20 --confirm-spend 15 # first real 50-agent run
|
||||
$S --cleanup-only <run id> # finish a cleanup after a crash
|
||||
```
|
||||
|
||||
The script asks you to type the host name before it creates anything, unless
|
||||
`--yes` is given. Optional environment: `PAPERCLIP_SCALE_COMPANY_ID`,
|
||||
`PAPERCLIP_SCALE_ENVIRONMENT_ID`, `PAPERCLIP_SCALE_PROVIDER`,
|
||||
`PAPERCLIP_SCALE_MODEL` (default `openrouter/deepseek/deepseek-v4-flash-0731`,
|
||||
the qualified OpenCode model that restart-chaos hosted runs also use),
|
||||
`PAPERCLIP_SCALE_ADAPTER_CONFIG` (JSON merged into the agent's adapter config,
|
||||
for example to use a scripted runner when one exists), and
|
||||
`PAPERCLIP_SCALE_METRICS_URL` with `PAPERCLIP_SCALE_METRICS_TOKEN`. Results go
|
||||
to `scripts/scale-ramp/results/<run id>/`: `result.json`, `SUMMARY.md` and
|
||||
`state.json`. Exit codes: 0 pass, 1 fail, 2 refused, 3 incomplete, 4 error or
|
||||
interrupted.
|
||||
|
||||
**Target guard.** `--target-kind` is required and must match the host:
|
||||
`local` (localhost), `test-stack` (`<name>.staging.paperclip.app` or
|
||||
`<name>.test.paperclip.app`, the restart-chaos hosted pattern),
|
||||
`shared-staging` (`paperclip.staging.paperclip.app`, which also needs
|
||||
`--allow-shared-staging`), or `production` (any other host, which also needs
|
||||
`--allow-production-host <the same host>`). Every request is checked against
|
||||
the one origin.
|
||||
|
||||
**Cost.** The script prints an estimate before it starts and refuses a run
|
||||
above `--spend-threshold-usd` (default $5) unless `--confirm-spend <max USD>`
|
||||
is at least the estimate. It never creates more tasks than the printed cap,
|
||||
and it stops when the model spend the runs record passes the confirmed
|
||||
amount. The defaults are assumptions: $0.02 model spend per task and $0.10
|
||||
per sandbox-hour (`PAPERCLIP_SCALE_MODEL_USD_PER_TASK`,
|
||||
`PAPERCLIP_SCALE_SANDBOX_USD_PER_HOUR`). Correct them from the recorded spend
|
||||
of the first run.
|
||||
|
||||
| Steps | Hold | Tasks (cap) | Estimate |
|
||||
| --- | --- | --- | --- |
|
||||
| 10, 50 | 20 min | 420 (525) | $13 |
|
||||
| 200 | 60 min | 4,000 (5,000) | $122 |
|
||||
| 1,000 | 60 min | 20,000 (25,000) | $610 |
|
||||
|
||||
**Pass.** The criteria are written into every result. A step passes when no
|
||||
measured criterion fails. The run passes when every step passes and every
|
||||
criterion was measured; it is `incomplete` when a criterion was not measured.
|
||||
A missing source is reported as "not measured", never as zero.
|
||||
|
||||
| Criterion | Source | Target | Measured from |
|
||||
| --- | --- | --- | --- |
|
||||
| Runs that need Retry | finish line | 0, stops at once | task view offers Retry |
|
||||
| Failed or timed-out runs | harness | 0, stops at once | run list |
|
||||
| Tasks at rest but not done | harness | 0 | task status |
|
||||
| Peak running agents | harness | at least 90% of the step | run list |
|
||||
| Board page API requests, p90 | finish line | below 500 ms | a GET mix every 10 s: dashboard, issues, agents, live runs, sidebar badges |
|
||||
| Board API failures | harness | 0 | the same requests |
|
||||
| Queue wait before start, p95 | harness | at most 60 s | run created to started |
|
||||
| First visible activity, p95 | harness | at most 60 s | run events of up to 50 runs per step |
|
||||
| Main-thread CPU or event-loop utilization | finish line | below 0.6 | metrics endpoint |
|
||||
| Run event lateness, p99 | finish line | below 1 s | metrics endpoint |
|
||||
| Agent API time before the handler, p99 | finish line | below 100 ms | metrics endpoint |
|
||||
| Lease renewals that miss their deadline | finish line | 0 | metrics endpoint |
|
||||
| Kill one server process | finish line | runs resume, no Retry | not measured by this harness |
|
||||
|
||||
The server has no metrics endpoint today. `PAPERCLIP_SCALE_METRICS_URL` must
|
||||
answer a JSON object with any of `mainThreadCpu`, `eventLoopUtilization`
|
||||
(0 to 1, highest server process), `eventLoopDelayP99Ms`,
|
||||
`statementsPerSecond`, `poolWaitP99Ms`, `runEventLatenessP99Ms`,
|
||||
`agentApiPreHandlerP99Ms` and `leaseRenewalsMissed` (a counter). Without it,
|
||||
the four server criteria are not measured, and a run can be at best
|
||||
`incomplete`. The harness does not read the database.
|
||||
|
||||
**Proof order.** A hosted stack that is not `paperclip.staging` first
|
||||
(`--target-kind test-stack`): 10 and 50, then 200, then 500 and 1,000, one
|
||||
hour per step from 200 on. `paperclip.staging` only after the owner's
|
||||
explicit go-ahead. Production last.
|
||||
|
||||
**Check the target first.**
|
||||
|
||||
- Sandbox quota and concurrency of the provider account for the step size
|
||||
(50, 200, 1,000 sandboxes at once), and its start rate.
|
||||
- Provider rate limits (OpenRouter requests and tokens per minute) for the
|
||||
step size.
|
||||
- Database connection budget: each server process opens its main,
|
||||
checkpoint and liveness pools plus dedicated connections (about 10 to 20
|
||||
at 1,000 runs, see change 1g), and a burst of starts can open more.
|
||||
- Company budget and hard stop: the company must allow the run's spend, or
|
||||
the budget stop pauses it part way.
|
||||
- Nothing else important runs on the instance; the run creates real tasks.
|
||||
|
||||
**Risks to know before approving a real run.**
|
||||
|
||||
- Sandbox cost and quota: the estimate counts sandbox-hours at an assumed
|
||||
rate. 200 agents for an hour is about 220 sandbox-hours, and 1,000 is about
|
||||
1,100.
|
||||
- Model spend: about $0.02 per task is an assumption; 1,000 agents create up
|
||||
to 25,000 tasks in an hour.
|
||||
- The run creates real companies, agents, tasks, comments, runs, cost rows
|
||||
and sandboxes on the instance. Cleanup archives the tasks, project and
|
||||
created company, deletes agents (an agent with cost history stays
|
||||
terminated) and the stored secret. It cannot remove run history or cost
|
||||
rows.
|
||||
- Cleanup cancels live runs, terminates agents and waits up to 5 minutes for
|
||||
agents to terminate and up to 5 minutes for the run's sandbox leases to be
|
||||
released. At 1,000 agents expect 10 to 20 minutes, mostly requests at 8 in
|
||||
parallel. If the harness dies, `--cleanup-only <run id>` finishes from
|
||||
`state.json`, or by name when the file is gone.
|
||||
- At 1,000 agents the harness itself adds load: one run list read, ten task
|
||||
reads and a board GET mix per poll (15 s), plus task creation.
|
||||
|
||||
## Decisions for the owner
|
||||
|
||||
| # | Decision | Recommendation |
|
||||
@@ -376,3 +509,6 @@ of other servers keep their records.
|
||||
- Staging's pool size and run-log storage mode were not read.
|
||||
- Tool-gateway calls, recovery sweeps, and browser fan-out are not in the
|
||||
model.
|
||||
- The real-agent ramp (`scripts/scale-agents-ramp.ts`) has run only against
|
||||
the fake API of its tests. Its cost defaults are assumptions, and the
|
||||
server criteria need a metrics endpoint that does not exist yet.
|
||||
+1
-1
@@ -61,7 +61,7 @@
|
||||
"smoke:notion-generic-live": "node scripts/smoke/notion-generic-live.mjs",
|
||||
"smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
|
||||
"smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
|
||||
"test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/ci/retry-reporter.test.mjs scripts/__tests__/runner-vitest-package-lanes.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/public-install-registry.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs scripts/__tests__/agent-build-cache.test.mjs scripts/__tests__/agent-preflight.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/draft-stable-notes.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs scripts/typecheck-test-programs.test.mjs scripts/__tests__/local-verify.test.mjs scripts/test-sandbox-providers.test.mjs scripts/__tests__/ci-typecheck-lane.test.mjs",
|
||||
"test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/ci/retry-reporter.test.mjs scripts/__tests__/runner-vitest-package-lanes.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/public-install-registry.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs scripts/__tests__/agent-build-cache.test.mjs scripts/__tests__/agent-preflight.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/draft-stable-notes.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs scripts/typecheck-test-programs.test.mjs scripts/__tests__/local-verify.test.mjs scripts/test-sandbox-providers.test.mjs scripts/__tests__/ci-typecheck-lane.test.mjs scripts/scale-ramp/scale-ramp.test.ts",
|
||||
"storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
|
||||
"test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
|
||||
"test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
import { main, processDeps } from "./scale-ramp/main.ts";
|
||||
|
||||
process.exitCode = await main(process.argv.slice(2), processDeps());
|
||||
@@ -0,0 +1,180 @@
|
||||
import { ApiError, type Clock, type PaperclipClient } from "./client.ts";
|
||||
import { agentPrefix, companyName, mapLimit, projectName, secretPrefix, type RunState } from "./state.ts";
|
||||
|
||||
type Json = Record<string, any>;
|
||||
|
||||
export interface CleanupReport {
|
||||
agentsPaused: number;
|
||||
runsCancelled: number;
|
||||
tasksArchived: number;
|
||||
agentsTerminated: number;
|
||||
agentsDeleted: number;
|
||||
agentsKeptForCostHistory: number;
|
||||
secretDeleted: boolean;
|
||||
projectArchived: boolean;
|
||||
companyArchived: boolean;
|
||||
activeSandboxLeases: number | null;
|
||||
durationMs: number;
|
||||
errors: string[];
|
||||
}
|
||||
|
||||
export const LIVE_RUN_STATUSES = new Set(["queued", "running", "scheduled_retry"]);
|
||||
const PARALLELISM = 8;
|
||||
|
||||
export interface CleanupInput {
|
||||
api: PaperclipClient;
|
||||
clock: Clock;
|
||||
label: string;
|
||||
state: RunState;
|
||||
terminateWaitMs?: number;
|
||||
leaseWaitMs?: number;
|
||||
pollMs?: number;
|
||||
log?: (line: string) => void;
|
||||
}
|
||||
|
||||
export async function discoverState(api: PaperclipClient, label: string, companyId: string | null, baseUrl: string, now: number): Promise<RunState> {
|
||||
let resolved = companyId;
|
||||
let created = false;
|
||||
if (!resolved) {
|
||||
const companies = await api.get<Json[]>("/api/companies");
|
||||
const match = companies.find(company => company.name === companyName(label));
|
||||
if (!match) throw new Error(`No state file for ${label} and no company named "${companyName(label)}"; pass --company <id>`);
|
||||
resolved = String(match.id);
|
||||
created = true;
|
||||
}
|
||||
return {
|
||||
schema: "paperclip.scale-ramp.state/v1", label, baseUrl, companyId: resolved, companyCreated: created, environmentId: null,
|
||||
projectId: null, secretId: null, agentIds: [], taskIds: [], createdAt: new Date(now).toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
export async function cleanupRun(input: CleanupInput): Promise<CleanupReport> {
|
||||
const { api, clock, label, state } = input;
|
||||
const started = clock.now();
|
||||
const pollMs = input.pollMs ?? 3_000;
|
||||
const report: CleanupReport = {
|
||||
agentsPaused: 0, runsCancelled: 0, tasksArchived: 0, agentsTerminated: 0, agentsDeleted: 0, agentsKeptForCostHistory: 0,
|
||||
secretDeleted: false, projectArchived: false, companyArchived: false, activeSandboxLeases: null, durationMs: 0, errors: [],
|
||||
};
|
||||
const attempt = async (what: string, action: () => Promise<unknown>): Promise<boolean> => {
|
||||
try {
|
||||
await action();
|
||||
return true;
|
||||
} catch (error) {
|
||||
report.errors.push(`${what}: ${error instanceof Error ? error.message : String(error)}`.slice(0, 300));
|
||||
return false;
|
||||
}
|
||||
};
|
||||
const read = async <T>(what: string, route: string, fallback: T): Promise<T> => {
|
||||
try {
|
||||
return await api.get<T>(route);
|
||||
} catch (error) {
|
||||
report.errors.push(`${what}: ${error instanceof Error ? error.message : String(error)}`.slice(0, 300));
|
||||
return fallback;
|
||||
}
|
||||
};
|
||||
const companyId = state.companyId;
|
||||
if (!companyId) {
|
||||
report.durationMs = clock.now() - started;
|
||||
return report;
|
||||
}
|
||||
const prefix = agentPrefix(label);
|
||||
const listed = await read<Json[]>("list agents", `/api/companies/${companyId}/agents`, []);
|
||||
const agents = listed.filter(agent => String(agent.name ?? "").startsWith(prefix));
|
||||
const agentIds = new Set(agents.map(agent => String(agent.id)));
|
||||
input.log?.(`Cleanup ${label}: ${agents.length} agents, ${state.taskIds.length} recorded tasks`);
|
||||
|
||||
await mapLimit(agents.filter(agent => agent.status !== "terminated" && agent.status !== "paused"), PARALLELISM, async agent => {
|
||||
if (await attempt(`pause agent ${agent.id}`, () => api.patch(`/api/agents/${agent.id}`, { status: "paused" }))) report.agentsPaused += 1;
|
||||
});
|
||||
|
||||
const runs = await read<Json[]>("list runs", `/api/companies/${companyId}/heartbeat-runs?summary=true&limit=1000`, []);
|
||||
const live = new Map<string, Json>();
|
||||
for (const run of runs) if (agentIds.has(String(run.agentId)) && LIVE_RUN_STATUSES.has(String(run.status))) live.set(String(run.id), run);
|
||||
if (runs.length >= 1_000) {
|
||||
await mapLimit([...agentIds], PARALLELISM, async agentId => {
|
||||
const own = await read<Json[]>(`list runs of ${agentId}`, `/api/companies/${companyId}/heartbeat-runs?agentId=${agentId}&summary=true&limit=50`, []);
|
||||
for (const run of own) if (LIVE_RUN_STATUSES.has(String(run.status))) live.set(String(run.id), run);
|
||||
});
|
||||
}
|
||||
await mapLimit([...live.keys()], PARALLELISM, async runId => {
|
||||
if (await attempt(`cancel run ${runId}`, () => api.post(`/api/heartbeat-runs/${runId}/cancel`))) report.runsCancelled += 1;
|
||||
});
|
||||
|
||||
const taskIds = new Set(state.taskIds);
|
||||
await mapLimit([...agentIds], PARALLELISM, async agentId => {
|
||||
const issues = await read<Json[]>(`list tasks of ${agentId}`, `/api/companies/${companyId}/issues?assigneeAgentId=${agentId}&limit=500`, []);
|
||||
for (const issue of issues) if (issue.assigneeAgentId === agentId && !issue.hiddenAt) taskIds.add(String(issue.id));
|
||||
});
|
||||
await mapLimit([...taskIds], PARALLELISM, async taskId => {
|
||||
const archived = await attempt(`archive task ${taskId}`, async () => {
|
||||
try {
|
||||
await api.patch(`/api/issues/${taskId}`, { hiddenAt: new Date(clock.now()).toISOString() });
|
||||
} catch (error) {
|
||||
if (!(error instanceof ApiError && error.status === 404)) throw error;
|
||||
}
|
||||
});
|
||||
if (archived) report.tasksArchived += 1;
|
||||
});
|
||||
|
||||
await mapLimit(agents.filter(agent => agent.status !== "terminated"), PARALLELISM, async agent => {
|
||||
if (await attempt(`terminate agent ${agent.id}`, () => api.post(`/api/agents/${agent.id}/terminate`))) report.agentsTerminated += 1;
|
||||
});
|
||||
const pending = new Set(agentIds);
|
||||
for (const until = clock.now() + (input.terminateWaitMs ?? 300_000); pending.size && clock.now() < until;) {
|
||||
const current = await read<Json[]>("list agents", `/api/companies/${companyId}/agents`, []);
|
||||
for (const agent of current) {
|
||||
if (!pending.has(String(agent.id))) continue;
|
||||
if (agent.status === "terminated" && (agent.lifecycleState == null || agent.lifecycleState === "terminated")) pending.delete(String(agent.id));
|
||||
}
|
||||
for (const id of [...pending]) if (!current.some(agent => String(agent.id) === id)) pending.delete(id);
|
||||
if (pending.size) await clock.sleep(pollMs);
|
||||
}
|
||||
if (pending.size) report.errors.push(`${pending.size} agents were not terminated in time; run --cleanup-only ${label} again`);
|
||||
await mapLimit([...agentIds].filter(id => !pending.has(id)), PARALLELISM, async agentId => {
|
||||
try {
|
||||
await api.delete(`/api/agents/${agentId}`);
|
||||
report.agentsDeleted += 1;
|
||||
} catch (error) {
|
||||
if (error instanceof ApiError && error.status === 409 && error.message.includes("agent_cost_history_retained")) report.agentsKeptForCostHistory += 1;
|
||||
else report.errors.push(`delete agent ${agentId}: ${error instanceof Error ? error.message : String(error)}`.slice(0, 300));
|
||||
}
|
||||
});
|
||||
|
||||
const secrets = await read<Json[]>("list secrets", `/api/companies/${companyId}/secrets`, []);
|
||||
for (const secret of secrets.filter(secret => String(secret.name ?? "").startsWith(secretPrefix(label)))) {
|
||||
if (await attempt(`delete secret ${secret.id}`, () => api.delete(`/api/secrets/${secret.id}`))) report.secretDeleted = true;
|
||||
}
|
||||
|
||||
const projects = await read<Json[]>("list projects", `/api/companies/${companyId}/projects`, []);
|
||||
for (const project of projects.filter(project => project.name === projectName(label) && !project.archivedAt)) {
|
||||
if (await attempt(`archive project ${project.id}`, () => api.patch(`/api/projects/${project.id}`, { archivedAt: new Date(clock.now()).toISOString() }))) {
|
||||
report.projectArchived = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (state.environmentId && taskIds.size) {
|
||||
for (const until = clock.now() + (input.leaseWaitMs ?? 300_000); ;) {
|
||||
let leases: Json[] | null = null;
|
||||
try {
|
||||
leases = await api.get<Json[]>(`/api/environments/${state.environmentId}/leases?status=active`);
|
||||
} catch {
|
||||
leases = null;
|
||||
}
|
||||
if (leases === null) break;
|
||||
report.activeSandboxLeases = leases.filter(lease => taskIds.has(String(lease.issueId))).length;
|
||||
if (report.activeSandboxLeases === 0 || clock.now() >= until) break;
|
||||
await clock.sleep(pollMs);
|
||||
}
|
||||
if (report.activeSandboxLeases) report.errors.push(`${report.activeSandboxLeases} sandbox leases of this run are still active`);
|
||||
}
|
||||
|
||||
if (state.companyCreated) {
|
||||
const company = await read<Json | null>("read company", `/api/companies/${companyId}`, null);
|
||||
if (company && company.name === companyName(label) && company.status !== "archived") {
|
||||
if (await attempt(`archive company ${companyId}`, () => api.post(`/api/companies/${companyId}/archive`))) report.companyArchived = true;
|
||||
}
|
||||
}
|
||||
report.durationMs = clock.now() - started;
|
||||
return report;
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
import { assertSameOrigin, type Target } from "./guard.ts";
|
||||
|
||||
export type Fetcher = (url: string, init: RequestInit) => Promise<Response>;
|
||||
|
||||
export interface Clock {
|
||||
now(): number;
|
||||
sleep(ms: number): Promise<void>;
|
||||
}
|
||||
|
||||
export const realClock: Clock = {
|
||||
now: () => Date.now(),
|
||||
sleep: ms => new Promise(resolve => setTimeout(resolve, ms)),
|
||||
};
|
||||
|
||||
export interface RequestRecord {
|
||||
at: number;
|
||||
method: string;
|
||||
path: string;
|
||||
status: number | null;
|
||||
ms: number;
|
||||
ok: boolean;
|
||||
kind: "board" | "harness";
|
||||
}
|
||||
|
||||
export class ApiError extends Error {
|
||||
readonly status: number | null;
|
||||
readonly code: string | null;
|
||||
|
||||
constructor(status: number | null, message: string, code: string | null = null) {
|
||||
super(message);
|
||||
this.name = "ApiError";
|
||||
this.status = status;
|
||||
this.code = code;
|
||||
}
|
||||
}
|
||||
|
||||
export class Redactor {
|
||||
private readonly values: string[];
|
||||
|
||||
constructor(values: ReadonlyArray<string | null | undefined>) {
|
||||
this.values = values.filter((value): value is string => typeof value === "string" && value.length >= 4)
|
||||
.sort((a, b) => b.length - a.length);
|
||||
}
|
||||
|
||||
text(input: string): string {
|
||||
let out = input;
|
||||
for (const value of this.values) out = out.split(value).join("[redacted]");
|
||||
return out.replace(/(bearer\s+)[A-Za-z0-9._~+/=-]{8,}/gi, "$1[redacted]");
|
||||
}
|
||||
|
||||
json<T>(value: T): T {
|
||||
return JSON.parse(this.text(JSON.stringify(value))) as T;
|
||||
}
|
||||
}
|
||||
|
||||
const UUID = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi;
|
||||
const TRANSIENT = new Set([502, 503, 504]);
|
||||
|
||||
export function pathClass(route: string): string {
|
||||
return route.split("?")[0]!.replace(UUID, ":id");
|
||||
}
|
||||
|
||||
export class PaperclipClient {
|
||||
readonly records: RequestRecord[] = [];
|
||||
private readonly target: Target;
|
||||
private readonly token: string;
|
||||
private readonly fetcher: Fetcher;
|
||||
private readonly clock: Clock;
|
||||
private readonly redactor: Redactor;
|
||||
private readonly timeoutMs: number;
|
||||
|
||||
constructor(target: Target, token: string, fetcher: Fetcher, clock: Clock, redactor: Redactor, timeoutMs = 30_000) {
|
||||
this.target = target;
|
||||
this.token = token;
|
||||
this.fetcher = fetcher;
|
||||
this.clock = clock;
|
||||
this.redactor = redactor;
|
||||
this.timeoutMs = timeoutMs;
|
||||
}
|
||||
|
||||
private async once(method: string, route: string, body: unknown, kind: RequestRecord["kind"]) {
|
||||
const url = `${this.target.baseUrl}${route}`;
|
||||
assertSameOrigin(this.target, url);
|
||||
const started = this.clock.now();
|
||||
try {
|
||||
const response = await this.fetcher(url, {
|
||||
method,
|
||||
cache: "no-store",
|
||||
signal: AbortSignal.timeout(this.timeoutMs),
|
||||
headers: {
|
||||
authorization: `Bearer ${this.token}`,
|
||||
accept: "application/json",
|
||||
origin: this.target.baseUrl,
|
||||
...(body === undefined ? {} : { "content-type": "application/json" }),
|
||||
},
|
||||
body: body === undefined ? undefined : JSON.stringify(body),
|
||||
});
|
||||
const text = await response.text();
|
||||
const ms = this.clock.now() - started;
|
||||
const answered = response.ok || (kind === "harness" && [400, 404, 409, 422].includes(response.status));
|
||||
this.records.push({ at: started, method, path: pathClass(route), status: response.status, ms, ok: answered, kind });
|
||||
return { status: response.status, ok: response.ok, text };
|
||||
} catch (error) {
|
||||
this.records.push({ at: started, method, path: pathClass(route), status: null, ms: this.clock.now() - started, ok: false, kind });
|
||||
return { status: null, ok: false, text: error instanceof Error ? error.message : String(error) };
|
||||
}
|
||||
}
|
||||
|
||||
async request<T>(method: string, route: string, body?: unknown, kind: RequestRecord["kind"] = "harness"): Promise<T> {
|
||||
for (let attempt = 1; ; attempt += 1) {
|
||||
const result = await this.once(method, route, body, kind);
|
||||
if (result.ok) return (result.text ? JSON.parse(result.text) : null) as T;
|
||||
const transient = result.status === null || TRANSIENT.has(result.status) || result.status === 429;
|
||||
if (!transient || (method !== "GET" && result.status !== 429) || attempt >= 3) {
|
||||
let code: string | null = null;
|
||||
try {
|
||||
const parsed = JSON.parse(result.text) as { code?: unknown; details?: { code?: unknown } };
|
||||
code = typeof parsed.code === "string" ? parsed.code : typeof parsed.details?.code === "string" ? parsed.details.code : null;
|
||||
} catch {
|
||||
code = null;
|
||||
}
|
||||
const detail = this.redactor.text(result.text.slice(0, 300));
|
||||
throw new ApiError(result.status, `${method} ${pathClass(route)} ${result.status === null ? "got no answer" : `returned ${result.status}`}: ${detail}`, code);
|
||||
}
|
||||
await this.clock.sleep(1_000 * attempt);
|
||||
}
|
||||
}
|
||||
|
||||
get<T>(route: string, kind: RequestRecord["kind"] = "harness") {
|
||||
return this.request<T>("GET", route, undefined, kind);
|
||||
}
|
||||
|
||||
post<T>(route: string, body: unknown = {}) {
|
||||
return this.request<T>("POST", route, body);
|
||||
}
|
||||
|
||||
patch<T>(route: string, body: unknown = {}) {
|
||||
return this.request<T>("PATCH", route, body);
|
||||
}
|
||||
|
||||
async delete(route: string, allowNotFound = true): Promise<void> {
|
||||
try {
|
||||
await this.request<unknown>("DELETE", route);
|
||||
} catch (error) {
|
||||
if (!(allowNotFound && error instanceof ApiError && error.status === 404)) throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function readMetrics(fetcher: Fetcher, url: string, token: string | null, clock: Clock): Promise<Record<string, unknown> | null> {
|
||||
try {
|
||||
const response = await fetcher(url, {
|
||||
method: "GET",
|
||||
cache: "no-store",
|
||||
signal: AbortSignal.timeout(10_000),
|
||||
headers: { accept: "application/json", ...(token ? { authorization: `Bearer ${token}` } : {}) },
|
||||
});
|
||||
if (!response.ok) return null;
|
||||
const body = await response.json() as unknown;
|
||||
return body && typeof body === "object" && !Array.isArray(body) ? { ...(body as Record<string, unknown>), sampledAt: clock.now() } : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
import path from "node:path";
|
||||
import { parseArgs } from "node:util";
|
||||
import { resolveTarget, type Target } from "./guard.ts";
|
||||
|
||||
export const DEFAULT_STEPS = [10, 50, 100, 200, 500, 1_000];
|
||||
export const QUALIFIED_OPENCODE_MODEL = "openrouter/deepseek/deepseek-v4-flash-0731";
|
||||
export const TOKEN_ENV = "PAPERCLIP_SCALE_TOKEN";
|
||||
export const FALLBACK_TOKEN_ENV = "PAPERCLIP_HOSTED_TOKEN";
|
||||
export const RESULTS_ROOT = path.join(import.meta.dirname, "results");
|
||||
|
||||
export type Mode = "run" | "dry-run" | "cleanup";
|
||||
|
||||
export interface Step {
|
||||
index: number;
|
||||
concurrency: number;
|
||||
holdMs: number;
|
||||
}
|
||||
|
||||
export interface Thresholds {
|
||||
boardP90Ms: number;
|
||||
queueWaitP95Ms: number;
|
||||
firstActivityP95Ms: number;
|
||||
minConcurrencyRatio: number;
|
||||
maxEventLoopUtilization: number;
|
||||
runEventLatenessP99Ms: number;
|
||||
agentApiPreHandlerP99Ms: number;
|
||||
}
|
||||
|
||||
export interface Pricing {
|
||||
modelUsdPerTask: number;
|
||||
sandboxUsdPerHour: number;
|
||||
}
|
||||
|
||||
export interface AgentProfile {
|
||||
provider: string;
|
||||
model: string;
|
||||
credentialKey: string | null;
|
||||
providerSecretId: string | null;
|
||||
adapterOverrides: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface Options {
|
||||
mode: Mode;
|
||||
target: Target;
|
||||
label: string;
|
||||
outDir: string;
|
||||
steps: Step[];
|
||||
drainMs: number;
|
||||
pollMs: number;
|
||||
boardProbeMs: number;
|
||||
taskMinutes: number;
|
||||
maxTasks: number | null;
|
||||
companyId: string | null;
|
||||
environmentId: string | null;
|
||||
yes: boolean;
|
||||
confirmSpendUsd: number | null;
|
||||
spendThresholdUsd: number;
|
||||
eventSamplePerStep: number;
|
||||
thresholds: Thresholds;
|
||||
pricing: Pricing;
|
||||
profile: AgentProfile;
|
||||
metricsUrl: string | null;
|
||||
}
|
||||
|
||||
export interface Secrets {
|
||||
token: string | null;
|
||||
credential: string | null;
|
||||
metricsToken: string | null;
|
||||
}
|
||||
|
||||
const LABEL = /^scale-[a-z0-9-]{3,40}$/;
|
||||
|
||||
function number(raw: string | undefined, name: string, fallback: number, min: number, max: number): number {
|
||||
if (raw === undefined || raw === "") return fallback;
|
||||
const value = Number(raw);
|
||||
if (!Number.isFinite(value) || value < min || value > max) throw new Error(`${name} must be a number from ${min} to ${max}`);
|
||||
return value;
|
||||
}
|
||||
|
||||
export function parseSteps(raw: string, holdMs: number): Step[] {
|
||||
const values = raw.split(",").map(part => part.trim()).filter(Boolean).map(part => {
|
||||
if (!/^[0-9]+$/.test(part)) throw new Error(`--steps takes whole numbers separated by commas, not "${part}"`);
|
||||
return Number(part);
|
||||
});
|
||||
if (!values.length) throw new Error("--steps needs at least one step");
|
||||
values.forEach((value, index) => {
|
||||
if (value < 1 || value > 5_000) throw new Error("each step must be from 1 to 5000 agents");
|
||||
if (index > 0 && value <= values[index - 1]!) throw new Error("--steps must increase");
|
||||
});
|
||||
return values.map((concurrency, index) => ({ index, concurrency, holdMs }));
|
||||
}
|
||||
|
||||
export function defaultLabel(now: number, random: () => number = Math.random): string {
|
||||
const stamp = new Date(now).toISOString().slice(0, 16).replace(/[-:T]/g, "").toLowerCase();
|
||||
return `scale-${stamp}-${Math.floor(random() * 0xffff).toString(16).padStart(4, "0")}`;
|
||||
}
|
||||
|
||||
export function readSecrets(env: NodeJS.ProcessEnv, credentialKey: string | null): Secrets {
|
||||
const pick = (value: string | undefined) => value?.trim() ? value.trim() : null;
|
||||
return {
|
||||
token: pick(env[TOKEN_ENV]) ?? pick(env[FALLBACK_TOKEN_ENV]),
|
||||
credential: credentialKey ? pick(env[credentialKey]) : null,
|
||||
metricsToken: pick(env.PAPERCLIP_SCALE_METRICS_TOKEN),
|
||||
};
|
||||
}
|
||||
|
||||
function adapterOverrides(raw: string | undefined): Record<string, unknown> {
|
||||
if (!raw) return {};
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(raw);
|
||||
} catch {
|
||||
throw new Error("PAPERCLIP_SCALE_ADAPTER_CONFIG must be a JSON object");
|
||||
}
|
||||
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) throw new Error("PAPERCLIP_SCALE_ADAPTER_CONFIG must be a JSON object");
|
||||
return parsed as Record<string, unknown>;
|
||||
}
|
||||
|
||||
export function parseOptions(argv: readonly string[], env: NodeJS.ProcessEnv, now: number, random?: () => number): Options {
|
||||
const { values } = parseArgs({
|
||||
args: argv[0] === "--" ? argv.slice(1) : [...argv],
|
||||
strict: true,
|
||||
options: {
|
||||
"base-url": { type: "string" },
|
||||
"target-kind": { type: "string" },
|
||||
"allow-shared-staging": { type: "boolean" },
|
||||
"allow-production-host": { type: "string" },
|
||||
steps: { type: "string" },
|
||||
"hold-minutes": { type: "string" },
|
||||
"drain-minutes": { type: "string" },
|
||||
"poll-seconds": { type: "string" },
|
||||
"board-probe-seconds": { type: "string" },
|
||||
"task-minutes": { type: "string" },
|
||||
"max-tasks": { type: "string" },
|
||||
company: { type: "string" },
|
||||
environment: { type: "string" },
|
||||
label: { type: "string" },
|
||||
out: { type: "string" },
|
||||
"dry-run": { type: "boolean" },
|
||||
"cleanup-only": { type: "string" },
|
||||
yes: { type: "boolean" },
|
||||
"confirm-spend": { type: "string" },
|
||||
"spend-threshold-usd": { type: "string" },
|
||||
"event-sample": { type: "string" },
|
||||
"max-queue-wait-p95-seconds": { type: "string" },
|
||||
"max-first-activity-p95-seconds": { type: "string" },
|
||||
"min-concurrency-ratio": { type: "string" },
|
||||
},
|
||||
});
|
||||
if (values["dry-run"] && values["cleanup-only"]) throw new Error("--dry-run and --cleanup-only exclude each other");
|
||||
const mode: Mode = values["cleanup-only"] ? "cleanup" : values["dry-run"] ? "dry-run" : "run";
|
||||
const target = resolveTarget({
|
||||
baseUrl: values["base-url"] ?? env.PAPERCLIP_SCALE_BASE_URL,
|
||||
kind: values["target-kind"],
|
||||
allowSharedStaging: values["allow-shared-staging"] === true,
|
||||
allowProductionHost: values["allow-production-host"],
|
||||
});
|
||||
const label = values["cleanup-only"] ?? values.label ?? defaultLabel(now, random);
|
||||
if (!LABEL.test(label)) throw new Error("The run id must look like scale-<3 to 40 lowercase letters, digits or dashes>");
|
||||
const holdMs = number(values["hold-minutes"], "--hold-minutes", 10, 1, 240) * 60_000;
|
||||
const credentialKey = env.PAPERCLIP_SCALE_CREDENTIAL_KEY === "" ? null : env.PAPERCLIP_SCALE_CREDENTIAL_KEY ?? "OPENROUTER_API_KEY";
|
||||
const confirmSpend = values["confirm-spend"];
|
||||
return {
|
||||
mode,
|
||||
target,
|
||||
label,
|
||||
outDir: path.resolve(values.out ?? env.PAPERCLIP_SCALE_RESULTS_DIR ?? RESULTS_ROOT, label),
|
||||
steps: parseSteps(values.steps ?? DEFAULT_STEPS.join(","), holdMs),
|
||||
drainMs: number(values["drain-minutes"], "--drain-minutes", 15, 0, 240) * 60_000,
|
||||
pollMs: number(values["poll-seconds"], "--poll-seconds", 15, 1, 300) * 1_000,
|
||||
boardProbeMs: number(values["board-probe-seconds"], "--board-probe-seconds", 10, 1, 300) * 1_000,
|
||||
taskMinutes: number(values["task-minutes"], "--task-minutes", 3, 0.5, 120),
|
||||
maxTasks: values["max-tasks"] === undefined ? null : number(values["max-tasks"], "--max-tasks", 0, 1, 1_000_000),
|
||||
companyId: values.company ?? env.PAPERCLIP_SCALE_COMPANY_ID ?? null,
|
||||
environmentId: values.environment ?? env.PAPERCLIP_SCALE_ENVIRONMENT_ID ?? null,
|
||||
yes: values.yes === true,
|
||||
confirmSpendUsd: confirmSpend === undefined ? null : number(confirmSpend, "--confirm-spend", 0, 0, 1_000_000),
|
||||
spendThresholdUsd: number(values["spend-threshold-usd"], "--spend-threshold-usd", 5, 0, 1_000_000),
|
||||
eventSamplePerStep: number(values["event-sample"], "--event-sample", 50, 0, 10_000),
|
||||
thresholds: {
|
||||
boardP90Ms: 500,
|
||||
queueWaitP95Ms: number(values["max-queue-wait-p95-seconds"], "--max-queue-wait-p95-seconds", 60, 1, 3_600) * 1_000,
|
||||
firstActivityP95Ms: number(values["max-first-activity-p95-seconds"], "--max-first-activity-p95-seconds", 60, 1, 3_600) * 1_000,
|
||||
minConcurrencyRatio: number(values["min-concurrency-ratio"], "--min-concurrency-ratio", 0.9, 0.1, 1),
|
||||
maxEventLoopUtilization: 0.6,
|
||||
runEventLatenessP99Ms: 1_000,
|
||||
agentApiPreHandlerP99Ms: 100,
|
||||
},
|
||||
pricing: {
|
||||
modelUsdPerTask: number(env.PAPERCLIP_SCALE_MODEL_USD_PER_TASK, "PAPERCLIP_SCALE_MODEL_USD_PER_TASK", 0.02, 0, 100),
|
||||
sandboxUsdPerHour: number(env.PAPERCLIP_SCALE_SANDBOX_USD_PER_HOUR, "PAPERCLIP_SCALE_SANDBOX_USD_PER_HOUR", 0.1, 0, 100),
|
||||
},
|
||||
profile: {
|
||||
provider: env.PAPERCLIP_SCALE_PROVIDER ?? "opencode",
|
||||
model: env.PAPERCLIP_SCALE_MODEL ?? QUALIFIED_OPENCODE_MODEL,
|
||||
credentialKey,
|
||||
providerSecretId: env.PAPERCLIP_SCALE_PROVIDER_SECRET_ID ?? null,
|
||||
adapterOverrides: adapterOverrides(env.PAPERCLIP_SCALE_ADAPTER_CONFIG),
|
||||
},
|
||||
metricsUrl: env.PAPERCLIP_SCALE_METRICS_URL ?? null,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
import type { Step, Thresholds } from "./config.ts";
|
||||
import type { Distribution } from "./stats.ts";
|
||||
|
||||
export interface ServerMetrics {
|
||||
mainThreadCpuMax: number | null;
|
||||
eventLoopUtilizationMax: number | null;
|
||||
eventLoopDelayP99MsMax: number | null;
|
||||
statementsPerSecondMax: number | null;
|
||||
poolWaitP99MsMax: number | null;
|
||||
runEventLatenessP99MsMax: number | null;
|
||||
agentApiPreHandlerP99MsMax: number | null;
|
||||
leaseRenewalsMissed: number | null;
|
||||
samples: number;
|
||||
}
|
||||
|
||||
export interface StepMetrics {
|
||||
step: Step;
|
||||
startedAt: number;
|
||||
endedAt: number | null;
|
||||
tasksCreated: number;
|
||||
tasksStarted: number;
|
||||
tasksFinished: number;
|
||||
tasksFailed: number;
|
||||
tasksNeedingRetry: number;
|
||||
runsFailed: number;
|
||||
runsInterrupted: number;
|
||||
runsCancelled: number;
|
||||
peakRunning: number;
|
||||
meanRunning: number | null;
|
||||
queueWaitMs: Distribution;
|
||||
taskToStartMs: Distribution;
|
||||
ttftMs: Distribution;
|
||||
firstActivityMs: Distribution;
|
||||
runDurationMs: Distribution;
|
||||
board: { requests: number; failures: number; ms: Distribution };
|
||||
server: ServerMetrics;
|
||||
recordedSpendUsd: number;
|
||||
}
|
||||
|
||||
export type CriterionStatus = "pass" | "fail" | "not_measured";
|
||||
|
||||
export interface CriterionResult {
|
||||
id: string;
|
||||
source: "finish-line" | "harness";
|
||||
description: string;
|
||||
threshold: string;
|
||||
status: CriterionStatus;
|
||||
observed: number | string | null;
|
||||
}
|
||||
|
||||
interface Criterion {
|
||||
id: string;
|
||||
source: CriterionResult["source"];
|
||||
description: string;
|
||||
immediate: boolean;
|
||||
threshold: (t: Thresholds) => string;
|
||||
evaluate: (m: StepMetrics, t: Thresholds) => { status: CriterionStatus; observed: number | string | null };
|
||||
}
|
||||
|
||||
const below = (value: number | null, limit: number, strict = true) =>
|
||||
value === null ? { status: "not_measured" as const, observed: null }
|
||||
: { status: (strict ? value < limit : value <= limit) ? "pass" as const : "fail" as const, observed: value };
|
||||
|
||||
export const CRITERIA: readonly Criterion[] = [
|
||||
{
|
||||
id: "retry_needed_zero", source: "finish-line", immediate: true,
|
||||
description: "Runs that need Retry (a task offered the Retry action)",
|
||||
threshold: () => "= 0",
|
||||
evaluate: m => ({ status: m.tasksNeedingRetry === 0 ? "pass" : "fail", observed: m.tasksNeedingRetry }),
|
||||
},
|
||||
{
|
||||
id: "failed_runs_zero", source: "harness", immediate: true,
|
||||
description: "Runs that ended failed or timed out",
|
||||
threshold: () => "= 0",
|
||||
evaluate: m => ({ status: m.runsFailed === 0 ? "pass" : "fail", observed: m.runsFailed }),
|
||||
},
|
||||
{
|
||||
id: "tasks_rested_not_done_zero", source: "harness", immediate: false,
|
||||
description: "Tasks that came to rest without being done (blocked, in review or cancelled)",
|
||||
threshold: () => "= 0",
|
||||
evaluate: m => ({ status: m.tasksFailed === 0 ? "pass" : "fail", observed: m.tasksFailed }),
|
||||
},
|
||||
{
|
||||
id: "concurrency_reached", source: "harness", immediate: false,
|
||||
description: "Peak running agents against the step target",
|
||||
threshold: t => `>= ${Math.round(t.minConcurrencyRatio * 100)}% of the step`,
|
||||
evaluate: (m, t) => ({ status: m.peakRunning >= Math.ceil(m.step.concurrency * t.minConcurrencyRatio) ? "pass" : "fail", observed: m.peakRunning }),
|
||||
},
|
||||
{
|
||||
id: "board_api_p90", source: "finish-line", immediate: false,
|
||||
description: "Board page API requests, p90 (ms)",
|
||||
threshold: t => `< ${t.boardP90Ms} ms`,
|
||||
evaluate: (m, t) => m.board.ms.n < 3 ? { status: "not_measured", observed: null } : below(m.board.ms.p90, t.boardP90Ms),
|
||||
},
|
||||
{
|
||||
id: "board_api_errors_zero", source: "harness", immediate: false,
|
||||
description: "Board API requests that failed (no answer, 4xx or 5xx)",
|
||||
threshold: () => "= 0",
|
||||
evaluate: m => m.board.requests === 0 ? { status: "not_measured", observed: null } : { status: m.board.failures === 0 ? "pass" : "fail", observed: m.board.failures },
|
||||
},
|
||||
{
|
||||
id: "queue_wait_p95", source: "harness", immediate: false,
|
||||
description: "Queue wait before a run starts, p95 (ms)",
|
||||
threshold: t => `<= ${t.queueWaitP95Ms} ms`,
|
||||
evaluate: (m, t) => below(m.queueWaitMs.p95, t.queueWaitP95Ms, false),
|
||||
},
|
||||
{
|
||||
id: "first_activity_p95", source: "harness", immediate: false,
|
||||
description: "First visible agent activity from run creation, p95 (ms), from sampled run events",
|
||||
threshold: t => `<= ${t.firstActivityP95Ms} ms`,
|
||||
evaluate: (m, t) => below(m.firstActivityMs.p95, t.firstActivityP95Ms, false),
|
||||
},
|
||||
{
|
||||
id: "main_thread_cpu", source: "finish-line", immediate: false,
|
||||
description: "Main-thread CPU (or event-loop utilization) of every server process, max",
|
||||
threshold: t => `< ${t.maxEventLoopUtilization}`,
|
||||
evaluate: (m, t) => below(m.server.mainThreadCpuMax ?? m.server.eventLoopUtilizationMax, t.maxEventLoopUtilization),
|
||||
},
|
||||
{
|
||||
id: "run_event_lateness_p99", source: "finish-line", immediate: false,
|
||||
description: "Run event lateness (runner emits to row committed), p99 (ms)",
|
||||
threshold: t => `< ${t.runEventLatenessP99Ms} ms`,
|
||||
evaluate: (m, t) => below(m.server.runEventLatenessP99MsMax, t.runEventLatenessP99Ms),
|
||||
},
|
||||
{
|
||||
id: "agent_api_pre_handler_p99", source: "finish-line", immediate: false,
|
||||
description: "Agent API request time before the handler, p99 (ms)",
|
||||
threshold: t => `< ${t.agentApiPreHandlerP99Ms} ms`,
|
||||
evaluate: (m, t) => below(m.server.agentApiPreHandlerP99MsMax, t.agentApiPreHandlerP99Ms),
|
||||
},
|
||||
{
|
||||
id: "lease_renewals_missed_zero", source: "finish-line", immediate: false,
|
||||
description: "Lease renewals that missed their deadline",
|
||||
threshold: () => "= 0",
|
||||
evaluate: m => m.server.leaseRenewalsMissed === null ? { status: "not_measured", observed: null }
|
||||
: { status: m.server.leaseRenewalsMissed === 0 ? "pass" : "fail", observed: m.server.leaseRenewalsMissed },
|
||||
},
|
||||
{
|
||||
id: "kill_one_process_resumes", source: "finish-line", immediate: false,
|
||||
description: "Killing one server process: its runs resume on another, no Retry",
|
||||
threshold: () => "resume, no Retry",
|
||||
evaluate: () => ({ status: "not_measured", observed: "this harness does not kill server processes" }),
|
||||
},
|
||||
];
|
||||
|
||||
export function evaluate(metrics: StepMetrics, thresholds: Thresholds, onlyImmediate = false): CriterionResult[] {
|
||||
return CRITERIA.filter(criterion => !onlyImmediate || criterion.immediate).map(criterion => ({
|
||||
id: criterion.id,
|
||||
source: criterion.source,
|
||||
description: criterion.description,
|
||||
threshold: criterion.threshold(thresholds),
|
||||
...criterion.evaluate(metrics, thresholds),
|
||||
}));
|
||||
}
|
||||
|
||||
export function criteriaTable(thresholds: Thresholds): Array<Pick<CriterionResult, "id" | "source" | "description" | "threshold">> {
|
||||
return CRITERIA.map(criterion => ({ id: criterion.id, source: criterion.source, description: criterion.description, threshold: criterion.threshold(thresholds) }));
|
||||
}
|
||||
|
||||
export const firstFailure = (results: readonly CriterionResult[]) => results.find(result => result.status === "fail") ?? null;
|
||||
@@ -0,0 +1,56 @@
|
||||
import type { Options } from "./config.ts";
|
||||
|
||||
export interface SpendEstimate {
|
||||
agents: number;
|
||||
tasks: number;
|
||||
maxTasks: number;
|
||||
sandboxHours: number;
|
||||
modelUsd: number;
|
||||
sandboxUsd: number;
|
||||
totalUsd: number;
|
||||
wallMinutes: number;
|
||||
assumptions: string[];
|
||||
}
|
||||
|
||||
const round = (value: number, places = 2) => Math.round(value * 10 ** places) / 10 ** places;
|
||||
|
||||
export function estimateSpend(options: Pick<Options, "steps" | "drainMs" | "taskMinutes" | "maxTasks" | "pricing">): SpendEstimate {
|
||||
const agents = Math.max(...options.steps.map(step => step.concurrency));
|
||||
const holdMinutes = (step: { holdMs: number }) => step.holdMs / 60_000;
|
||||
const tasks = options.steps.reduce((sum, step) => sum + step.concurrency * Math.ceil(holdMinutes(step) / options.taskMinutes), 0);
|
||||
const maxTasks = options.maxTasks ?? Math.ceil(tasks * 1.25);
|
||||
const drainMinutes = options.drainMs / 60_000;
|
||||
const sandboxHours = options.steps.reduce((sum, step) => sum + step.concurrency * holdMinutes(step) / 60, 0)
|
||||
+ agents * Math.min(drainMinutes, options.taskMinutes * 2) / 60;
|
||||
const modelUsd = maxTasks * options.pricing.modelUsdPerTask;
|
||||
const sandboxUsd = sandboxHours * options.pricing.sandboxUsdPerHour;
|
||||
return {
|
||||
agents,
|
||||
tasks,
|
||||
maxTasks,
|
||||
sandboxHours: round(sandboxHours, 1),
|
||||
modelUsd: round(modelUsd),
|
||||
sandboxUsd: round(sandboxUsd),
|
||||
totalUsd: round(modelUsd + sandboxUsd),
|
||||
wallMinutes: Math.round(options.steps.reduce((sum, step) => sum + holdMinutes(step), 0) + drainMinutes),
|
||||
assumptions: [
|
||||
`each agent finishes one task every ${options.taskMinutes} min, so a step creates concurrency x ceil(hold / task minutes) tasks`,
|
||||
`model spend is the task cap (${maxTasks}) x $${options.pricing.modelUsdPerTask} per task (PAPERCLIP_SCALE_MODEL_USD_PER_TASK)`,
|
||||
`sandbox spend is agent-hours of holds plus the drain x $${options.pricing.sandboxUsdPerHour} per sandbox-hour (PAPERCLIP_SCALE_SANDBOX_USD_PER_HOUR)`,
|
||||
"these are estimates; the result reports the model spend the runs recorded",
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
export type SpendDecision = { ok: true } | { ok: false; reason: string };
|
||||
|
||||
export function checkSpend(estimate: SpendEstimate, thresholdUsd: number, confirmedUsd: number | null): SpendDecision {
|
||||
if (estimate.totalUsd <= thresholdUsd) return { ok: true };
|
||||
if (confirmedUsd === null) {
|
||||
return { ok: false, reason: `Estimated spend $${estimate.totalUsd} is above $${thresholdUsd}; rerun with --confirm-spend <max USD> of at least ${Math.ceil(estimate.totalUsd)}` };
|
||||
}
|
||||
if (confirmedUsd < estimate.totalUsd) {
|
||||
return { ok: false, reason: `--confirm-spend ${confirmedUsd} is below the estimated $${estimate.totalUsd}` };
|
||||
}
|
||||
return { ok: true };
|
||||
}
|
||||
@@ -0,0 +1,308 @@
|
||||
import type { Clock, Fetcher } from "./client.ts";
|
||||
|
||||
type Json = Record<string, any>;
|
||||
|
||||
export class VirtualClock implements Clock {
|
||||
t = Date.parse("2026-11-14T10:00:00.000Z");
|
||||
|
||||
now(): number {
|
||||
return this.t;
|
||||
}
|
||||
|
||||
async sleep(ms: number): Promise<void> {
|
||||
this.t += ms;
|
||||
await Promise.resolve();
|
||||
}
|
||||
}
|
||||
|
||||
export interface FakeConfig {
|
||||
token: string;
|
||||
origin: string;
|
||||
cloudManaged: boolean;
|
||||
startDelayMs: number;
|
||||
runDurationMs: number;
|
||||
capacity: number;
|
||||
failWhenRunningAbove: number;
|
||||
retryOnFailure: boolean;
|
||||
costPerRunUsd: number;
|
||||
boardLatencyMs: (running: number) => number;
|
||||
failRequest: (method: string, path: string, count: number) => { status: number; body: string } | null;
|
||||
metricsUrl: string | null;
|
||||
metrics: (running: number) => Record<string, unknown> | null;
|
||||
}
|
||||
|
||||
interface FakeRun {
|
||||
id: string;
|
||||
agentId: string;
|
||||
issueId: string;
|
||||
status: string;
|
||||
createdAt: number;
|
||||
startedAt: number | null;
|
||||
finishedAt: number | null;
|
||||
willFail: boolean;
|
||||
}
|
||||
|
||||
export interface RequestLog {
|
||||
method: string;
|
||||
url: string;
|
||||
authorization: string | null;
|
||||
body: string | null;
|
||||
}
|
||||
|
||||
const json = (status: number, body: unknown) => new Response(JSON.stringify(body), { status, headers: { "content-type": "application/json" } });
|
||||
|
||||
export class FakePaperclip {
|
||||
readonly config: FakeConfig;
|
||||
readonly requests: RequestLog[] = [];
|
||||
readonly companies = new Map<string, Json>();
|
||||
readonly agents = new Map<string, Json>();
|
||||
readonly issues = new Map<string, Json>();
|
||||
readonly runs = new Map<string, FakeRun>();
|
||||
readonly secrets = new Map<string, Json>();
|
||||
readonly projects = new Map<string, Json>();
|
||||
readonly memberCompanyIds: string[] = [];
|
||||
readonly environmentId = "00000000-0000-4000-8000-0000000000e1";
|
||||
private seq = 0;
|
||||
private counts = new Map<string, number>();
|
||||
|
||||
readonly clock: VirtualClock;
|
||||
|
||||
constructor(clock: VirtualClock, config: Partial<FakeConfig> = {}) {
|
||||
this.clock = clock;
|
||||
this.config = {
|
||||
token: "board-token-value",
|
||||
origin: "https://scale1.test.paperclip.app",
|
||||
cloudManaged: false,
|
||||
startDelayMs: 2_000,
|
||||
runDurationMs: 60_000,
|
||||
capacity: Number.POSITIVE_INFINITY,
|
||||
failWhenRunningAbove: Number.POSITIVE_INFINITY,
|
||||
retryOnFailure: false,
|
||||
costPerRunUsd: 0.01,
|
||||
boardLatencyMs: () => 40,
|
||||
failRequest: () => null,
|
||||
metricsUrl: null,
|
||||
metrics: () => null,
|
||||
...config,
|
||||
};
|
||||
}
|
||||
|
||||
private id(): string {
|
||||
this.seq += 1;
|
||||
return `00000000-0000-4000-8000-${this.seq.toString(16).padStart(12, "0")}`;
|
||||
}
|
||||
|
||||
addMemberCompany(name: string): string {
|
||||
const id = this.id();
|
||||
this.companies.set(id, { id, name, status: "active" });
|
||||
this.memberCompanyIds.push(id);
|
||||
return id;
|
||||
}
|
||||
|
||||
addNoiseRuns(count: number): void {
|
||||
for (let index = 0; index < count; index += 1) {
|
||||
const id = this.id();
|
||||
this.runs.set(id, { id, agentId: "noise-agent", issueId: "noise-issue", status: "succeeded", createdAt: this.clock.now() + 1e10 + index,
|
||||
startedAt: null, finishedAt: null, willFail: false });
|
||||
}
|
||||
}
|
||||
|
||||
running(): number {
|
||||
return [...this.runs.values()].filter(run => run.status === "running").length;
|
||||
}
|
||||
|
||||
liveRuns(): FakeRun[] {
|
||||
return [...this.runs.values()].filter(run => ["queued", "running"].includes(run.status));
|
||||
}
|
||||
|
||||
advance(): void {
|
||||
const now = this.clock.now();
|
||||
for (const run of [...this.runs.values()].sort((a, b) => a.createdAt - b.createdAt)) {
|
||||
if (run.status === "running" && run.startedAt !== null && now >= run.startedAt + this.config.runDurationMs) {
|
||||
run.finishedAt = run.startedAt + this.config.runDurationMs;
|
||||
const issue = this.issues.get(run.issueId)!;
|
||||
if (run.willFail) {
|
||||
run.status = "failed";
|
||||
issue.interruption = this.config.retryOnFailure ? { state: "failed", canRetry: true, retryWait: false } : null;
|
||||
} else {
|
||||
run.status = "succeeded";
|
||||
issue.status = "done";
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const run of [...this.runs.values()].sort((a, b) => a.createdAt - b.createdAt)) {
|
||||
if (run.status !== "queued" || now < run.createdAt + this.config.startDelayMs) continue;
|
||||
if (this.running() >= this.config.capacity) break;
|
||||
run.status = "running";
|
||||
run.startedAt = Math.max(run.createdAt + this.config.startDelayMs, now - 1_000);
|
||||
run.willFail = this.running() > this.config.failWhenRunningAbove;
|
||||
this.issues.get(run.issueId)!.status = "in_progress";
|
||||
}
|
||||
}
|
||||
|
||||
private runRow(run: FakeRun): Json {
|
||||
const iso = (value: number | null) => value === null ? null : new Date(value).toISOString();
|
||||
return {
|
||||
id: run.id, agentId: run.agentId, issueId: run.issueId, status: run.status, createdAt: iso(run.createdAt), startedAt: iso(run.startedAt),
|
||||
finishedAt: iso(run.finishedAt), costUsd: run.status === "succeeded" || run.status === "failed" ? this.config.costPerRunUsd : null,
|
||||
};
|
||||
}
|
||||
|
||||
readonly fetch: Fetcher = async (url, init) => {
|
||||
const method = String(init.method ?? "GET");
|
||||
const headers = (init.headers ?? {}) as Record<string, string>;
|
||||
const body = typeof init.body === "string" ? init.body : null;
|
||||
this.requests.push({ method, url, authorization: headers.authorization ?? null, body });
|
||||
if (this.config.metricsUrl && url === this.config.metricsUrl) {
|
||||
this.advance();
|
||||
const sample = this.config.metrics(this.running());
|
||||
return sample ? json(200, sample) : json(503, { error: "unavailable" });
|
||||
}
|
||||
const parsed = new URL(url);
|
||||
if (parsed.origin !== this.config.origin) throw new Error(`fake: request outside ${this.config.origin}: ${parsed.origin}`);
|
||||
if (headers.authorization !== `Bearer ${this.config.token}`) return json(401, { error: "unauthorized" });
|
||||
const route = parsed.pathname;
|
||||
const key = `${method} ${route.replace(/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/g, ":id")}`;
|
||||
const count = (this.counts.get(key) ?? 0) + 1;
|
||||
this.counts.set(key, count);
|
||||
const injected = this.config.failRequest(method, key.slice(method.length + 1), count);
|
||||
if (injected) return new Response(injected.body, { status: injected.status });
|
||||
this.advance();
|
||||
return this.route(method, route, parsed.searchParams, body ? JSON.parse(body) as Json : {});
|
||||
};
|
||||
|
||||
private route(method: string, route: string, query: URLSearchParams, body: Json): Response {
|
||||
const parts = route.split("/").filter(Boolean).slice(1);
|
||||
const now = this.clock.now();
|
||||
const [a, b, c] = parts;
|
||||
if (method === "GET" && a === "cli-auth" && b === "me") return json(200, { companyIds: this.memberCompanyIds });
|
||||
if (a === "companies" && !b) {
|
||||
if (method === "GET") return json(200, [...this.companies.values()]);
|
||||
if (this.config.cloudManaged) return json(403, { error: "Company creation is managed by Paperclip Cloud", code: "cloud_managed" });
|
||||
const id = this.id();
|
||||
this.companies.set(id, { id, name: body.name, status: "active" });
|
||||
this.memberCompanyIds.push(id);
|
||||
return json(201, this.companies.get(id));
|
||||
}
|
||||
if (a === "companies" && b) {
|
||||
const companyId = b;
|
||||
if (!this.companies.has(companyId)) return json(404, { error: "Company not found" });
|
||||
if (!c && method === "GET") return json(200, this.companies.get(companyId));
|
||||
if (c === "archive" && method === "POST") {
|
||||
this.companies.get(companyId)!.status = "archived";
|
||||
return json(200, this.companies.get(companyId));
|
||||
}
|
||||
if (c === "environments") return json(200, [{ id: this.environmentId, driver: "sandbox", status: "active" }]);
|
||||
if (c === "secrets") {
|
||||
if (method === "GET") return json(200, [...this.secrets.values()].filter(secret => secret.companyId === companyId).map(({ value: _value, ...rest }) => rest));
|
||||
const id = this.id();
|
||||
this.secrets.set(id, { id, companyId, name: body.name, value: body.value });
|
||||
return json(201, { id, name: body.name });
|
||||
}
|
||||
if (c === "projects") {
|
||||
if (method === "GET") return json(200, [...this.projects.values()].filter(project => project.companyId === companyId));
|
||||
const id = this.id();
|
||||
this.projects.set(id, { id, companyId, name: body.name, archivedAt: null });
|
||||
return json(201, this.projects.get(id));
|
||||
}
|
||||
if (c === "agents") {
|
||||
if (method === "GET") return json(200, [...this.agents.values()].filter(agent => agent.companyId === companyId));
|
||||
const id = this.id();
|
||||
this.agents.set(id, { id, companyId, name: body.name, status: "idle", lifecycleState: null, payload: body });
|
||||
return json(201, this.agents.get(id));
|
||||
}
|
||||
if (c === "issues") {
|
||||
if (method === "POST") {
|
||||
const id = this.id();
|
||||
this.issues.set(id, { id, companyId, title: body.title, description: body.description, status: "todo", assigneeAgentId: body.assigneeAgentId,
|
||||
projectId: body.projectId ?? null, hiddenAt: null, interruption: null });
|
||||
const agent = this.agents.get(body.assigneeAgentId);
|
||||
if (agent && agent.status !== "paused" && agent.status !== "terminated") {
|
||||
const runId = this.id();
|
||||
this.runs.set(runId, { id: runId, agentId: body.assigneeAgentId, issueId: id, status: "queued", createdAt: now, startedAt: null, finishedAt: null, willFail: false });
|
||||
}
|
||||
return json(201, this.issues.get(id));
|
||||
}
|
||||
let list = [...this.issues.values()].filter(issue => issue.companyId === companyId);
|
||||
const ids = query.get("ids");
|
||||
if (ids) list = list.filter(issue => ids.split(",").includes(issue.id));
|
||||
const assignee = query.get("assigneeAgentId");
|
||||
if (assignee) list = list.filter(issue => issue.assigneeAgentId === assignee);
|
||||
return json(200, list.slice(0, Number(query.get("limit") ?? 500)));
|
||||
}
|
||||
if (c === "heartbeat-runs") {
|
||||
let list = [...this.runs.values()].sort((x, y) => y.createdAt - x.createdAt);
|
||||
const agentId = query.get("agentId");
|
||||
if (agentId) list = list.filter(run => run.agentId === agentId);
|
||||
return json(200, list.slice(0, Math.min(1_000, Number(query.get("limit") ?? 200))).map(run => this.runRow(run)));
|
||||
}
|
||||
if (["dashboard", "live-runs", "sidebar-badges"].includes(c ?? "")) {
|
||||
this.clock.t += this.config.boardLatencyMs(this.running());
|
||||
return json(200, c === "live-runs" ? this.liveRuns().map(run => this.runRow(run)) : {});
|
||||
}
|
||||
}
|
||||
if (a === "agents" && b) {
|
||||
const agent = this.agents.get(b);
|
||||
if (!agent) return json(404, { error: "Agent not found" });
|
||||
if (method === "PATCH") {
|
||||
Object.assign(agent, body);
|
||||
return json(200, agent);
|
||||
}
|
||||
if (method === "POST" && c === "terminate") {
|
||||
agent.status = "terminated";
|
||||
agent.lifecycleState = "terminated";
|
||||
return json(200, agent);
|
||||
}
|
||||
if (method === "DELETE") {
|
||||
this.agents.delete(b);
|
||||
return json(200, { ok: true });
|
||||
}
|
||||
}
|
||||
if (a === "issues" && b) {
|
||||
const issue = this.issues.get(b);
|
||||
if (!issue) return json(404, { error: "Issue not found" });
|
||||
if (method === "GET" && !c) return json(200, issue);
|
||||
if (method === "GET" && c === "runs") return json(200, [...this.runs.values()].filter(run => run.issueId === b).map(run => ({ ...this.runRow(run), runId: run.id })));
|
||||
if (method === "PATCH") {
|
||||
Object.assign(issue, body);
|
||||
return json(200, issue);
|
||||
}
|
||||
}
|
||||
if (a === "heartbeat-runs" && b) {
|
||||
const run = this.runs.get(b);
|
||||
if (!run) return json(404, { error: "Run not found" });
|
||||
if (method === "POST" && c === "cancel") {
|
||||
if (["queued", "running"].includes(run.status)) {
|
||||
run.status = "cancelled";
|
||||
run.finishedAt = now;
|
||||
}
|
||||
return json(200, this.runRow(run));
|
||||
}
|
||||
if (method === "GET" && c === "events") {
|
||||
if (Number(query.get("afterSeq") ?? 0) > 0 || run.startedAt === null) return json(200, []);
|
||||
const delay = run.startedAt - run.createdAt;
|
||||
return json(200, [
|
||||
{ seq: 1, eventType: "run.performance.span", createdAt: new Date(run.startedAt).toISOString(), payload: { span: "heartbeat.queue", startOffsetMs: 0, durationMs: delay } },
|
||||
{ seq: 2, eventType: "run.performance.span", createdAt: new Date(run.startedAt + 3_000).toISOString(),
|
||||
payload: { span: "task.time_to_first_agent_activity", startOffsetMs: 0, durationMs: delay + 3_000 } },
|
||||
{ seq: 3, eventType: "run.performance.span", createdAt: new Date(run.startedAt + 5_000).toISOString(),
|
||||
payload: { span: "task.time_to_first_token", startOffsetMs: 0, durationMs: delay + 5_000 } },
|
||||
]);
|
||||
}
|
||||
}
|
||||
if (a === "secrets" && b && method === "DELETE") {
|
||||
this.secrets.delete(b);
|
||||
return json(200, { ok: true });
|
||||
}
|
||||
if (a === "projects" && b && method === "PATCH") {
|
||||
const project = this.projects.get(b);
|
||||
if (!project) return json(404, { error: "Project not found" });
|
||||
Object.assign(project, body);
|
||||
return json(200, project);
|
||||
}
|
||||
if (a === "environments" && b === this.environmentId && c === "leases") {
|
||||
return json(200, this.liveRuns().map(run => ({ id: run.id, issueId: run.issueId, status: "active" })));
|
||||
}
|
||||
return json(404, { error: `fake: no route for ${method} ${route}` });
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
import { REFUSED_HOSTS, TEST_HOST_PATTERN } from "../../tests/restart-chaos/hosted-guard.ts";
|
||||
|
||||
export const TARGET_KINDS = ["local", "test-stack", "shared-staging", "production"] as const;
|
||||
export type TargetKind = (typeof TARGET_KINDS)[number];
|
||||
|
||||
export interface Target {
|
||||
baseUrl: string;
|
||||
host: string;
|
||||
kind: TargetKind;
|
||||
}
|
||||
|
||||
export class TargetRefused extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = "TargetRefused";
|
||||
}
|
||||
}
|
||||
|
||||
const LOCAL_HOSTS = new Set(["localhost", "127.0.0.1", "[::1]", "::1"]);
|
||||
const RESERVED_LABELS = new Set(["admin", "admin-staging", "my", "my-staging", "api", "cloud", "www", "app"]);
|
||||
|
||||
export function classifyHost(host: string): TargetKind {
|
||||
const lower = host.toLowerCase();
|
||||
if (LOCAL_HOSTS.has(lower) || lower.endsWith(".localhost")) return "local";
|
||||
if (REFUSED_HOSTS.has(lower)) return "shared-staging";
|
||||
if (TEST_HOST_PATTERN.test(lower) && !RESERVED_LABELS.has(lower.split(".")[0]!)) return "test-stack";
|
||||
return "production";
|
||||
}
|
||||
|
||||
export interface TargetInput {
|
||||
baseUrl: string | undefined;
|
||||
kind: string | undefined;
|
||||
allowSharedStaging: boolean;
|
||||
allowProductionHost: string | undefined;
|
||||
}
|
||||
|
||||
export function resolveTarget(input: TargetInput): Target {
|
||||
if (!input.baseUrl) throw new TargetRefused("Set the target with --base-url <origin> or PAPERCLIP_SCALE_BASE_URL");
|
||||
if (!input.kind) throw new TargetRefused(`--target-kind is required: one of ${TARGET_KINDS.join(", ")}`);
|
||||
if (!(TARGET_KINDS as readonly string[]).includes(input.kind)) {
|
||||
throw new TargetRefused(`--target-kind ${input.kind} is not one of ${TARGET_KINDS.join(", ")}`);
|
||||
}
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(input.baseUrl);
|
||||
} catch {
|
||||
throw new TargetRefused("The base URL is not a valid URL");
|
||||
}
|
||||
if (url.username || url.password || url.search || url.hash || (url.pathname !== "/" && url.pathname !== "")) {
|
||||
throw new TargetRefused("The base URL must be a bare origin, with no path, query or credentials");
|
||||
}
|
||||
const host = url.hostname.toLowerCase();
|
||||
const kind = classifyHost(host);
|
||||
if (kind !== input.kind) {
|
||||
throw new TargetRefused(`${host} looks like a ${kind} host, not ${input.kind}; refusing`);
|
||||
}
|
||||
if (kind !== "local" && url.protocol !== "https:") throw new TargetRefused("A hosted target must use https");
|
||||
if (kind !== "local" && url.port) throw new TargetRefused("A hosted target must not name a port");
|
||||
if (kind === "shared-staging" && !input.allowSharedStaging) {
|
||||
throw new TargetRefused(`${host} is the shared staging instance; it needs the owner's go-ahead and --allow-shared-staging`);
|
||||
}
|
||||
if (kind === "production" && input.allowProductionHost?.toLowerCase() !== host) {
|
||||
throw new TargetRefused(`${host} looks like a production host; refusing unless --allow-production-host ${host} is also given`);
|
||||
}
|
||||
return { baseUrl: url.origin, host, kind };
|
||||
}
|
||||
|
||||
export function assertSameOrigin(target: Target, url: string): void {
|
||||
if (new URL(url).origin !== target.baseUrl) throw new TargetRefused(`Refusing a request outside ${target.baseUrl}`);
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { createInterface } from "node:readline/promises";
|
||||
import { cleanupRun, discoverState, type CleanupReport } from "./cleanup.ts";
|
||||
import { PaperclipClient, Redactor, realClock, type Clock, type Fetcher } from "./client.ts";
|
||||
import { FALLBACK_TOKEN_ENV, parseOptions, readSecrets, TOKEN_ENV, type Options } from "./config.ts";
|
||||
import { criteriaTable } from "./criteria.ts";
|
||||
import { checkSpend, estimateSpend, type SpendEstimate } from "./estimate.ts";
|
||||
import { Ramp, type RampOutcome } from "./ramp.ts";
|
||||
import { buildResult, summaryLines, verdictOf } from "./report.ts";
|
||||
import { prepareCompany } from "./setup.ts";
|
||||
import { emptyState, fileStateStore, type RunState } from "./state.ts";
|
||||
|
||||
export interface MainDeps {
|
||||
env: NodeJS.ProcessEnv;
|
||||
fetcher: Fetcher;
|
||||
clock: Clock;
|
||||
out: (line: string) => void;
|
||||
err: (line: string) => void;
|
||||
confirm: (question: string) => Promise<string>;
|
||||
onInterrupt: (handler: () => void) => () => void;
|
||||
random?: () => number;
|
||||
}
|
||||
|
||||
export const EXIT = { pass: 0, fail: 1, usage: 2, incomplete: 3, error: 4 } as const;
|
||||
|
||||
export function planLines(options: Options, estimate: SpendEstimate): string[] {
|
||||
const steps = options.steps.map(step => `${step.concurrency}`).join(" -> ");
|
||||
const credential = options.profile.credentialKey === null ? "none"
|
||||
: options.profile.providerSecretId ? `existing secret for ${options.profile.credentialKey}`
|
||||
: `${options.profile.credentialKey} from the environment, stored as a company secret and deleted at cleanup`;
|
||||
return [
|
||||
`Target: ${options.target.baseUrl} (${options.target.kind})`,
|
||||
`Run id: ${options.label}`,
|
||||
`Company: ${options.companyId ?? "a new company named for the run id"}`,
|
||||
`Ramp: ${steps} agents, ${options.steps[0]!.holdMs / 60_000} min per step, drain ${options.drainMs / 60_000} min, stop on the first failed criterion`,
|
||||
`Agents: up to ${estimate.agents}, ${options.profile.provider} ${options.profile.model}; credential: ${credential}`,
|
||||
`Tasks: about ${estimate.tasks}, hard cap ${estimate.maxTasks}`,
|
||||
`Estimated spend: $${estimate.totalUsd} (model $${estimate.modelUsd}, sandboxes $${estimate.sandboxUsd} for ${estimate.sandboxHours} sandbox-hours), about ${estimate.wallMinutes} min`,
|
||||
...estimate.assumptions.map(line => ` assumption: ${line}`),
|
||||
"Criteria:",
|
||||
...criteriaTable(options.thresholds).map(criterion => ` ${criterion.id} [${criterion.source}] ${criterion.description}: ${criterion.threshold}`),
|
||||
];
|
||||
}
|
||||
|
||||
async function writeOutputs(dir: string, name: string, body: unknown, summary: string[] | null, redactor: Redactor): Promise<void> {
|
||||
await mkdir(dir, { recursive: true, mode: 0o700 });
|
||||
await writeFile(path.join(dir, name), `${redactor.text(JSON.stringify(body, null, 2))}\n`, { mode: 0o600 });
|
||||
if (summary) await writeFile(path.join(dir, "SUMMARY.md"), `${redactor.text(summary.join("\n"))}\n`, { mode: 0o600 });
|
||||
}
|
||||
|
||||
export async function main(argv: readonly string[], deps: MainDeps): Promise<number> {
|
||||
let options: Options;
|
||||
try {
|
||||
options = parseOptions(argv, deps.env, deps.clock.now(), deps.random);
|
||||
} catch (error) {
|
||||
deps.err(`scale ramp: ${error instanceof Error ? error.message : String(error)}`);
|
||||
return EXIT.usage;
|
||||
}
|
||||
const secrets = readSecrets(deps.env, options.profile.providerSecretId ? null : options.profile.credentialKey);
|
||||
const redactor = new Redactor([secrets.token, secrets.credential, secrets.metricsToken]);
|
||||
const out = (line: string) => deps.out(redactor.text(line));
|
||||
const err = (line: string) => deps.err(redactor.text(line));
|
||||
const estimate = estimateSpend(options);
|
||||
|
||||
if (options.mode !== "cleanup") for (const line of planLines(options, estimate)) out(line);
|
||||
if (options.mode === "dry-run") {
|
||||
out("Dry run: nothing was created or contacted.");
|
||||
return EXIT.pass;
|
||||
}
|
||||
if (options.mode === "run") {
|
||||
const spend = checkSpend(estimate, options.spendThresholdUsd, options.confirmSpendUsd);
|
||||
if (!spend.ok) {
|
||||
err(spend.reason);
|
||||
return EXIT.usage;
|
||||
}
|
||||
}
|
||||
if (!secrets.token) {
|
||||
err(`Set ${TOKEN_ENV} (or ${FALLBACK_TOKEN_ENV}) to a board token for ${options.target.host}`);
|
||||
return EXIT.usage;
|
||||
}
|
||||
if (!options.yes) {
|
||||
const action = options.mode === "cleanup" ? `clean up ${options.label} on` : "start creating agents and tasks on";
|
||||
const answer = await deps.confirm(`Type ${options.target.host} to ${action} it: `);
|
||||
if (answer.trim().toLowerCase() !== options.target.host) {
|
||||
err("Confirmation did not match the target host; nothing was done.");
|
||||
return EXIT.usage;
|
||||
}
|
||||
}
|
||||
|
||||
const api = new PaperclipClient(options.target, secrets.token, deps.fetcher, deps.clock, redactor);
|
||||
const store = fileStateStore(options.outDir);
|
||||
|
||||
if (options.mode === "cleanup") {
|
||||
let report: CleanupReport | null = null;
|
||||
let error: string | null = null;
|
||||
try {
|
||||
const state = await store.load() ?? await discoverState(api, options.label, options.companyId, options.target.baseUrl, deps.clock.now());
|
||||
report = await cleanupRun({ api, clock: deps.clock, label: options.label, state, log: out });
|
||||
} catch (caught) {
|
||||
error = caught instanceof Error ? caught.message : String(caught);
|
||||
}
|
||||
await writeOutputs(options.outDir, `cleanup-${deps.clock.now()}.json`, { runId: options.label, report, error }, null, redactor);
|
||||
if (error) err(`Cleanup failed: ${error}`);
|
||||
if (report) out(`Cleanup ${options.label}: ${report.runsCancelled} runs cancelled, ${report.tasksArchived} tasks archived, ${report.agentsDeleted} agents deleted, ${report.errors.length} errors`);
|
||||
return error || (report?.errors.length ?? 0) > 0 ? EXIT.fail : EXIT.pass;
|
||||
}
|
||||
|
||||
let interrupted = false;
|
||||
const release = deps.onInterrupt(() => {
|
||||
interrupted = true;
|
||||
err("Interrupt received: stopping the ramp and cleaning up");
|
||||
});
|
||||
const startedAt = deps.clock.now();
|
||||
const state: RunState = emptyState(options.label, options.target.baseUrl, startedAt);
|
||||
let outcome: RampOutcome | null = null;
|
||||
let cleanup: CleanupReport | null = null;
|
||||
let error: string | null = null;
|
||||
try {
|
||||
await store.save(state);
|
||||
await prepareCompany({ api, options, state, store, credential: secrets.credential, log: out });
|
||||
const ramp = new Ramp({
|
||||
api, clock: deps.clock, fetcher: deps.fetcher, options, state, store, maxTasks: estimate.maxTasks,
|
||||
spendCapUsd: options.confirmSpendUsd ?? Math.max(options.spendThresholdUsd, estimate.totalUsd), metricsToken: secrets.metricsToken,
|
||||
log: out, shouldStop: () => interrupted,
|
||||
});
|
||||
outcome = await ramp.run();
|
||||
} catch (caught) {
|
||||
error = caught instanceof Error ? caught.message : String(caught);
|
||||
err(`Ramp stopped by an error: ${error}`);
|
||||
} finally {
|
||||
out(`Cleaning up ${options.label}`);
|
||||
try {
|
||||
cleanup = await cleanupRun({ api, clock: deps.clock, label: options.label, state, log: out });
|
||||
} catch (caught) {
|
||||
err(`Cleanup failed; finish it with --cleanup-only ${options.label}: ${caught instanceof Error ? caught.message : String(caught)}`);
|
||||
}
|
||||
release();
|
||||
}
|
||||
const failedRequests = api.records.filter(record => !record.ok).length;
|
||||
const result = buildResult({
|
||||
options, estimate, startedAt, finishedAt: deps.clock.now(), outcome, cleanup, error,
|
||||
requests: { total: api.records.length, failed: failedRequests },
|
||||
credentialSource: options.profile.credentialKey === null ? "none" : options.profile.providerSecretId ? "existing secret" : "stored for the run and deleted at cleanup",
|
||||
});
|
||||
const summary = summaryLines(result);
|
||||
await writeOutputs(options.outDir, "result.json", result, summary, redactor);
|
||||
for (const line of summary) out(line);
|
||||
out(`Result: ${path.join(options.outDir, "result.json")}`);
|
||||
if (cleanup && cleanup.errors.length) err(`Cleanup left ${cleanup.errors.length} errors; finish it with --cleanup-only ${options.label}`);
|
||||
const verdict = verdictOf(outcome, error);
|
||||
return verdict === "pass" ? EXIT.pass : verdict === "fail" ? EXIT.fail : verdict === "incomplete" ? EXIT.incomplete : EXIT.error;
|
||||
}
|
||||
|
||||
export function processDeps(): MainDeps {
|
||||
return {
|
||||
env: process.env,
|
||||
fetcher: (url, init) => fetch(url, init),
|
||||
clock: realClock,
|
||||
out: line => process.stdout.write(`${line}\n`),
|
||||
err: line => process.stderr.write(`${line}\n`),
|
||||
confirm: async question => {
|
||||
const rl = createInterface({ input: process.stdin, output: process.stderr });
|
||||
try {
|
||||
return await rl.question(question);
|
||||
} finally {
|
||||
rl.close();
|
||||
}
|
||||
},
|
||||
onInterrupt: handler => {
|
||||
const listener = () => handler();
|
||||
process.once("SIGINT", listener);
|
||||
process.once("SIGTERM", listener);
|
||||
return () => {
|
||||
process.off("SIGINT", listener);
|
||||
process.off("SIGTERM", listener);
|
||||
};
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,474 @@
|
||||
import { readMetrics, type Clock, type Fetcher, type PaperclipClient } from "./client.ts";
|
||||
import type { Options, Step } from "./config.ts";
|
||||
import { evaluate, firstFailure, type CriterionResult, type ServerMetrics, type StepMetrics } from "./criteria.ts";
|
||||
import { LIVE_RUN_STATUSES } from "./cleanup.ts";
|
||||
import { ensureAgents } from "./setup.ts";
|
||||
import { distribution } from "./stats.ts";
|
||||
import { mapLimit, taskTag, type RunState, type StateStore } from "./state.ts";
|
||||
import { runTiming, taskBrief, taskNonce, taskTitle, type RunEvent } from "./workload.ts";
|
||||
|
||||
type Json = Record<string, any>;
|
||||
|
||||
export const BOARD_MIX = [
|
||||
(companyId: string) => `/api/companies/${companyId}/dashboard`,
|
||||
(companyId: string) => `/api/companies/${companyId}/issues?limit=50`,
|
||||
(companyId: string) => `/api/companies/${companyId}/agents`,
|
||||
(companyId: string) => `/api/companies/${companyId}/live-runs`,
|
||||
(companyId: string) => `/api/companies/${companyId}/sidebar-badges`,
|
||||
];
|
||||
|
||||
const FAILED_RUN = new Set(["failed", "timed_out"]);
|
||||
const RESTING_NOT_DONE = new Set(["cancelled", "blocked", "in_review"]);
|
||||
const RUN_PAGE = 1_000;
|
||||
const IDS_PER_READ = 100;
|
||||
const RETRY_CHECK_AFTER_MS = 60_000;
|
||||
const RETRY_RECHECK_MS = 30_000;
|
||||
const EVENT_FETCHES_PER_TICK = 10;
|
||||
const EVENT_PAGES = 4;
|
||||
const SINGLE_READS_PER_TICK = 50;
|
||||
const AGENT_RUN_PAGE = 20;
|
||||
const AGENT_RUN_READ_PARALLELISM = 8;
|
||||
|
||||
interface TaskRecord {
|
||||
id: string;
|
||||
agentId: string;
|
||||
stepIndex: number;
|
||||
nonce: string;
|
||||
createdAt: number;
|
||||
status: string;
|
||||
outcome: "open" | "done" | "failed";
|
||||
neededRetry: boolean;
|
||||
lastRetryCheckAt: number | null;
|
||||
}
|
||||
|
||||
interface RunRecord {
|
||||
id: string;
|
||||
taskId: string;
|
||||
agentId: string;
|
||||
status: string;
|
||||
createdAt: number | null;
|
||||
startedAt: number | null;
|
||||
finishedAt: number | null;
|
||||
costUsd: number;
|
||||
sampled: boolean;
|
||||
}
|
||||
|
||||
interface StepAccumulator {
|
||||
step: Step;
|
||||
startedAt: number;
|
||||
endedAt: number | null;
|
||||
runningSamples: number[];
|
||||
boardMs: number[];
|
||||
boardRequests: number;
|
||||
boardFailures: number;
|
||||
serverSamples: Array<Record<string, unknown>>;
|
||||
ttftMs: number[];
|
||||
firstActivityMs: number[];
|
||||
sampled: number;
|
||||
}
|
||||
|
||||
export interface StepResult {
|
||||
index: number;
|
||||
concurrency: number;
|
||||
holdMs: number;
|
||||
metrics: StepMetrics;
|
||||
criteriaAtEnd: CriterionResult[];
|
||||
criteriaFinal: CriterionResult[];
|
||||
}
|
||||
|
||||
export interface StopReason {
|
||||
stepIndex: number;
|
||||
reason: string;
|
||||
criterion: string | null;
|
||||
}
|
||||
|
||||
export interface RampOutcome {
|
||||
steps: StepResult[];
|
||||
stop: StopReason | null;
|
||||
drained: boolean;
|
||||
openTasksAtEnd: number;
|
||||
tasksCreated: number;
|
||||
taskCapReached: boolean;
|
||||
recordedSpendUsd: number;
|
||||
runsListTruncated: boolean;
|
||||
metricsSource: "endpoint" | "not provided" | "unavailable";
|
||||
}
|
||||
|
||||
export interface RampDeps {
|
||||
api: PaperclipClient;
|
||||
clock: Clock;
|
||||
fetcher: Fetcher;
|
||||
options: Options;
|
||||
state: RunState;
|
||||
store: StateStore;
|
||||
maxTasks: number;
|
||||
spendCapUsd: number;
|
||||
metricsToken: string | null;
|
||||
log: (line: string) => void;
|
||||
shouldStop: () => boolean;
|
||||
}
|
||||
|
||||
const ms = (value: unknown): number | null => {
|
||||
if (value === null || value === undefined || value === "") return null;
|
||||
const parsed = typeof value === "number" ? value : Date.parse(String(value));
|
||||
return Number.isFinite(parsed) ? parsed : null;
|
||||
};
|
||||
|
||||
const num = (value: unknown): number | null => {
|
||||
const parsed = typeof value === "number" ? value : typeof value === "string" && value !== "" ? Number(value) : NaN;
|
||||
return Number.isFinite(parsed) ? parsed : null;
|
||||
};
|
||||
|
||||
function rows(body: unknown): Json[] {
|
||||
if (Array.isArray(body)) return body as Json[];
|
||||
if (body && typeof body === "object") {
|
||||
for (const key of ["issues", "rows", "items"]) {
|
||||
const value = (body as Json)[key];
|
||||
if (Array.isArray(value)) return value as Json[];
|
||||
}
|
||||
}
|
||||
return [];
|
||||
}
|
||||
|
||||
export function serverMetrics(samples: ReadonlyArray<Record<string, unknown>>): ServerMetrics {
|
||||
const max = (key: string) => {
|
||||
const values = samples.map(sample => num(sample[key])).filter((value): value is number => value !== null);
|
||||
return values.length ? Math.max(...values) : null;
|
||||
};
|
||||
const missed = samples.map(sample => num(sample.leaseRenewalsMissed)).filter((value): value is number => value !== null);
|
||||
return {
|
||||
mainThreadCpuMax: max("mainThreadCpu"),
|
||||
eventLoopUtilizationMax: max("eventLoopUtilization"),
|
||||
eventLoopDelayP99MsMax: max("eventLoopDelayP99Ms"),
|
||||
statementsPerSecondMax: max("statementsPerSecond"),
|
||||
poolWaitP99MsMax: max("poolWaitP99Ms"),
|
||||
runEventLatenessP99MsMax: max("runEventLatenessP99Ms"),
|
||||
agentApiPreHandlerP99MsMax: max("agentApiPreHandlerP99Ms"),
|
||||
leaseRenewalsMissed: missed.length ? missed[missed.length - 1]! - missed[0]! : null,
|
||||
samples: samples.length,
|
||||
};
|
||||
}
|
||||
|
||||
export class Ramp {
|
||||
private readonly tasks = new Map<string, TaskRecord>();
|
||||
private readonly runs = new Map<string, RunRecord>();
|
||||
private readonly runsByTask = new Map<string, Set<string>>();
|
||||
private readonly openByAgent = new Map<string, string>();
|
||||
private readonly accumulators: StepAccumulator[] = [];
|
||||
private nextBoardAt = 0;
|
||||
private nextPollAt = 0;
|
||||
private taskCapReached = false;
|
||||
private fallbackCursor = 0;
|
||||
private runsListTruncated = false;
|
||||
private metricsSource: RampOutcome["metricsSource"];
|
||||
private readonly deps: RampDeps;
|
||||
|
||||
constructor(deps: RampDeps) {
|
||||
this.deps = deps;
|
||||
this.metricsSource = deps.options.metricsUrl ? "unavailable" : "not provided";
|
||||
}
|
||||
|
||||
private get companyId(): string {
|
||||
return this.deps.state.companyId!;
|
||||
}
|
||||
|
||||
private current(): StepAccumulator {
|
||||
return this.accumulators[this.accumulators.length - 1]!;
|
||||
}
|
||||
|
||||
async run(): Promise<RampOutcome> {
|
||||
const { options, clock, log } = this.deps;
|
||||
const results: StepResult[] = [];
|
||||
let stop: StopReason | null = null;
|
||||
for (const step of options.steps) {
|
||||
if (this.deps.shouldStop()) {
|
||||
stop = { stepIndex: step.index, reason: "interrupted before the step", criterion: null };
|
||||
break;
|
||||
}
|
||||
log(`Step ${step.index + 1}/${options.steps.length}: ${step.concurrency} agents for ${Math.round(step.holdMs / 60_000)} min`);
|
||||
await ensureAgents({ api: this.deps.api, options, state: this.deps.state, store: this.deps.store, count: step.concurrency });
|
||||
const accumulator: StepAccumulator = { step, startedAt: clock.now(), endedAt: null, runningSamples: [], boardMs: [], boardRequests: 0,
|
||||
boardFailures: 0, serverSamples: [], ttftMs: [], firstActivityMs: [], sampled: 0 };
|
||||
this.accumulators.push(accumulator);
|
||||
this.nextPollAt = 0;
|
||||
this.nextBoardAt = 0;
|
||||
const until = accumulator.startedAt + step.holdMs;
|
||||
stop = await this.loop(until, step.concurrency);
|
||||
accumulator.endedAt = clock.now();
|
||||
const atEnd = evaluate(this.metrics(step.index), options.thresholds);
|
||||
results.push({ index: step.index, concurrency: step.concurrency, holdMs: step.holdMs, metrics: this.metrics(step.index), criteriaAtEnd: atEnd, criteriaFinal: atEnd });
|
||||
if (stop) break;
|
||||
const failed = firstFailure(atEnd);
|
||||
if (failed) {
|
||||
stop = { stepIndex: step.index, reason: `${failed.id} failed at the end of the step (observed ${failed.observed}, needs ${failed.threshold})`, criterion: failed.id };
|
||||
break;
|
||||
}
|
||||
log(`Step ${step.index + 1} passed its measured criteria`);
|
||||
}
|
||||
let drained = false;
|
||||
if (!stop && options.drainMs > 0) {
|
||||
log(`Draining open tasks for up to ${Math.round(options.drainMs / 60_000)} min`);
|
||||
stop = await this.loop(clock.now() + options.drainMs, 0, true);
|
||||
drained = !stop && this.openTasks().length === 0;
|
||||
const last = this.current();
|
||||
last.endedAt = clock.now();
|
||||
}
|
||||
for (const result of results) {
|
||||
result.metrics = this.metrics(result.index);
|
||||
result.criteriaFinal = evaluate(result.metrics, options.thresholds);
|
||||
}
|
||||
return {
|
||||
steps: results, stop, drained, openTasksAtEnd: this.openTasks().length, tasksCreated: this.tasks.size, taskCapReached: this.taskCapReached,
|
||||
recordedSpendUsd: this.recordedSpend(), runsListTruncated: this.runsListTruncated, metricsSource: this.metricsSource,
|
||||
};
|
||||
}
|
||||
|
||||
private async loop(until: number, concurrency: number, draining = false): Promise<StopReason | null> {
|
||||
const { clock, options } = this.deps;
|
||||
const stepIndex = this.current().step.index;
|
||||
while (clock.now() < until) {
|
||||
if (this.deps.shouldStop()) return { stepIndex, reason: "interrupted", criterion: null };
|
||||
if (clock.now() >= this.nextBoardAt) {
|
||||
this.nextBoardAt = clock.now() + options.boardProbeMs;
|
||||
await this.probeBoard();
|
||||
}
|
||||
if (clock.now() >= this.nextPollAt) {
|
||||
this.nextPollAt = clock.now() + options.pollMs;
|
||||
await this.poll();
|
||||
const stop = this.immediateStop(stepIndex);
|
||||
if (stop) return stop;
|
||||
if (!draining) await this.topUp(concurrency, stepIndex, until);
|
||||
else if (this.openTasks().length === 0) return null;
|
||||
}
|
||||
const wake = Math.min(this.nextBoardAt, this.nextPollAt, until);
|
||||
await clock.sleep(Math.max(1, wake - clock.now()));
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private immediateStop(stepIndex: number): StopReason | null {
|
||||
if (this.recordedSpend() > this.deps.spendCapUsd) {
|
||||
return { stepIndex, reason: `recorded model spend $${this.recordedSpend().toFixed(2)} passed the cap of $${this.deps.spendCapUsd}`, criterion: "spend_cap" };
|
||||
}
|
||||
for (const accumulator of this.accumulators) {
|
||||
const failed = firstFailure(evaluate(this.metrics(accumulator.step.index), this.deps.options.thresholds, true));
|
||||
if (failed) return { stepIndex, reason: `${failed.id} failed (observed ${failed.observed}, needs ${failed.threshold})`, criterion: failed.id };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private openTasks(): TaskRecord[] {
|
||||
return [...this.tasks.values()].filter(task => task.outcome === "open");
|
||||
}
|
||||
|
||||
private recordedSpend(): number {
|
||||
let total = 0;
|
||||
for (const run of this.runs.values()) total += run.costUsd;
|
||||
return Math.round(total * 100) / 100;
|
||||
}
|
||||
|
||||
private async topUp(concurrency: number, stepIndex: number, until: number): Promise<void> {
|
||||
const { api, state, store, options } = this.deps;
|
||||
for (const agentId of state.agentIds.slice(0, concurrency)) {
|
||||
if (this.deps.shouldStop() || this.deps.clock.now() >= until) return;
|
||||
if (this.openByAgent.has(agentId)) continue;
|
||||
if (this.tasks.size >= this.deps.maxTasks) {
|
||||
this.taskCapReached = true;
|
||||
return;
|
||||
}
|
||||
const index = state.taskIds.length;
|
||||
const nonce = taskNonce(options.label, index);
|
||||
const created = await api.post<{ id: string }>(`/api/companies/${this.companyId}/issues`, {
|
||||
title: taskTitle(taskTag(options.label), nonce),
|
||||
description: taskBrief(nonce),
|
||||
status: "todo",
|
||||
priority: "medium",
|
||||
assigneeAgentId: agentId,
|
||||
...(state.projectId ? { projectId: state.projectId } : {}),
|
||||
});
|
||||
const id = String(created.id);
|
||||
state.taskIds.push(id);
|
||||
await store.save(state);
|
||||
this.tasks.set(id, { id, agentId, stepIndex, nonce, createdAt: this.deps.clock.now(), status: "todo", outcome: "open", neededRetry: false, lastRetryCheckAt: null });
|
||||
this.openByAgent.set(agentId, id);
|
||||
}
|
||||
}
|
||||
|
||||
private async probeBoard(): Promise<void> {
|
||||
const accumulator = this.current();
|
||||
for (const route of BOARD_MIX) {
|
||||
const before = this.deps.api.records.length;
|
||||
await this.deps.api.get(route(this.companyId), "board").catch(() => null);
|
||||
const record = this.deps.api.records[before];
|
||||
if (!record) continue;
|
||||
accumulator.boardRequests += 1;
|
||||
if (record.ok) accumulator.boardMs.push(record.ms);
|
||||
else accumulator.boardFailures += 1;
|
||||
}
|
||||
}
|
||||
|
||||
private async poll(): Promise<void> {
|
||||
await this.readRuns();
|
||||
await this.readTasks();
|
||||
await this.checkRetry();
|
||||
await this.sampleEvents();
|
||||
let running = 0;
|
||||
for (const run of this.runs.values()) if (run.status === "running") running += 1;
|
||||
this.current().runningSamples.push(running);
|
||||
if (this.deps.options.metricsUrl) {
|
||||
const sample = await readMetrics(this.deps.fetcher, this.deps.options.metricsUrl, this.deps.metricsToken, this.deps.clock);
|
||||
if (sample) {
|
||||
this.metricsSource = "endpoint";
|
||||
this.current().serverSamples.push(sample);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private noteRun(row: Json): void {
|
||||
const taskId = String(row.issueId ?? row.contextSnapshot?.issueId ?? "");
|
||||
if (!this.tasks.has(taskId)) return;
|
||||
const id = String(row.id ?? row.runId);
|
||||
const previous = this.runs.get(id);
|
||||
this.runs.set(id, {
|
||||
id, taskId, agentId: String(row.agentId), status: String(row.status), createdAt: ms(row.createdAt), startedAt: ms(row.startedAt),
|
||||
finishedAt: ms(row.finishedAt), costUsd: num(row.costUsd) ?? previous?.costUsd ?? 0, sampled: previous?.sampled ?? false,
|
||||
});
|
||||
const own = this.runsByTask.get(taskId) ?? new Set<string>();
|
||||
own.add(id);
|
||||
this.runsByTask.set(taskId, own);
|
||||
}
|
||||
|
||||
private async readRuns(): Promise<void> {
|
||||
const page = await this.deps.api.get<Json[]>(`/api/companies/${this.companyId}/heartbeat-runs?summary=true&limit=${RUN_PAGE}`);
|
||||
for (const row of page) this.noteRun(row);
|
||||
if (page.length < RUN_PAGE) return;
|
||||
this.runsListTruncated = true;
|
||||
const listed = new Set(page.map(row => String(row.id)));
|
||||
const agents = new Set<string>();
|
||||
for (const run of this.runs.values()) if (LIVE_RUN_STATUSES.has(run.status) && !listed.has(run.id)) agents.add(run.agentId);
|
||||
for (const task of this.openTasks()) if (!(this.runsByTask.get(task.id)?.size)) agents.add(task.agentId);
|
||||
await mapLimit([...agents], AGENT_RUN_READ_PARALLELISM, async agentId => {
|
||||
const own = await this.deps.api.get<Json[]>(`/api/companies/${this.companyId}/heartbeat-runs?agentId=${agentId}&summary=true&limit=${AGENT_RUN_PAGE}`)
|
||||
.catch(() => [] as Json[]);
|
||||
for (const row of own) this.noteRun(row);
|
||||
});
|
||||
}
|
||||
|
||||
private async readTasks(): Promise<void> {
|
||||
const open = this.openTasks();
|
||||
const missing: TaskRecord[] = [];
|
||||
for (let offset = 0; offset < open.length; offset += IDS_PER_READ) {
|
||||
const batch = open.slice(offset, offset + IDS_PER_READ);
|
||||
const body = await this.deps.api.get<unknown>(`/api/companies/${this.companyId}/issues?view=rows&ids=${batch.map(task => task.id).join(",")}`).catch(() => null);
|
||||
const byId = new Map(rows(body).map(row => [String(row.id), row]));
|
||||
for (const task of batch) {
|
||||
const row = byId.get(task.id);
|
||||
if (row?.status) this.settle(task, String(row.status));
|
||||
else missing.push(task);
|
||||
}
|
||||
}
|
||||
if (!missing.length) return;
|
||||
const start = this.fallbackCursor % missing.length;
|
||||
const chosen = [...missing.slice(start), ...missing.slice(0, start)].slice(0, SINGLE_READS_PER_TICK);
|
||||
this.fallbackCursor += chosen.length;
|
||||
for (const task of chosen) {
|
||||
const row = await this.deps.api.get<Json>(`/api/issues/${task.id}`).catch(() => null);
|
||||
if (row?.status) this.settle(task, String(row.status));
|
||||
}
|
||||
}
|
||||
|
||||
private settle(task: TaskRecord, status: string): void {
|
||||
task.status = status;
|
||||
if (status === "done") task.outcome = "done";
|
||||
else if (RESTING_NOT_DONE.has(status)) task.outcome = "failed";
|
||||
if (task.outcome !== "open" && this.openByAgent.get(task.agentId) === task.id) this.openByAgent.delete(task.agentId);
|
||||
}
|
||||
|
||||
private runsOf(taskId: string): RunRecord[] {
|
||||
return [...(this.runsByTask.get(taskId) ?? [])].map(id => this.runs.get(id)!).sort((a, b) => (a.createdAt ?? 0) - (b.createdAt ?? 0));
|
||||
}
|
||||
|
||||
private async checkRetry(): Promise<void> {
|
||||
const now = this.deps.clock.now();
|
||||
for (const task of this.tasks.values()) {
|
||||
if (task.neededRetry || task.outcome === "done") continue;
|
||||
if (task.lastRetryCheckAt !== null && now - task.lastRetryCheckAt < RETRY_RECHECK_MS) continue;
|
||||
const runs = this.runsOf(task.id);
|
||||
if (!runs.length || runs.some(run => LIVE_RUN_STATUSES.has(run.status))) continue;
|
||||
const last = runs[runs.length - 1]!;
|
||||
const idle = last.finishedAt !== null && now - last.finishedAt >= RETRY_CHECK_AFTER_MS;
|
||||
if (task.outcome === "open" && last.status === "succeeded" && !idle) continue;
|
||||
if (task.outcome === "failed" && task.lastRetryCheckAt !== null) continue;
|
||||
task.lastRetryCheckAt = now;
|
||||
const view = await this.deps.api.get<Json>(`/api/issues/${task.id}`).catch(() => null);
|
||||
if (view && view.status !== "done" && view.interruption?.canRetry === true && !view.interruption?.retryWait) {
|
||||
task.neededRetry = true;
|
||||
this.deps.log(`Task ${task.nonce} offers Retry`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async sampleEvents(): Promise<void> {
|
||||
const cap = this.deps.options.eventSamplePerStep;
|
||||
let fetched = 0;
|
||||
for (const run of this.runs.values()) {
|
||||
if (fetched >= EVENT_FETCHES_PER_TICK) return;
|
||||
if (run.sampled || run.status !== "succeeded") continue;
|
||||
const accumulator = this.accumulators[this.tasks.get(run.taskId)!.stepIndex]!;
|
||||
if (accumulator.sampled >= cap) continue;
|
||||
run.sampled = true;
|
||||
accumulator.sampled += 1;
|
||||
fetched += 1;
|
||||
const events: RunEvent[] = [];
|
||||
for (let page = 0, afterSeq = 0; page < EVENT_PAGES; page += 1) {
|
||||
const rowsPage = await this.deps.api.get<Json[]>(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=500`).catch(() => [] as Json[]);
|
||||
for (const row of rowsPage) events.push({ eventType: String(row.eventType), createdAt: String(row.createdAt), payload: (row.payload ?? null) as Json | null });
|
||||
const last = Number(rowsPage.at(-1)?.seq);
|
||||
if (rowsPage.length < 500 || !Number.isFinite(last)) break;
|
||||
afterSeq = last;
|
||||
}
|
||||
const timing = runTiming(events, run.createdAt);
|
||||
if (timing.ttftMs !== null) accumulator.ttftMs.push(timing.ttftMs);
|
||||
if (timing.firstActivityMs !== null) accumulator.firstActivityMs.push(timing.firstActivityMs);
|
||||
}
|
||||
}
|
||||
|
||||
metrics(stepIndex: number): StepMetrics {
|
||||
const accumulator = this.accumulators[stepIndex]!;
|
||||
const tasks = [...this.tasks.values()].filter(task => task.stepIndex === stepIndex);
|
||||
const taskIds = new Set(tasks.map(task => task.id));
|
||||
const runs = [...this.runs.values()].filter(run => taskIds.has(run.taskId));
|
||||
const firstStart = new Map<string, number>();
|
||||
for (const run of runs) {
|
||||
if (run.startedAt === null) continue;
|
||||
const known = firstStart.get(run.taskId);
|
||||
if (known === undefined || run.startedAt < known) firstStart.set(run.taskId, run.startedAt);
|
||||
}
|
||||
const samples = accumulator.runningSamples;
|
||||
let spend = 0;
|
||||
for (const run of runs) spend += run.costUsd;
|
||||
return {
|
||||
step: accumulator.step,
|
||||
startedAt: accumulator.startedAt,
|
||||
endedAt: accumulator.endedAt,
|
||||
tasksCreated: tasks.length,
|
||||
tasksStarted: firstStart.size,
|
||||
tasksFinished: tasks.filter(task => task.outcome === "done").length,
|
||||
tasksFailed: tasks.filter(task => task.outcome === "failed").length,
|
||||
tasksNeedingRetry: tasks.filter(task => task.neededRetry).length,
|
||||
runsFailed: runs.filter(run => FAILED_RUN.has(run.status)).length,
|
||||
runsInterrupted: runs.filter(run => run.status === "interrupted").length,
|
||||
runsCancelled: runs.filter(run => run.status === "cancelled").length,
|
||||
peakRunning: samples.length ? Math.max(...samples) : 0,
|
||||
meanRunning: samples.length ? Math.round((samples.reduce((sum, value) => sum + value, 0) / samples.length) * 10) / 10 : null,
|
||||
queueWaitMs: distribution(runs.flatMap(run => run.startedAt !== null && run.createdAt !== null ? [run.startedAt - run.createdAt] : [])),
|
||||
taskToStartMs: distribution(tasks.flatMap(task => firstStart.has(task.id) ? [firstStart.get(task.id)! - task.createdAt] : [])),
|
||||
ttftMs: distribution(accumulator.ttftMs),
|
||||
firstActivityMs: distribution(accumulator.firstActivityMs),
|
||||
runDurationMs: distribution(runs.flatMap(run => run.startedAt !== null && run.finishedAt !== null ? [run.finishedAt - run.startedAt] : [])),
|
||||
board: { requests: accumulator.boardRequests, failures: accumulator.boardFailures, ms: distribution(accumulator.boardMs) },
|
||||
server: serverMetrics(accumulator.serverSamples),
|
||||
recordedSpendUsd: Math.round(spend * 100) / 100,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
import type { CleanupReport } from "./cleanup.ts";
|
||||
import type { Options } from "./config.ts";
|
||||
import { criteriaTable, type CriterionResult } from "./criteria.ts";
|
||||
import type { SpendEstimate } from "./estimate.ts";
|
||||
import type { RampOutcome } from "./ramp.ts";
|
||||
import type { Distribution } from "./stats.ts";
|
||||
|
||||
export type Verdict = "pass" | "fail" | "incomplete" | "interrupted" | "error";
|
||||
|
||||
export interface RampResult {
|
||||
schema: "paperclip.scale-ramp.result/v1";
|
||||
runId: string;
|
||||
target: { baseUrl: string; host: string; kind: string };
|
||||
startedAt: string;
|
||||
finishedAt: string;
|
||||
plan: {
|
||||
steps: Array<{ concurrency: number; holdMinutes: number }>;
|
||||
drainMinutes: number;
|
||||
pollSeconds: number;
|
||||
boardProbeSeconds: number;
|
||||
agent: { provider: string; model: string; credential: string };
|
||||
estimate: SpendEstimate;
|
||||
};
|
||||
criteria: ReturnType<typeof criteriaTable>;
|
||||
verdict: Verdict;
|
||||
stop: RampOutcome["stop"];
|
||||
steps: RampOutcome["steps"];
|
||||
notMeasured: string[];
|
||||
incompleteReasons: string[];
|
||||
drained: boolean;
|
||||
openTasksAtEnd: number;
|
||||
tasksCreated: number;
|
||||
taskCapReached: boolean;
|
||||
recordedSpendUsd: number | null;
|
||||
sources: Record<string, string>;
|
||||
cleanup: CleanupReport | null;
|
||||
requests: { total: number; failed: number };
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
export function verdictOf(outcome: RampOutcome | null, error: string | null): Verdict {
|
||||
if (error || !outcome) return "error";
|
||||
if (outcome.stop && outcome.stop.criterion) return "fail";
|
||||
if (outcome.stop) return "interrupted";
|
||||
if (outcome.steps.some(step => step.criteriaFinal.some(result => result.status === "fail"))) return "fail";
|
||||
if (incompleteReasons(outcome).length) return "incomplete";
|
||||
return "pass";
|
||||
}
|
||||
|
||||
export function incompleteReasons(outcome: RampOutcome | null): string[] {
|
||||
if (!outcome) return [];
|
||||
const reasons: string[] = [];
|
||||
const missing = notMeasured(outcome);
|
||||
if (missing.length) reasons.push(`criteria not measured: ${missing.join(", ")}`);
|
||||
if (outcome.taskCapReached) reasons.push("the task cap was reached, so a step may have run below its concurrency");
|
||||
if (outcome.openTasksAtEnd > 0) reasons.push(`${outcome.openTasksAtEnd} tasks were still open when the drain ended`);
|
||||
return reasons;
|
||||
}
|
||||
|
||||
export function notMeasured(outcome: RampOutcome | null): string[] {
|
||||
const ids = new Set<string>();
|
||||
for (const step of outcome?.steps ?? []) for (const result of step.criteriaFinal) if (result.status === "not_measured") ids.add(result.id);
|
||||
return [...ids];
|
||||
}
|
||||
|
||||
export function buildResult(input: {
|
||||
options: Options;
|
||||
estimate: SpendEstimate;
|
||||
startedAt: number;
|
||||
finishedAt: number;
|
||||
outcome: RampOutcome | null;
|
||||
cleanup: CleanupReport | null;
|
||||
error: string | null;
|
||||
requests: { total: number; failed: number };
|
||||
credentialSource: string;
|
||||
}): RampResult {
|
||||
const { options, outcome } = input;
|
||||
return {
|
||||
schema: "paperclip.scale-ramp.result/v1",
|
||||
runId: options.label,
|
||||
target: { baseUrl: options.target.baseUrl, host: options.target.host, kind: options.target.kind },
|
||||
startedAt: new Date(input.startedAt).toISOString(),
|
||||
finishedAt: new Date(input.finishedAt).toISOString(),
|
||||
plan: {
|
||||
steps: options.steps.map(step => ({ concurrency: step.concurrency, holdMinutes: step.holdMs / 60_000 })),
|
||||
drainMinutes: options.drainMs / 60_000,
|
||||
pollSeconds: options.pollMs / 1_000,
|
||||
boardProbeSeconds: options.boardProbeMs / 1_000,
|
||||
agent: { provider: options.profile.provider, model: options.profile.model, credential: input.credentialSource },
|
||||
estimate: input.estimate,
|
||||
},
|
||||
criteria: criteriaTable(options.thresholds),
|
||||
verdict: verdictOf(outcome, input.error),
|
||||
stop: outcome?.stop ?? null,
|
||||
steps: outcome?.steps ?? [],
|
||||
notMeasured: notMeasured(outcome),
|
||||
incompleteReasons: verdictOf(outcome, input.error) === "incomplete" ? incompleteReasons(outcome) : [],
|
||||
drained: outcome?.drained ?? false,
|
||||
openTasksAtEnd: outcome?.openTasksAtEnd ?? 0,
|
||||
tasksCreated: outcome?.tasksCreated ?? 0,
|
||||
taskCapReached: outcome?.taskCapReached ?? false,
|
||||
recordedSpendUsd: outcome ? outcome.recordedSpendUsd : null,
|
||||
sources: {
|
||||
runsAndTasks: outcome?.runsListTruncated ? "API (company run list was full; open tasks without a listed run were read one by one)" : "API",
|
||||
boardLatency: "harness GET mix (dashboard, issues, agents, live runs, sidebar badges)",
|
||||
firstTokenAndActivity: `run events of up to ${options.eventSamplePerStep} succeeded runs per step`,
|
||||
serverMetrics: outcome?.metricsSource === "endpoint" ? "PAPERCLIP_SCALE_METRICS_URL"
|
||||
: outcome?.metricsSource === "unavailable" ? "PAPERCLIP_SCALE_METRICS_URL gave no readable sample: not measured" : "not provided: not measured",
|
||||
database: "not read by this harness: not measured",
|
||||
},
|
||||
cleanup: input.cleanup,
|
||||
requests: input.requests,
|
||||
error: input.error,
|
||||
};
|
||||
}
|
||||
|
||||
const seconds = (value: number | null) => value === null ? "not measured" : `${(value / 1_000).toFixed(1)} s`;
|
||||
const millis = (value: number | null) => value === null ? "not measured" : `${Math.round(value)} ms`;
|
||||
const dist = (d: Distribution, format: (value: number | null) => string) => d.n === 0 ? "not measured" : `${format(d.p50)} / ${format(d.p95)} / ${format(d.p99)} (n=${d.n})`;
|
||||
|
||||
function criterionLine(result: CriterionResult): string {
|
||||
const observed = result.observed === null ? "not measured" : String(result.observed);
|
||||
return ` - ${result.status === "pass" ? "PASS" : result.status === "fail" ? "FAIL" : "not measured"} ${result.id}: ${observed} (needs ${result.threshold})`;
|
||||
}
|
||||
|
||||
export function summaryLines(result: RampResult): string[] {
|
||||
const lines = [
|
||||
`# Scale ramp ${result.runId}`,
|
||||
"",
|
||||
`Verdict: **${result.verdict}**. Target ${result.target.baseUrl} (${result.target.kind}).`,
|
||||
result.stop ? `Stopped at step ${result.stop.stepIndex + 1}: ${result.stop.reason}.` : "No stop.",
|
||||
`Tasks created: ${result.tasksCreated}${result.taskCapReached ? " (task cap reached)" : ""}. Recorded model spend: ${result.recordedSpendUsd === null ? "not measured" : `$${result.recordedSpendUsd}`}; estimate was $${result.plan.estimate.totalUsd}.`,
|
||||
"",
|
||||
];
|
||||
for (const step of result.steps) {
|
||||
const m = step.metrics;
|
||||
lines.push(
|
||||
`## Step ${step.index + 1}: ${step.concurrency} agents, ${step.holdMs / 60_000} min`,
|
||||
"",
|
||||
`- Tasks created / started / finished / failed / needed Retry: ${m.tasksCreated} / ${m.tasksStarted} / ${m.tasksFinished} / ${m.tasksFailed} / ${m.tasksNeedingRetry}`,
|
||||
`- Runs failed / interrupted / cancelled: ${m.runsFailed} / ${m.runsInterrupted} / ${m.runsCancelled}`,
|
||||
`- Running agents peak / mean: ${m.peakRunning} / ${m.meanRunning ?? "not measured"}`,
|
||||
`- Queue wait p50 / p95 / p99: ${dist(m.queueWaitMs, seconds)}`,
|
||||
`- Time to first token p50 / p95 / p99: ${dist(m.ttftMs, seconds)}`,
|
||||
`- First activity p50 / p95 / p99: ${dist(m.firstActivityMs, seconds)}`,
|
||||
`- Run duration p50 / p95 / p99: ${dist(m.runDurationMs, seconds)}`,
|
||||
`- Board API p50 / p95 / p99: ${dist(m.board.ms, millis)}; failed ${m.board.failures} of ${m.board.requests}`,
|
||||
`- Event-loop utilization max: ${m.server.eventLoopUtilizationMax ?? "not measured"}; delay p99 max: ${millis(m.server.eventLoopDelayP99MsMax)}; statements/s max: ${m.server.statementsPerSecondMax ?? "not measured"}; pool wait p99 max: ${millis(m.server.poolWaitP99MsMax)}`,
|
||||
"- Criteria (final):",
|
||||
...step.criteriaFinal.map(criterionLine),
|
||||
"",
|
||||
);
|
||||
}
|
||||
if (result.notMeasured.length) lines.push(`Not measured: ${result.notMeasured.join(", ")}.`, "");
|
||||
if (result.incompleteReasons.length) lines.push(`Incomplete because: ${result.incompleteReasons.join("; ")}.`, "");
|
||||
lines.push(`Drain: ${result.drained ? "every task finished" : `${result.openTasksAtEnd} tasks still open`}.`, "");
|
||||
if (result.cleanup) {
|
||||
const c = result.cleanup;
|
||||
lines.push(`Cleanup in ${seconds(c.durationMs)}: ${c.runsCancelled} runs cancelled, ${c.tasksArchived} tasks archived, ${c.agentsDeleted} agents deleted`
|
||||
+ ` (${c.agentsKeptForCostHistory} kept terminated for cost history), sandbox leases still active: ${c.activeSandboxLeases ?? "not readable"}, errors: ${c.errors.length}.`);
|
||||
}
|
||||
if (result.error) lines.push(`Error: ${result.error}`);
|
||||
return lines;
|
||||
}
|
||||
@@ -0,0 +1,544 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, readdir, readFile, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { after, describe, it } from "node:test";
|
||||
import { PaperclipClient, Redactor } from "./client.ts";
|
||||
import { parseOptions, QUALIFIED_OPENCODE_MODEL } from "./config.ts";
|
||||
import { evaluate, type StepMetrics } from "./criteria.ts";
|
||||
import { checkSpend, estimateSpend } from "./estimate.ts";
|
||||
import { FakePaperclip, VirtualClock, type FakeConfig } from "./fake-paperclip.ts";
|
||||
import { classifyHost, resolveTarget, TargetRefused } from "./guard.ts";
|
||||
import { EXIT, main, type MainDeps } from "./main.ts";
|
||||
import { Ramp } from "./ramp.ts";
|
||||
import { verdictOf, type RampResult } from "./report.ts";
|
||||
import { ensureAgents, prepareCompany } from "./setup.ts";
|
||||
import { distribution } from "./stats.ts";
|
||||
import { emptyState, fileStateStore, mapLimit, type RunState } from "./state.ts";
|
||||
import { runTiming } from "./workload.ts";
|
||||
|
||||
const TOKEN = "board-token-SECRET-4f2a9c";
|
||||
const PROVIDER_KEY = "sk-or-v1-SECRET-provider-key-77";
|
||||
const METRICS_TOKEN = "metrics-token-SECRET-91";
|
||||
const HOST = "scale1.test.paperclip.app";
|
||||
const ORIGIN = `https://${HOST}`;
|
||||
const METRICS_URL = "https://metrics.scale1.test/metrics";
|
||||
|
||||
const dirs: string[] = [];
|
||||
after(async () => {
|
||||
for (const dir of dirs) await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function tempDir(): Promise<string> {
|
||||
const dir = await mkdtemp(path.join(tmpdir(), "scale-ramp-test-"));
|
||||
dirs.push(dir);
|
||||
return dir;
|
||||
}
|
||||
|
||||
interface Harness {
|
||||
clock: VirtualClock;
|
||||
fake: FakePaperclip;
|
||||
out: string[];
|
||||
err: string[];
|
||||
deps: MainDeps;
|
||||
fetchCalls: () => number;
|
||||
interrupt: () => void;
|
||||
}
|
||||
|
||||
function harness(config: Partial<FakeConfig> = {}, env: Record<string, string> = {}, answer = HOST): Harness {
|
||||
const clock = new VirtualClock();
|
||||
const fake = new FakePaperclip(clock, { token: TOKEN, origin: ORIGIN, ...config });
|
||||
const out: string[] = [];
|
||||
const err: string[] = [];
|
||||
let calls = 0;
|
||||
let handler: (() => void) | null = null;
|
||||
const deps: MainDeps = {
|
||||
env: { PAPERCLIP_SCALE_TOKEN: TOKEN, OPENROUTER_API_KEY: PROVIDER_KEY, ...env },
|
||||
fetcher: async (url, init) => {
|
||||
calls += 1;
|
||||
return fake.fetch(url, init);
|
||||
},
|
||||
clock,
|
||||
out: line => out.push(line),
|
||||
err: line => err.push(line),
|
||||
confirm: async () => answer,
|
||||
onInterrupt: fn => {
|
||||
handler = fn;
|
||||
return () => {
|
||||
handler = null;
|
||||
};
|
||||
},
|
||||
random: () => 0.5,
|
||||
};
|
||||
return { clock, fake, out, err, deps, fetchCalls: () => calls, interrupt: () => handler?.() };
|
||||
}
|
||||
|
||||
const baseArgs = (out: string, extra: string[] = []) => [
|
||||
"--base-url", ORIGIN, "--target-kind", "test-stack", "--out", out, "--label", "scale-test-run", "--yes",
|
||||
"--hold-minutes", "2", "--drain-minutes", "3", "--poll-seconds", "15", "--board-probe-seconds", "10", "--task-minutes", "1", ...extra,
|
||||
];
|
||||
|
||||
async function readResult(out: string): Promise<RampResult> {
|
||||
return JSON.parse(await readFile(path.join(out, "scale-test-run", "result.json"), "utf8")) as RampResult;
|
||||
}
|
||||
|
||||
function assertCleanedUp(fake: FakePaperclip): void {
|
||||
assert.equal(fake.liveRuns().length, 0, "no queued or running run is left");
|
||||
assert.equal(fake.agents.size, 0, "every harness agent is deleted");
|
||||
assert.ok([...fake.issues.values()].every(issue => issue.hiddenAt), "every task is archived");
|
||||
assert.equal(fake.secrets.size, 0, "the stored provider secret is deleted");
|
||||
assert.ok([...fake.projects.values()].every(project => project.archivedAt), "the project is archived");
|
||||
}
|
||||
|
||||
const goodMetrics = (running: number) => ({
|
||||
eventLoopUtilization: 0.2 + running / 1_000, mainThreadCpu: 0.25, eventLoopDelayP99Ms: 20, statementsPerSecond: 40 * running,
|
||||
poolWaitP99Ms: 2, runEventLatenessP99Ms: 120, agentApiPreHandlerP99Ms: 15, leaseRenewalsMissed: 0,
|
||||
});
|
||||
|
||||
describe("target guard", () => {
|
||||
it("classifies hosts", () => {
|
||||
assert.equal(classifyHost("localhost"), "local");
|
||||
assert.equal(classifyHost("app.localhost"), "local");
|
||||
assert.equal(classifyHost("127.0.0.1"), "local");
|
||||
assert.equal(classifyHost("scale1.test.paperclip.app"), "test-stack");
|
||||
assert.equal(classifyHost("chaos-a.staging.paperclip.app"), "test-stack");
|
||||
assert.equal(classifyHost("paperclip.staging.paperclip.app"), "shared-staging");
|
||||
assert.equal(classifyHost("admin.staging.paperclip.app"), "production");
|
||||
assert.equal(classifyHost("paperclip.app"), "production");
|
||||
assert.equal(classifyHost("acme.paperclip.app"), "production");
|
||||
assert.equal(classifyHost("example.com"), "production");
|
||||
});
|
||||
|
||||
it("requires a target kind that matches the host", () => {
|
||||
const base = { allowSharedStaging: false, allowProductionHost: undefined };
|
||||
assert.throws(() => resolveTarget({ ...base, baseUrl: ORIGIN, kind: undefined }), /--target-kind is required/);
|
||||
assert.throws(() => resolveTarget({ ...base, baseUrl: "https://acme.paperclip.app", kind: "test-stack" }), /looks like a production host/);
|
||||
assert.throws(() => resolveTarget({ ...base, baseUrl: ORIGIN, kind: "local" }), /looks like a test-stack host/);
|
||||
assert.throws(() => resolveTarget({ ...base, baseUrl: `http://${HOST}`, kind: "test-stack" }), /must use https/);
|
||||
assert.throws(() => resolveTarget({ ...base, baseUrl: `${ORIGIN}/api`, kind: "test-stack" }), /bare origin/);
|
||||
assert.deepEqual(resolveTarget({ ...base, baseUrl: "http://localhost:3100", kind: "local" }), { baseUrl: "http://localhost:3100", host: "localhost", kind: "local" });
|
||||
});
|
||||
|
||||
it("refuses production and shared staging without the second explicit flag", () => {
|
||||
const base = { allowSharedStaging: false, allowProductionHost: undefined };
|
||||
assert.throws(() => resolveTarget({ ...base, baseUrl: "https://acme.paperclip.app", kind: "production" }), TargetRefused);
|
||||
assert.throws(() => resolveTarget({ ...base, baseUrl: "https://acme.paperclip.app", kind: "production", allowProductionHost: "other.paperclip.app" }), TargetRefused);
|
||||
assert.equal(resolveTarget({ ...base, baseUrl: "https://acme.paperclip.app", kind: "production", allowProductionHost: "acme.paperclip.app" }).kind, "production");
|
||||
assert.throws(() => resolveTarget({ ...base, baseUrl: "https://paperclip.staging.paperclip.app", kind: "shared-staging" }), /owner's go-ahead/);
|
||||
assert.equal(resolveTarget({ ...base, allowSharedStaging: true, baseUrl: "https://paperclip.staging.paperclip.app", kind: "shared-staging" }).kind, "shared-staging");
|
||||
});
|
||||
|
||||
it("stops before any request when the host looks like production", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness();
|
||||
const code = await main(["--base-url", "https://acme.paperclip.app", "--target-kind", "production", "--out", out, "--yes"], h.deps);
|
||||
assert.equal(code, EXIT.usage);
|
||||
assert.equal(h.fetchCalls(), 0);
|
||||
assert.match(h.err.join("\n"), /looks like a production host/);
|
||||
});
|
||||
|
||||
it("stops before any request when the typed confirmation does not match", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness({}, {}, "yes");
|
||||
const args = baseArgs(out).filter(arg => arg !== "--yes");
|
||||
assert.equal(await main(args, h.deps), EXIT.usage);
|
||||
assert.equal(h.fetchCalls(), 0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("dry run and spend", () => {
|
||||
it("prints the plan and estimate and touches nothing", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness();
|
||||
const code = await main(["--base-url", ORIGIN, "--target-kind", "test-stack", "--out", out, "--dry-run"], h.deps);
|
||||
assert.equal(code, EXIT.pass);
|
||||
assert.equal(h.fetchCalls(), 0);
|
||||
assert.deepEqual(await readdir(out), []);
|
||||
const text = h.out.join("\n");
|
||||
assert.match(text, /Ramp: 10 -> 50 -> 100 -> 200 -> 500 -> 1000 agents/);
|
||||
assert.match(text, /Estimated spend: \$\d/);
|
||||
assert.match(text, /retry_needed_zero \[finish-line\]/);
|
||||
assert.match(text, /Dry run: nothing was created or contacted/);
|
||||
});
|
||||
|
||||
it("estimates tasks, sandbox-hours and cost from the steps", () => {
|
||||
const options = parseOptions(["--base-url", ORIGIN, "--target-kind", "test-stack", "--steps", "10,50", "--hold-minutes", "10", "--task-minutes", "5",
|
||||
"--drain-minutes", "10"], {}, 0);
|
||||
const estimate = estimateSpend(options);
|
||||
assert.equal(estimate.agents, 50);
|
||||
assert.equal(estimate.tasks, 10 * 2 + 50 * 2);
|
||||
assert.equal(estimate.maxTasks, 150);
|
||||
assert.equal(estimate.sandboxHours, 18.3);
|
||||
assert.equal(estimate.modelUsd, 3);
|
||||
assert.equal(estimate.totalUsd, 4.83);
|
||||
});
|
||||
|
||||
it("requires --confirm-spend of at least the estimate above the threshold", async () => {
|
||||
const estimate = { totalUsd: 120 } as ReturnType<typeof estimateSpend>;
|
||||
assert.equal(checkSpend(estimate, 5, null).ok, false);
|
||||
assert.equal(checkSpend(estimate, 5, 100).ok, false);
|
||||
assert.equal(checkSpend(estimate, 5, 130).ok, true);
|
||||
assert.equal(checkSpend(estimate, 200, null).ok, true);
|
||||
const out = await tempDir();
|
||||
const h = harness();
|
||||
const code = await main(["--base-url", ORIGIN, "--target-kind", "test-stack", "--out", out, "--yes"], h.deps);
|
||||
assert.equal(code, EXIT.usage);
|
||||
assert.equal(h.fetchCalls(), 0);
|
||||
assert.match(h.err.join("\n"), /--confirm-spend/);
|
||||
});
|
||||
|
||||
it("defaults to the qualified OpenCode model of the runner", async () => {
|
||||
const source = await readFile(path.join(import.meta.dirname, "../../packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts"), "utf8");
|
||||
const match = /QUALIFIED_OPENCODE_MODEL\s*=\s*"([^"]+)"/.exec(source);
|
||||
assert.equal(match?.[1], QUALIFIED_OPENCODE_MODEL);
|
||||
});
|
||||
});
|
||||
|
||||
describe("criteria", () => {
|
||||
const empty = distribution([]);
|
||||
const metrics = (patch: Partial<StepMetrics> = {}): StepMetrics => ({
|
||||
step: { index: 0, concurrency: 10, holdMs: 60_000 }, startedAt: 0, endedAt: 1, tasksCreated: 10, tasksStarted: 10, tasksFinished: 10,
|
||||
tasksFailed: 0, tasksNeedingRetry: 0, runsFailed: 0, runsInterrupted: 0, runsCancelled: 0, peakRunning: 10, meanRunning: 9,
|
||||
queueWaitMs: distribution([1_000, 2_000]), taskToStartMs: empty, ttftMs: empty, firstActivityMs: distribution([4_000]), runDurationMs: empty,
|
||||
board: { requests: 10, failures: 0, ms: distribution([100, 120, 130, 140]) },
|
||||
server: { mainThreadCpuMax: null, eventLoopUtilizationMax: null, eventLoopDelayP99MsMax: null, statementsPerSecondMax: null, poolWaitP99MsMax: null,
|
||||
runEventLatenessP99MsMax: null, agentApiPreHandlerP99MsMax: null, leaseRenewalsMissed: null, samples: 0 },
|
||||
recordedSpendUsd: 0, ...patch,
|
||||
});
|
||||
const thresholds = parseOptions(["--base-url", ORIGIN, "--target-kind", "test-stack"], {}, 0).thresholds;
|
||||
const byId = (results: ReturnType<typeof evaluate>) => Object.fromEntries(results.map(result => [result.id, result.status]));
|
||||
|
||||
it("reports missing server metrics as not measured, never as zero", () => {
|
||||
const results = byId(evaluate(metrics(), thresholds));
|
||||
assert.equal(results.retry_needed_zero, "pass");
|
||||
assert.equal(results.board_api_p90, "pass");
|
||||
assert.equal(results.main_thread_cpu, "not_measured");
|
||||
assert.equal(results.lease_renewals_missed_zero, "not_measured");
|
||||
assert.equal(results.run_event_lateness_p99, "not_measured");
|
||||
assert.equal(results.kill_one_process_resumes, "not_measured");
|
||||
assert.equal(byId(evaluate(metrics({ board: { requests: 0, failures: 0, ms: empty } }), thresholds)).board_api_p90, "not_measured");
|
||||
});
|
||||
|
||||
it("fails each finish-line threshold it can measure", () => {
|
||||
const server = { mainThreadCpuMax: 0.7, eventLoopUtilizationMax: 0.5, eventLoopDelayP99MsMax: 10, statementsPerSecondMax: 1, poolWaitP99MsMax: 1,
|
||||
runEventLatenessP99MsMax: 1_500, agentApiPreHandlerP99MsMax: 100, leaseRenewalsMissed: 2, samples: 3 };
|
||||
const results = byId(evaluate(metrics({ tasksNeedingRetry: 1, peakRunning: 8, board: { requests: 4, failures: 1, ms: distribution([100, 600, 700]) }, server }), thresholds));
|
||||
assert.deepEqual(results, {
|
||||
retry_needed_zero: "fail", failed_runs_zero: "pass", tasks_rested_not_done_zero: "pass", concurrency_reached: "fail", board_api_p90: "fail", board_api_errors_zero: "fail",
|
||||
queue_wait_p95: "pass", first_activity_p95: "pass", main_thread_cpu: "fail", run_event_lateness_p99: "fail", agent_api_pre_handler_p99: "fail",
|
||||
lease_renewals_missed_zero: "fail", kill_one_process_resumes: "not_measured",
|
||||
});
|
||||
});
|
||||
|
||||
it("only evaluates retry and failed runs as immediate stops", () => {
|
||||
assert.deepEqual(evaluate(metrics(), thresholds, true).map(result => result.id), ["retry_needed_zero", "failed_runs_zero"]);
|
||||
});
|
||||
|
||||
it("reads first token and first activity from run spans, with an event fallback", () => {
|
||||
const created = Date.parse("2026-11-14T10:00:00Z");
|
||||
const spans = runTiming([
|
||||
{ eventType: "run.performance.span", createdAt: "2026-11-14T10:00:01Z", payload: { span: "heartbeat.queue", startOffsetMs: 200, durationMs: 800 } },
|
||||
{ eventType: "run.performance.span", createdAt: "2026-11-14T10:00:06Z", payload: { span: "task.time_to_first_token", startOffsetMs: 0, durationMs: 6_200 } },
|
||||
], created);
|
||||
assert.deepEqual(spans, { ttftMs: 6_000, firstActivityMs: null });
|
||||
const fallback = runTiming([{ eventType: "tool.execution.started", createdAt: "2026-11-14T10:00:03Z" }, { eventType: "item.delta", createdAt: "2026-11-14T10:00:07Z" }], created);
|
||||
assert.deepEqual(fallback, { ttftMs: 7_000, firstActivityMs: 3_000 });
|
||||
});
|
||||
});
|
||||
|
||||
describe("ramp against the fake API", () => {
|
||||
it("ramps every step, measures it, passes with a metrics endpoint and cleans up", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness({ metricsUrl: METRICS_URL, metrics: goodMetrics }, { PAPERCLIP_SCALE_METRICS_URL: METRICS_URL, PAPERCLIP_SCALE_METRICS_TOKEN: METRICS_TOKEN });
|
||||
const code = await main(baseArgs(out, ["--steps", "2,4"]), h.deps);
|
||||
const result = await readResult(out);
|
||||
assert.equal(result.verdict, "incomplete", "the process-kill criterion is never measured here");
|
||||
assert.equal(code, EXIT.incomplete);
|
||||
assert.deepEqual(result.notMeasured, ["kill_one_process_resumes"]);
|
||||
assert.equal(result.steps.length, 2);
|
||||
const [first, second] = result.steps;
|
||||
assert.equal(first!.concurrency, 2);
|
||||
assert.equal(second!.metrics.peakRunning, 4);
|
||||
assert.ok(first!.metrics.tasksFinished >= 2);
|
||||
assert.equal(second!.metrics.tasksNeedingRetry, 0);
|
||||
assert.ok(second!.metrics.firstActivityMs.n > 0);
|
||||
assert.ok(second!.metrics.board.ms.n >= 3);
|
||||
assert.equal(second!.metrics.server.statementsPerSecondMax, 160);
|
||||
assert.ok(second!.criteriaFinal.filter(criterion => criterion.id !== "kill_one_process_resumes").every(criterion => criterion.status === "pass"));
|
||||
assert.equal(result.sources.serverMetrics, "PAPERCLIP_SCALE_METRICS_URL");
|
||||
assert.ok(result.recordedSpendUsd! > 0);
|
||||
const agents = h.fake.requests.filter(request => request.method === "POST" && new URL(request.url).pathname.endsWith("/agents"));
|
||||
assert.equal(agents.length, 4, "agents are created per step up to the largest step");
|
||||
const payload = JSON.parse(agents[0]!.body!);
|
||||
assert.equal(payload.adapterConfig.model, QUALIFIED_OPENCODE_MODEL);
|
||||
assert.equal(payload.runtimeConfig.heartbeat.maxConcurrentRuns, 1);
|
||||
assert.equal(payload.adapterConfig.env.OPENROUTER_API_KEY.type, "secret_ref");
|
||||
assertCleanedUp(h.fake);
|
||||
assert.ok([...h.fake.companies.values()].every(company => company.status === "archived"), "the created company is archived");
|
||||
assert.equal(result.cleanup!.errors.length, 0);
|
||||
assert.equal(result.cleanup!.activeSandboxLeases, 0);
|
||||
});
|
||||
|
||||
it("reports missing optional sources as not measured", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness();
|
||||
await main(baseArgs(out, ["--steps", "2"]), h.deps);
|
||||
const result = await readResult(out);
|
||||
assert.equal(result.verdict, "incomplete");
|
||||
assert.match(result.sources.serverMetrics!, /not measured/);
|
||||
assert.ok(result.notMeasured.includes("main_thread_cpu"));
|
||||
const summary = await readFile(path.join(out, "scale-test-run", "SUMMARY.md"), "utf8");
|
||||
assert.match(summary, /Event-loop utilization max: not measured/);
|
||||
assert.doesNotMatch(summary, /Event-loop utilization max: 0\b/);
|
||||
});
|
||||
|
||||
it("stops on the first failed criterion and never starts the next step", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness({ failWhenRunningAbove: 3 });
|
||||
const code = await main(baseArgs(out, ["--steps", "2,4,8"]), h.deps);
|
||||
const result = await readResult(out);
|
||||
assert.equal(code, EXIT.fail);
|
||||
assert.equal(result.verdict, "fail");
|
||||
assert.equal(result.stop!.stepIndex, 1);
|
||||
assert.equal(result.stop!.criterion, "failed_runs_zero");
|
||||
assert.equal(result.steps.length, 2);
|
||||
const agents = h.fake.requests.filter(request => request.method === "POST" && new URL(request.url).pathname.endsWith("/agents"));
|
||||
assert.equal(agents.length, 4, "the 8-agent step never created agents");
|
||||
assertCleanedUp(h.fake);
|
||||
});
|
||||
|
||||
it("stops when a task offers Retry", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness({ failWhenRunningAbove: 1, retryOnFailure: true });
|
||||
await main(baseArgs(out, ["--steps", "2,4"]), h.deps);
|
||||
const result = await readResult(out);
|
||||
assert.equal(result.verdict, "fail");
|
||||
assert.equal(result.stop!.stepIndex, 0);
|
||||
assert.ok(["retry_needed_zero", "failed_runs_zero"].includes(result.stop!.criterion!));
|
||||
assert.ok(result.steps[0]!.metrics.tasksNeedingRetry + result.steps[0]!.metrics.runsFailed > 0);
|
||||
});
|
||||
|
||||
it("fails a step whose board requests are slow", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness({ boardLatencyMs: running => running >= 3 ? 900 : 50 });
|
||||
await main(baseArgs(out, ["--steps", "2,4"]), h.deps);
|
||||
const result = await readResult(out);
|
||||
assert.equal(result.verdict, "fail");
|
||||
assert.equal(result.stop!.criterion, "board_api_p90");
|
||||
assert.equal(result.stop!.stepIndex, 1);
|
||||
});
|
||||
|
||||
it("fails a step that cannot reach its concurrency", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness({ capacity: 2 });
|
||||
await main(baseArgs(out, ["--steps", "2,4"]), h.deps);
|
||||
const result = await readResult(out);
|
||||
assert.equal(result.stop!.criterion, "concurrency_reached");
|
||||
assert.equal(result.steps[1]!.metrics.peakRunning, 2);
|
||||
});
|
||||
|
||||
it("stops creating tasks at the task cap and grades the run incomplete", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness({ metricsUrl: METRICS_URL, metrics: goodMetrics }, { PAPERCLIP_SCALE_METRICS_URL: METRICS_URL });
|
||||
await main(baseArgs(out, ["--steps", "2", "--max-tasks", "3"]), h.deps);
|
||||
const result = await readResult(out);
|
||||
assert.equal(result.tasksCreated, 3);
|
||||
assert.equal(result.taskCapReached, true);
|
||||
assert.equal(result.verdict, "incomplete");
|
||||
assert.ok(result.incompleteReasons.some(reason => reason.includes("task cap")));
|
||||
});
|
||||
|
||||
it("grades a run incomplete when tasks are still open after the drain", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness({ runDurationMs: 20 * 60_000 });
|
||||
await main(baseArgs(out, ["--steps", "2", "--min-concurrency-ratio", "0.5"]), h.deps);
|
||||
const result = await readResult(out);
|
||||
assert.equal(result.drained, false);
|
||||
assert.ok(result.openTasksAtEnd > 0);
|
||||
assert.ok(result.incompleteReasons.some(reason => reason.includes("still open")));
|
||||
assertCleanedUp(h.fake);
|
||||
});
|
||||
|
||||
it("creates no more tasks once interrupted", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness();
|
||||
const original = h.deps.fetcher;
|
||||
h.deps.fetcher = async (url, init) => {
|
||||
if (init.method === "POST" && new URL(url).pathname.endsWith("/issues")) h.interrupt();
|
||||
return original(url, init);
|
||||
};
|
||||
await main(baseArgs(out, ["--steps", "4"]), h.deps);
|
||||
assert.equal(h.fake.issues.size, 1);
|
||||
assertCleanedUp(h.fake);
|
||||
});
|
||||
|
||||
it("keeps tracking its runs when the company run list is full of other runs", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness({ metricsUrl: METRICS_URL, metrics: goodMetrics }, { PAPERCLIP_SCALE_METRICS_URL: METRICS_URL });
|
||||
h.fake.addNoiseRuns(1_000);
|
||||
await main(baseArgs(out, ["--steps", "2"]), h.deps);
|
||||
const result = await readResult(out);
|
||||
assert.match(result.sources.runsAndTasks!, /full/);
|
||||
assert.equal(result.steps[0]!.metrics.peakRunning, 2);
|
||||
assert.ok(result.steps[0]!.metrics.tasksFinished >= 2);
|
||||
assert.ok(result.recordedSpendUsd! > 0);
|
||||
assert.ok(result.drained, "finished runs are not left counted as running");
|
||||
});
|
||||
|
||||
it("cleans up after a crash in the middle of the ramp", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness({ failRequest: (method, route, count) => method === "POST" && route.endsWith("/issues") && count === 5 ? { status: 500, body: "boom" } : null });
|
||||
const code = await main(baseArgs(out, ["--steps", "2,4"]), h.deps);
|
||||
const result = await readResult(out);
|
||||
assert.equal(code, EXIT.error);
|
||||
assert.equal(result.verdict, "error");
|
||||
assert.match(result.error!, /returned 500/);
|
||||
assertCleanedUp(h.fake);
|
||||
});
|
||||
|
||||
it("cleans up when interrupted", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness();
|
||||
const original = h.deps.fetcher;
|
||||
let requests = 0;
|
||||
h.deps.fetcher = async (url, init) => {
|
||||
requests += 1;
|
||||
if (requests === 40) h.interrupt();
|
||||
return original(url, init);
|
||||
};
|
||||
await main(baseArgs(out, ["--steps", "2,4"]), h.deps);
|
||||
const result = await readResult(out);
|
||||
assert.equal(result.verdict, "interrupted");
|
||||
assertCleanedUp(h.fake);
|
||||
});
|
||||
|
||||
it("finishes a cleanup with --cleanup-only after the harness process died", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness();
|
||||
const options = parseOptions(baseArgs(out, ["--steps", "2"]), h.deps.env, h.clock.now());
|
||||
const api = new PaperclipClient(options.target, TOKEN, h.deps.fetcher, h.clock, new Redactor([TOKEN]));
|
||||
const store = fileStateStore(options.outDir);
|
||||
const state: RunState = emptyState(options.label, options.target.baseUrl, h.clock.now());
|
||||
await prepareCompany({ api, options, state, store, credential: PROVIDER_KEY, log: () => {} });
|
||||
let ticks = 0;
|
||||
await new Ramp({ api, clock: h.clock, fetcher: h.deps.fetcher, options, state, store, maxTasks: 100, spendCapUsd: 100, metricsToken: null,
|
||||
log: () => {}, shouldStop: () => (ticks += 1) > 6 }).run();
|
||||
assert.ok(h.fake.liveRuns().length > 0, "the dead harness left live runs");
|
||||
assert.ok(h.fake.agents.size > 0);
|
||||
const code = await main(["--base-url", ORIGIN, "--target-kind", "test-stack", "--out", out, "--cleanup-only", "scale-test-run", "--yes"], h.deps);
|
||||
assert.equal(code, EXIT.pass);
|
||||
assertCleanedUp(h.fake);
|
||||
assert.ok([...h.fake.companies.values()].every(company => company.status === "archived"));
|
||||
});
|
||||
|
||||
it("archives a task the state file never recorded", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness();
|
||||
const options = parseOptions(baseArgs(out, ["--steps", "2"]), h.deps.env, h.clock.now());
|
||||
const api = new PaperclipClient(options.target, TOKEN, h.deps.fetcher, h.clock, new Redactor([TOKEN]));
|
||||
const store = fileStateStore(options.outDir);
|
||||
const state: RunState = emptyState(options.label, options.target.baseUrl, h.clock.now());
|
||||
await prepareCompany({ api, options, state, store, credential: PROVIDER_KEY, log: () => {} });
|
||||
await ensureAgents({ api, options, state, store, count: 2 });
|
||||
const recorded = await api.post<{ id: string }>(`/api/companies/${state.companyId}/issues`, { title: "recorded", assigneeAgentId: state.agentIds[0] });
|
||||
state.taskIds.push(recorded.id);
|
||||
await store.save(state);
|
||||
const unrecorded = await api.post<{ id: string }>(`/api/companies/${state.companyId}/issues`, { title: "unrecorded", assigneeAgentId: state.agentIds[1] });
|
||||
assert.equal(await main(["--base-url", ORIGIN, "--target-kind", "test-stack", "--out", out, "--cleanup-only", "scale-test-run", "--yes"], h.deps), EXIT.pass);
|
||||
assert.ok(h.fake.issues.get(unrecorded.id)!.hiddenAt);
|
||||
assertCleanedUp(h.fake);
|
||||
});
|
||||
|
||||
it("finds the run by name for --cleanup-only when the state file is gone", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness();
|
||||
const options = parseOptions(baseArgs(out, ["--steps", "2"]), h.deps.env, h.clock.now());
|
||||
const api = new PaperclipClient(options.target, TOKEN, h.deps.fetcher, h.clock, new Redactor([TOKEN]));
|
||||
const state: RunState = emptyState(options.label, options.target.baseUrl, h.clock.now());
|
||||
const otherCompany = h.fake.addMemberCompany("Someone else");
|
||||
await prepareCompany({ api, options, state, store: fileStateStore(path.join(out, "discarded")), credential: PROVIDER_KEY, log: () => {} });
|
||||
await api.post(`/api/companies/${otherCompany}/agents`, { name: "Unrelated agent" });
|
||||
const code = await main(["--base-url", ORIGIN, "--target-kind", "test-stack", "--out", out, "--cleanup-only", "scale-test-run", "--yes"], h.deps);
|
||||
assert.equal(code, EXIT.pass);
|
||||
assert.equal(h.fake.secrets.size, 0);
|
||||
assert.deepEqual([...h.fake.agents.values()].map(agent => agent.name), ["Unrelated agent"], "only objects carrying the run id are touched");
|
||||
assert.equal(h.fake.companies.get(otherCompany)!.status, "active");
|
||||
});
|
||||
|
||||
it("uses a supplied company when the instance cannot create one", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness({ cloudManaged: true });
|
||||
const refused = await main(baseArgs(out, ["--steps", "2"]), h.deps);
|
||||
assert.equal(refused, EXIT.error);
|
||||
assert.match((await readResult(out)).error!, /pass --company/);
|
||||
const companyId = h.fake.addMemberCompany("Existing workspace");
|
||||
const out2 = await tempDir();
|
||||
const code = await main(baseArgs(out2, ["--steps", "2", "--company", companyId]), h.deps);
|
||||
assert.equal(code, EXIT.incomplete);
|
||||
assertCleanedUp(h.fake);
|
||||
assert.equal(h.fake.companies.get(companyId)!.status, "active", "a supplied company is never archived");
|
||||
});
|
||||
});
|
||||
|
||||
describe("state and parallel work", () => {
|
||||
it("stops assigning work after a failure and waits for started work", async () => {
|
||||
const started: number[] = [];
|
||||
let running = 0;
|
||||
let finishedAfterFailure = 0;
|
||||
let failed = false;
|
||||
await assert.rejects(mapLimit([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 3, async item => {
|
||||
started.push(item);
|
||||
running += 1;
|
||||
await new Promise(resolve => setTimeout(resolve, item === 2 ? 1 : 10));
|
||||
running -= 1;
|
||||
if (item === 2) {
|
||||
failed = true;
|
||||
throw new Error("create failed");
|
||||
}
|
||||
if (failed) finishedAfterFailure += 1;
|
||||
}), /create failed/);
|
||||
assert.equal(running, 0, "every started worker finished before the error surfaced");
|
||||
assert.deepEqual(started, [1, 2, 3], "no item was started after the failure");
|
||||
assert.equal(finishedAfterFailure, 2);
|
||||
});
|
||||
|
||||
it("keeps state.json readable under overlapping saves", async () => {
|
||||
const dir = await tempDir();
|
||||
const store = fileStateStore(dir);
|
||||
const state = emptyState("scale-test-run", ORIGIN, 0);
|
||||
await Promise.all(Array.from({ length: 25 }, (_, index) => store.save({ ...state, agentIds: Array.from({ length: index + 1 }, (_, n) => `agent-${n}`) })));
|
||||
const saved = await store.load();
|
||||
assert.equal(saved!.agentIds.length, 25);
|
||||
assert.deepEqual(await readdir(dir), ["state.json"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("credentials", () => {
|
||||
it("never appear in output, results or state files", async () => {
|
||||
const out = await tempDir();
|
||||
const h = harness({
|
||||
metricsUrl: METRICS_URL, metrics: goodMetrics,
|
||||
failRequest: (method, route, count) => method === "GET" && route.endsWith("/sidebar-badges") && count === 1
|
||||
? { status: 400, body: `bad request with Authorization: Bearer ${TOKEN} and key ${PROVIDER_KEY}` } : null,
|
||||
}, { PAPERCLIP_SCALE_METRICS_URL: METRICS_URL, PAPERCLIP_SCALE_METRICS_TOKEN: METRICS_TOKEN });
|
||||
await main(baseArgs(out, ["--steps", "2"]), h.deps);
|
||||
const files = await readdir(path.join(out, "scale-test-run"));
|
||||
assert.ok(files.includes("result.json") && files.includes("state.json") && files.includes("SUMMARY.md"));
|
||||
const written = await Promise.all(files.map(file => readFile(path.join(out, "scale-test-run", file), "utf8")));
|
||||
const everything = [...h.out, ...h.err, ...written].join("\n");
|
||||
for (const secret of [TOKEN, PROVIDER_KEY, METRICS_TOKEN]) assert.ok(!everything.includes(secret), `output leaked ${secret.slice(0, 12)}`);
|
||||
assert.ok(h.fake.requests.some(request => request.body?.includes(PROVIDER_KEY)), "the key reaches the instance only through its secrets API");
|
||||
assert.ok(h.fake.requests.filter(request => request.body?.includes(PROVIDER_KEY)).every(request => new URL(request.url).pathname.endsWith("/secrets")));
|
||||
});
|
||||
|
||||
it("redacts bearer tokens and known values", () => {
|
||||
const redactor = new Redactor([TOKEN, null, "abc"]);
|
||||
assert.equal(redactor.text(`x ${TOKEN} y`), "x [redacted] y");
|
||||
assert.equal(redactor.text("Authorization: Bearer abcdefghijklmnop"), "Authorization: Bearer [redacted]");
|
||||
assert.equal(redactor.text("abc"), "abc");
|
||||
});
|
||||
|
||||
it("scores an empty outcome as an error", () => {
|
||||
assert.equal(verdictOf(null, "boom"), "error");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,140 @@
|
||||
import { ApiError, type PaperclipClient } from "./client.ts";
|
||||
import type { AgentProfile, Options } from "./config.ts";
|
||||
import { agentName, companyName, mapLimit, projectName, secretPrefix, type RunState, type StateStore } from "./state.ts";
|
||||
|
||||
type Json = Record<string, any>;
|
||||
|
||||
export class SetupError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = "SetupError";
|
||||
}
|
||||
}
|
||||
|
||||
export async function resolveCompany(api: PaperclipClient, options: Pick<Options, "companyId" | "label">, state: RunState, store: StateStore): Promise<void> {
|
||||
if (options.companyId) {
|
||||
const me = await api.get<{ companyIds?: string[] }>("/api/cli-auth/me");
|
||||
if (!(me.companyIds ?? []).includes(options.companyId)) throw new SetupError(`The signed-in board user is not a member of company ${options.companyId}`);
|
||||
state.companyId = options.companyId;
|
||||
await store.save(state);
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const company = await api.post<{ id: string }>("/api/companies", {
|
||||
name: companyName(options.label),
|
||||
description: `Scale ramp ${options.label}. Created and archived by scripts/scale-agents-ramp.ts.`,
|
||||
});
|
||||
state.companyId = String(company.id);
|
||||
state.companyCreated = true;
|
||||
await store.save(state);
|
||||
} catch (error) {
|
||||
if (error instanceof ApiError && error.status === 403) {
|
||||
throw new SetupError(`This instance does not let the harness create a company (${error.code ?? "forbidden"}); pass --company <id> or PAPERCLIP_SCALE_COMPANY_ID`);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
export async function pickEnvironment(api: PaperclipClient, companyId: string, requested: string | null, allowLocal: boolean): Promise<string> {
|
||||
const environments = await api.get<Json[]>(`/api/companies/${companyId}/environments`);
|
||||
if (requested) {
|
||||
if (!environments.some(environment => environment.id === requested)) throw new SetupError(`Environment ${requested} is not in the company`);
|
||||
return requested;
|
||||
}
|
||||
const active = environments.filter(environment => environment.status !== "archived" && (allowLocal || environment.driver !== "local"));
|
||||
const sandboxes = active.filter(environment => environment.driver === "sandbox");
|
||||
const candidates = sandboxes.length ? sandboxes : active;
|
||||
if (candidates.length !== 1) {
|
||||
throw new SetupError(`The company has ${candidates.length} usable environments; pass --environment <id>`);
|
||||
}
|
||||
return String(candidates[0]!.id);
|
||||
}
|
||||
|
||||
export const AGENT_INSTRUCTIONS = [
|
||||
"You are a load-test worker for a Paperclip scale ramp.",
|
||||
"Follow the assigned task literally and in order. Keep every reply short.",
|
||||
"Use only the runtime's real tool-call channel for tools. Never print tool-call markup as text.",
|
||||
"Never print, store or expose credential values. Never create other tasks, documents or agents.",
|
||||
].join("\n");
|
||||
|
||||
export function agentPayload(input: { label: string; index: number; environmentId: string; profile: AgentProfile; secretId: string | null }): Json {
|
||||
const { profile } = input;
|
||||
const env = profile.credentialKey && input.secretId
|
||||
? { [profile.credentialKey]: { type: "secret_ref", secretId: input.secretId, version: "latest" } }
|
||||
: {};
|
||||
return {
|
||||
name: agentName(input.label, input.index),
|
||||
role: "qa",
|
||||
title: "Scale ramp worker",
|
||||
capabilities: "Runs one small deterministic task at a time for a load test.",
|
||||
adapterType: "paperclip_runner",
|
||||
adapterConfig: {
|
||||
provider: profile.provider,
|
||||
model: profile.model,
|
||||
lifecycleMode: "warm",
|
||||
idleTimeoutMs: 300_000,
|
||||
...(profile.provider === "opencode" ? { opencodePermissionMode: "allow" } : {}),
|
||||
env,
|
||||
...profile.adapterOverrides,
|
||||
},
|
||||
defaultEnvironmentId: input.environmentId,
|
||||
budgetMonthlyCents: 0,
|
||||
runtimeConfig: { heartbeat: { enabled: false, maxConcurrentRuns: 1 } },
|
||||
instructionsBundle: { entryFile: "AGENTS.md", files: { "AGENTS.md": AGENT_INSTRUCTIONS } },
|
||||
};
|
||||
}
|
||||
|
||||
export async function prepareCompany(input: {
|
||||
api: PaperclipClient;
|
||||
options: Options;
|
||||
state: RunState;
|
||||
store: StateStore;
|
||||
credential: string | null;
|
||||
log: (line: string) => void;
|
||||
}): Promise<void> {
|
||||
const { api, options, state, store } = input;
|
||||
await resolveCompany(api, options, state, store);
|
||||
const companyId = state.companyId!;
|
||||
state.environmentId = await pickEnvironment(api, companyId, options.environmentId, options.target.kind === "local");
|
||||
await store.save(state);
|
||||
const profile = options.profile;
|
||||
if (profile.credentialKey && !profile.providerSecretId) {
|
||||
if (!input.credential) throw new SetupError(`Set ${profile.credentialKey} in the environment, or PAPERCLIP_SCALE_PROVIDER_SECRET_ID to reuse a stored secret`);
|
||||
const secret = await api.post<{ id: string }>(`/api/companies/${companyId}/secrets`, {
|
||||
name: `${secretPrefix(options.label)}${profile.credentialKey}`,
|
||||
value: input.credential,
|
||||
description: `Scale ramp ${options.label}; deleted at cleanup`,
|
||||
});
|
||||
state.secretId = String(secret.id);
|
||||
await store.save(state);
|
||||
}
|
||||
const project = await api.post<{ id: string }>(`/api/companies/${companyId}/projects`, {
|
||||
name: projectName(options.label),
|
||||
description: `Scale ramp ${options.label}. Created and archived by scripts/scale-agents-ramp.ts.`,
|
||||
});
|
||||
state.projectId = String(project.id);
|
||||
await store.save(state);
|
||||
input.log(`Prepared company ${companyId}: environment ${state.environmentId}, project ${state.projectId}`);
|
||||
}
|
||||
|
||||
export async function ensureAgents(input: {
|
||||
api: PaperclipClient;
|
||||
options: Options;
|
||||
state: RunState;
|
||||
store: StateStore;
|
||||
count: number;
|
||||
}): Promise<void> {
|
||||
const { api, options, state, store } = input;
|
||||
const secretId = options.profile.providerSecretId ?? state.secretId;
|
||||
const start = state.agentIds.length;
|
||||
const indices = Array.from({ length: Math.max(0, input.count - start) }, (_, offset) => start + offset);
|
||||
await mapLimit(indices, AGENT_CREATE_PARALLELISM, async index => {
|
||||
const created = await api.post<{ id: string }>(`/api/companies/${state.companyId}/agents`, agentPayload({
|
||||
label: options.label, index, environmentId: state.environmentId!, profile: options.profile, secretId,
|
||||
}));
|
||||
state.agentIds.push(String(created.id));
|
||||
await store.save(state);
|
||||
});
|
||||
}
|
||||
|
||||
export const AGENT_CREATE_PARALLELISM = 8;
|
||||
@@ -0,0 +1,93 @@
|
||||
import { mkdir, readFile, rename, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
|
||||
export interface RunState {
|
||||
schema: "paperclip.scale-ramp.state/v1";
|
||||
label: string;
|
||||
baseUrl: string;
|
||||
companyId: string | null;
|
||||
companyCreated: boolean;
|
||||
environmentId: string | null;
|
||||
projectId: string | null;
|
||||
secretId: string | null;
|
||||
agentIds: string[];
|
||||
taskIds: string[];
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export const companyName = (label: string) => `Scale ramp ${label}`;
|
||||
export const projectName = (label: string) => `Scale ramp ${label}`;
|
||||
export const agentPrefix = (label: string) => `SR ${label} `;
|
||||
export const agentName = (label: string, index: number) => `${agentPrefix(label)}${String(index + 1).padStart(4, "0")}`;
|
||||
export const secretPrefix = (label: string) => `scale-ramp ${label} `;
|
||||
export const taskTag = (label: string) => `[${label}]`;
|
||||
|
||||
export const statePath = (dir: string) => path.join(dir, "state.json");
|
||||
|
||||
export function emptyState(label: string, baseUrl: string, now: number): RunState {
|
||||
return {
|
||||
schema: "paperclip.scale-ramp.state/v1", label, baseUrl, companyId: null, companyCreated: false, environmentId: null,
|
||||
projectId: null, secretId: null, agentIds: [], taskIds: [], createdAt: new Date(now).toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
export interface StateStore {
|
||||
save(state: RunState): Promise<void>;
|
||||
load(): Promise<RunState | null>;
|
||||
}
|
||||
|
||||
export function fileStateStore(dir: string): StateStore {
|
||||
let queue: Promise<void> = Promise.resolve();
|
||||
const write = async (body: string) => {
|
||||
await mkdir(dir, { recursive: true, mode: 0o700 });
|
||||
const temporary = `${statePath(dir)}.${process.pid}.tmp`;
|
||||
await writeFile(temporary, body, { mode: 0o600 });
|
||||
await rename(temporary, statePath(dir));
|
||||
};
|
||||
return {
|
||||
save(state) {
|
||||
const body = `${JSON.stringify(state, null, 2)}\n`;
|
||||
const next = queue.then(() => write(body));
|
||||
queue = next.catch(() => undefined);
|
||||
return next;
|
||||
},
|
||||
async load() {
|
||||
try {
|
||||
return JSON.parse(await readFile(statePath(dir), "utf8")) as RunState;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function memoryStateStore(initial: RunState | null = null): StateStore & { current: RunState | null } {
|
||||
const store = {
|
||||
current: initial ? structuredClone(initial) : null,
|
||||
async save(state: RunState) {
|
||||
store.current = structuredClone(state);
|
||||
},
|
||||
async load() {
|
||||
return store.current ? structuredClone(store.current) : null;
|
||||
},
|
||||
};
|
||||
return store;
|
||||
}
|
||||
|
||||
export async function mapLimit<T>(items: readonly T[], limit: number, run: (item: T) => Promise<void>): Promise<void> {
|
||||
let next = 0;
|
||||
let failure: { error: unknown } | null = null;
|
||||
const workers = Array.from({ length: Math.min(limit, items.length) }, async () => {
|
||||
while (!failure && next < items.length) {
|
||||
const item = items[next]!;
|
||||
next += 1;
|
||||
try {
|
||||
await run(item);
|
||||
} catch (error) {
|
||||
failure ??= { error };
|
||||
}
|
||||
}
|
||||
});
|
||||
await Promise.all(workers);
|
||||
if (failure) throw (failure as { error: unknown }).error;
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
export interface Distribution {
|
||||
n: number;
|
||||
p50: number | null;
|
||||
p90: number | null;
|
||||
p95: number | null;
|
||||
p99: number | null;
|
||||
max: number | null;
|
||||
}
|
||||
|
||||
export function percentile(sorted: readonly number[], p: number): number | null {
|
||||
if (!sorted.length) return null;
|
||||
const rank = Math.ceil((p / 100) * sorted.length) - 1;
|
||||
return sorted[Math.min(sorted.length - 1, Math.max(0, rank))]!;
|
||||
}
|
||||
|
||||
export function distribution(values: readonly number[]): Distribution {
|
||||
const sorted = values.filter(value => Number.isFinite(value)).sort((a, b) => a - b);
|
||||
return {
|
||||
n: sorted.length,
|
||||
p50: percentile(sorted, 50),
|
||||
p90: percentile(sorted, 90),
|
||||
p95: percentile(sorted, 95),
|
||||
p99: percentile(sorted, 99),
|
||||
max: sorted.length ? sorted[sorted.length - 1]! : null,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"noEmit": true,
|
||||
"rootDir": "../..",
|
||||
"allowImportingTsExtensions": true,
|
||||
"erasableSyntaxOnly": true,
|
||||
"verbatimModuleSyntax": true,
|
||||
"types": ["node"],
|
||||
"typeRoots": ["../../cli/node_modules/@types"]
|
||||
},
|
||||
"include": ["./**/*.ts", "../scale-agents-ramp.ts"]
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
export interface RunSpan {
|
||||
name: string;
|
||||
startOffsetMs: number;
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
export interface RunEvent {
|
||||
eventType: string;
|
||||
createdAt: string;
|
||||
payload?: Record<string, unknown> | null;
|
||||
}
|
||||
|
||||
export function taskNonce(label: string, index: number): string {
|
||||
return createHash("sha256").update(`${label}:${index}`).digest("hex").slice(0, 10);
|
||||
}
|
||||
|
||||
export function taskTitle(tag: string, nonce: string): string {
|
||||
return `${tag} Scale ramp task ${nonce}`;
|
||||
}
|
||||
|
||||
export function taskBrief(nonce: string): string {
|
||||
return [
|
||||
`Scale ramp task ${nonce}. Do these steps in order and nothing else:`,
|
||||
`1. Write one short sentence saying you are starting task ${nonce}.`,
|
||||
"2. Run the shell command `pwd`, then run `ls`.",
|
||||
`3. Write a file named \`scale-${nonce}.txt\` in the workspace root containing the single line \`${nonce}\`.`,
|
||||
`4. Run the shell command \`cat scale-${nonce}.txt\`.`,
|
||||
`5. Post a comment on this task that says \`CHECKPOINT ${nonce}\`.`,
|
||||
`6. Post a final comment that says \`DONE ${nonce}\`, then mark this task done.`,
|
||||
"Do not create other tasks, documents or files.",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function spans(events: readonly RunEvent[]): RunSpan[] {
|
||||
const out: RunSpan[] = [];
|
||||
for (const event of events) {
|
||||
if (event.eventType !== "run.performance.span" || !event.payload) continue;
|
||||
const { span, startOffsetMs, durationMs } = event.payload as { span?: unknown; startOffsetMs?: unknown; durationMs?: unknown };
|
||||
if (typeof span === "string" && typeof startOffsetMs === "number" && typeof durationMs === "number") out.push({ name: span, startOffsetMs, durationMs });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export interface RunTiming {
|
||||
ttftMs: number | null;
|
||||
firstActivityMs: number | null;
|
||||
}
|
||||
|
||||
const ACTIVITY_EVENTS = new Set(["item.delta", "tool.execution.started", "item.started"]);
|
||||
|
||||
export function runTiming(events: readonly RunEvent[], runCreatedAt: number | null): RunTiming {
|
||||
const all = spans(events);
|
||||
const first = (name: string) => all.find(span => span.name === name) ?? null;
|
||||
const queue = first("heartbeat.queue");
|
||||
const fromRunCreated = (span: RunSpan) => queue ? Math.max(0, span.startOffsetMs + span.durationMs - queue.startOffsetMs) : span.durationMs;
|
||||
const fallback = (types: ReadonlySet<string>) => {
|
||||
if (runCreatedAt === null) return null;
|
||||
const at = events.filter(event => types.has(event.eventType)).map(event => Date.parse(event.createdAt)).filter(Number.isFinite);
|
||||
return at.length ? Math.max(0, Math.min(...at) - runCreatedAt) : null;
|
||||
};
|
||||
const ttft = first("task.time_to_first_token");
|
||||
const activity = first("task.time_to_first_agent_activity");
|
||||
return {
|
||||
ttftMs: ttft ? fromRunCreated(ttft) : fallback(new Set(["item.delta"])),
|
||||
firstActivityMs: activity ? fromRunCreated(activity) : fallback(ACTIVITY_EVENTS),
|
||||
};
|
||||
}
|
||||
@@ -52,6 +52,7 @@ export const PROGRAMS = [
|
||||
"packages/plugins/plugin-workspace-diff/tsconfig.test.json",
|
||||
"packages/plugins/sdk/tsconfig.test.json",
|
||||
...SANDBOX_PROVIDERS.map((directory) => `${directory}/tsconfig.test.json`),
|
||||
"scripts/scale-ramp/tsconfig.json",
|
||||
"tests/runner-acceptance/tsconfig.json",
|
||||
"tests/tsconfig.playwright.json",
|
||||
];
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"inputs": {
|
||||
"package.json": "590fd8f555086a3bdc4dfa41a8016a89be13b977",
|
||||
"package.json": "2d2a4abebb6a5931be1a8f0c8fb2615890324d21",
|
||||
"packages/plugins/sdk/package.json": "b7a4936e2803fd0d3b32dd7f17fc1ea86609ead0",
|
||||
"packages/plugins/sdk/src/bundlers.ts": "883bf3d9f2bd42877cfc0c3707cd882a6f6248d0",
|
||||
"packages/plugins/sdk/src/define-plugin.ts": "a99e07ce77ef72fcf1df1347346aacb8ba12e284",
|
||||
|
||||
Reference in new issue
Block a user