mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The scaling effort uses a real-agent ramp harness (`scripts/scale-agents-ramp.ts`, #21695 and #21757) to load a hosted test instance through the product API > - Its first run with agents that really ran (10 agents, 105 tasks, 106 runs) stopped on one failed run, recorded $0 model spend, and left cleanup errors > - Read-only inspection of the instance showed three harness faults. The failed run had a same-run bootstrap retry scheduled 33 s later, but the harness stopped and cleaned up first. Run rows carry no cost, but cost records do (63.7 cents for the run's agents). Terminated agents leave the company agent list, so the harness treated them as done and sent deletes that Paperclip refused with 409 > - These faults make the ramp stop too early, make its spend cap inert, and make cleanup report the wrong cause > - This pull request keeps the ramp going through runs that Paperclip recovers by itself, reads spend from cost records, records board latency per route, and makes cleanup wait for termination and name what it left behind > - The benefit is that the next ramp grades the product, not the harness, and its spend cap works ## Linked Issues or Issue Description Refs #21695 Refs #21757 **What happened?** The ramp stopped at step 1 with `failed_runs_zero failed (observed 1)`. The run failed with `runner_ingress_bind_conflict` before its provider session opened. Its run events show `same-run resume persisted` with `recoverable: true` and `nextAttemptAt` 33 s later. The harness stopped within 9 s and paused the agents, so the retry was blocked (`native_retry_admission_ineligible`). The result showed `Recorded model spend: $0`. The harness read `costUsd` from run rows, which is always null; `GET /api/companies/<id>/costs/by-agent` holds 63.71 cents over 104 cost events for the 10 agents. Cleanup reported two `DELETE /api/agents/:id returned 409: Complete termination before deleting the agent`. Those agents were still `lifecycleState: terminating`; the harness waited on the company agent list, which hides terminated agents, so it treated them as done. A later `--cleanup-only` changed nothing because it also found agents only through that list. The board API p90 failed (765 ms), but the result had no per-route timings to say which request was slow. **Expected behavior** A failed run whose task finishes by itself is counted, not a stop. A failed run whose task does not finish within a grace stops the ramp. The share of failed runs is graded at the step end. Spend comes from Paperclip's cost records. Cleanup deletes an agent only after termination completes, and names the agents and sandbox leases still pending. **Steps to reproduce** 1. Run `node --test scripts/scale-ramp/scale-ramp.test.ts` with the new fake and tests and the `origin/master` `ramp.ts`, `cleanup.ts`, `criteria.ts` and `report.ts`. 2. 16 tests fail, among them every new test (recovered run, unrecovered run, spend from cost records, spend cap, board per route, termination wait, still-terminating agents and `--cleanup-only`, leases released during the wait). 3. With this change, all 60 pass. **Paperclip version or commit** Instance on `2026.1106.0+2564.git.3244780bd`; harness from #21757. ## What Changed - `criteria.ts`: replace `failed_runs_zero` with `failed_runs_unrecovered_zero` (immediate) and `failed_run_rate` (at the step end). Step metrics add `runsEnded`, `runsFailedRecovered`, `runsFailedUnrecovered`, `runsFailedPending` and `board.byRoute`. - `ramp.ts`: a run counts as failed once it was seen `failed` or `timed_out`. It is recovered when its task reaches `done`. It is unrecovered when the task rests not done, needs Retry, or has no live run for `--recovery-grace-minutes` after the failure. A task whose interruption has an automatic retry scheduled (`state: retry_scheduled`, `nextRetryAt` or `retryWait`) does not count as needing Retry. Spend comes from `costs/by-agent` for the run's agents on each poll, and each step reports the spend in its window. Board probes keep a distribution per route. - `config.ts`: add `--max-failed-run-percent` (default 2) and `--recovery-grace-minutes` (default 15). - `cleanup.ts`: read agents by id as well as from the list, so terminated agents in the state file are found. Wait for `lifecycleState` `terminated` or `rejected` by id before a delete. Report `agentsStillTerminating`, `activeSandboxLeaseIds` and `sandboxLeasesReleasedDuringWait`, and name each lease still active with its run and acquire time. - `report.ts`: the summary shows recovered, unrecovered and pending failed runs, board latency per route, and each cleanup error. - `fake-paperclip.ts`: run rows carry no cost; `costs/by-agent` returns cents per agent. Optional failure of the n-th run, a scheduled recovery run, a termination delay (terminated agents leave the list and a delete answers 409 until termination completes), and leases that stay active after their run. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the runbook describes the failed-run criteria, the spend source and the cleanup behavior. ## Verification - `node --test scripts/scale-ramp/scale-ramp.test.ts`: 60 of 60 pass. - Red check: with the new fake and tests but the `origin/master` `ramp.ts`, `cleanup.ts`, `criteria.ts` and `report.ts`, 16 tests fail, including all 8 new ones. - `tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - Evidence from read-only GETs on the test instance: run `f1892d43` events (bind conflict, scheduled same-run retry, blocked after the pause), run rows (`costUsd` null on all 106), `costs/by-agent` (63.71 cents for the 10 agents), agent rows by id (`terminating` while absent from the list), lease rows. - Not run: a new real ramp (needs the owner's go-ahead). ## Risks - Low risk. The change touches only the scale-ramp harness, its fake, its tests and one runbook section. No product code changes. - A failed run no longer stops the ramp at once. An unrecovered one stops it after the grace (15 min by default), so a broken step can run up to that long. The step-end failed-run rate still fails a step with more than 2% failed runs. - Spend now needs cost read access for the board token. If `costs/by-agent` cannot be read, the result reports spend as not measured and the spend cap does not trigger. - Cleanup can now take up to 5 more minutes while it waits for termination by id. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, read-only HTTP GETs to the test instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
179 lines
9.6 KiB
TypeScript
179 lines
9.6 KiB
TypeScript
import type { CleanupReport } from "./cleanup.ts";
|
|
import type { Options } from "./config.ts";
|
|
import { criteriaTable, type CriterionResult } from "./criteria.ts";
|
|
import type { SpendEstimate } from "./estimate.ts";
|
|
import type { RampOutcome } from "./ramp.ts";
|
|
import type { Distribution } from "./stats.ts";
|
|
|
|
export type Verdict = "pass" | "fail" | "incomplete" | "interrupted" | "error";
|
|
|
|
export interface RampResult {
|
|
schema: "paperclip.scale-ramp.result/v1";
|
|
runId: string;
|
|
target: { baseUrl: string; host: string; kind: string };
|
|
startedAt: string;
|
|
finishedAt: string;
|
|
plan: {
|
|
steps: Array<{ concurrency: number; holdMinutes: number }>;
|
|
drainMinutes: number;
|
|
pollSeconds: number;
|
|
boardProbeSeconds: number;
|
|
agent: { provider: string; model: string; credential: string };
|
|
estimate: SpendEstimate;
|
|
};
|
|
criteria: ReturnType<typeof criteriaTable>;
|
|
verdict: Verdict;
|
|
stop: RampOutcome["stop"];
|
|
steps: RampOutcome["steps"];
|
|
notMeasured: string[];
|
|
incompleteReasons: string[];
|
|
drained: boolean;
|
|
openTasksAtEnd: number;
|
|
tasksCreated: number;
|
|
taskCapReached: boolean;
|
|
recordedSpendUsd: number | null;
|
|
unpricedCostEvents: number;
|
|
sources: Record<string, string>;
|
|
cleanup: CleanupReport | null;
|
|
requests: { total: number; failed: number };
|
|
error: string | null;
|
|
}
|
|
|
|
export function verdictOf(outcome: RampOutcome | null, error: string | null): Verdict {
|
|
if (error || !outcome) return "error";
|
|
if (outcome.stop && outcome.stop.criterion) return "fail";
|
|
if (outcome.stop) return "interrupted";
|
|
if (outcome.steps.some(step => step.criteriaFinal.some(result => result.status === "fail"))) return "fail";
|
|
if (incompleteReasons(outcome).length) return "incomplete";
|
|
return "pass";
|
|
}
|
|
|
|
export function incompleteReasons(outcome: RampOutcome | null): string[] {
|
|
if (!outcome) return [];
|
|
const reasons: string[] = [];
|
|
const missing = notMeasured(outcome);
|
|
if (missing.length) reasons.push(`criteria not measured: ${missing.join(", ")}`);
|
|
if (outcome.taskCapReached) reasons.push("the task cap was reached, so a step may have run below its concurrency");
|
|
if (outcome.recordedSpendUsd === null) reasons.push("recorded model spend could not be read at the end of the run");
|
|
if (outcome.unpricedCostEvents > 0) {
|
|
reasons.push(`${outcome.unpricedCostEvents} cost events have no known price, so recorded spend is a lower bound and the spend cap does not cover them`);
|
|
}
|
|
if (outcome.openTasksAtEnd > 0) reasons.push(`${outcome.openTasksAtEnd} tasks were still open when the drain ended`);
|
|
return reasons;
|
|
}
|
|
|
|
export function notMeasured(outcome: RampOutcome | null): string[] {
|
|
const ids = new Set<string>();
|
|
for (const step of outcome?.steps ?? []) for (const result of step.criteriaFinal) if (result.status === "not_measured") ids.add(result.id);
|
|
return [...ids];
|
|
}
|
|
|
|
export function buildResult(input: {
|
|
options: Options;
|
|
estimate: SpendEstimate;
|
|
startedAt: number;
|
|
finishedAt: number;
|
|
outcome: RampOutcome | null;
|
|
cleanup: CleanupReport | null;
|
|
error: string | null;
|
|
requests: { total: number; failed: number };
|
|
credentialSource: string;
|
|
}): RampResult {
|
|
const { options, outcome } = input;
|
|
return {
|
|
schema: "paperclip.scale-ramp.result/v1",
|
|
runId: options.label,
|
|
target: { baseUrl: options.target.baseUrl, host: options.target.host, kind: options.target.kind },
|
|
startedAt: new Date(input.startedAt).toISOString(),
|
|
finishedAt: new Date(input.finishedAt).toISOString(),
|
|
plan: {
|
|
steps: options.steps.map(step => ({ concurrency: step.concurrency, holdMinutes: step.holdMs / 60_000 })),
|
|
drainMinutes: options.drainMs / 60_000,
|
|
pollSeconds: options.pollMs / 1_000,
|
|
boardProbeSeconds: options.boardProbeMs / 1_000,
|
|
agent: { provider: options.profile.provider, model: options.profile.model, credential: input.credentialSource },
|
|
estimate: input.estimate,
|
|
},
|
|
criteria: criteriaTable(options.thresholds),
|
|
verdict: verdictOf(outcome, input.error),
|
|
stop: outcome?.stop ?? null,
|
|
steps: outcome?.steps ?? [],
|
|
notMeasured: notMeasured(outcome),
|
|
incompleteReasons: verdictOf(outcome, input.error) === "incomplete" ? incompleteReasons(outcome) : [],
|
|
drained: outcome?.drained ?? false,
|
|
openTasksAtEnd: outcome?.openTasksAtEnd ?? 0,
|
|
tasksCreated: outcome?.tasksCreated ?? 0,
|
|
taskCapReached: outcome?.taskCapReached ?? false,
|
|
recordedSpendUsd: outcome ? outcome.recordedSpendUsd : null,
|
|
unpricedCostEvents: outcome?.unpricedCostEvents ?? 0,
|
|
sources: {
|
|
runsAndTasks: outcome?.runsListTruncated ? "API (company run list was full; open tasks without a listed run were read one by one)" : "API",
|
|
boardLatency: "harness GET mix (dashboard, issues, agents, live runs, sidebar badges), with a distribution per route",
|
|
modelSpend: outcome?.spendReadError ? `not measured: the last read of costs by agent failed (${outcome.spendReadError})`
|
|
: `costs by agent for the run's agents over all time (Paperclip's recorded cost events)${outcome?.unpricedCostEvents
|
|
? `; ${outcome.unpricedCostEvents} events have no known price and are not included` : ""}`,
|
|
firstTokenAndActivity: `run events of up to ${options.eventSamplePerStep} succeeded runs per step`,
|
|
serverMetrics: outcome?.metricsSource === "endpoint" ? "PAPERCLIP_SCALE_METRICS_URL"
|
|
: outcome?.metricsSource === "unavailable" ? "PAPERCLIP_SCALE_METRICS_URL gave no readable sample: not measured" : "not provided: not measured",
|
|
database: "not read by this harness: not measured",
|
|
},
|
|
cleanup: input.cleanup,
|
|
requests: input.requests,
|
|
error: input.error,
|
|
};
|
|
}
|
|
|
|
const seconds = (value: number | null) => value === null ? "not measured" : `${(value / 1_000).toFixed(1)} s`;
|
|
const millis = (value: number | null) => value === null ? "not measured" : `${Math.round(value)} ms`;
|
|
const dist = (d: Distribution, format: (value: number | null) => string) => d.n === 0 ? "not measured" : `${format(d.p50)} / ${format(d.p95)} / ${format(d.p99)} (n=${d.n})`;
|
|
|
|
function criterionLine(result: CriterionResult): string {
|
|
const observed = result.observed === null ? "not measured" : String(result.observed);
|
|
return ` - ${result.status === "pass" ? "PASS" : result.status === "fail" ? "FAIL" : "not measured"} ${result.id}: ${observed} (needs ${result.threshold})`;
|
|
}
|
|
|
|
export function summaryLines(result: RampResult): string[] {
|
|
const lines = [
|
|
`# Scale ramp ${result.runId}`,
|
|
"",
|
|
`Verdict: **${result.verdict}**. Target ${result.target.baseUrl} (${result.target.kind}).`,
|
|
result.stop ? `Stopped at step ${result.stop.stepIndex + 1}: ${result.stop.reason}.` : "No stop.",
|
|
`Tasks created: ${result.tasksCreated}${result.taskCapReached ? " (task cap reached)" : ""}. Recorded model spend: ${result.recordedSpendUsd === null ? "not measured" : `$${result.recordedSpendUsd}`}`
|
|
+ `${result.unpricedCostEvents ? ` (incomplete: ${result.unpricedCostEvents} cost events have no known price)` : ""}; estimate was $${result.plan.estimate.totalUsd}.`,
|
|
"",
|
|
];
|
|
for (const step of result.steps) {
|
|
const m = step.metrics;
|
|
lines.push(
|
|
`## Step ${step.index + 1}: ${step.concurrency} agents, ${step.holdMs / 60_000} min`,
|
|
"",
|
|
`- Tasks created / started / finished / failed / needed Retry: ${m.tasksCreated} / ${m.tasksStarted} / ${m.tasksFinished} / ${m.tasksFailed} / ${m.tasksNeedingRetry}`,
|
|
`- Runs ended / failed / interrupted / cancelled: ${m.runsEnded} / ${m.runsFailed} / ${m.runsInterrupted} / ${m.runsCancelled}`,
|
|
`- Failed runs whose task finished by itself / did not / still pending: ${m.runsFailedRecovered} / ${m.runsFailedUnrecovered} / ${m.runsFailedPending}`,
|
|
`- Running agents peak / mean: ${m.peakRunning} / ${m.meanRunning ?? "not measured"}`,
|
|
`- Queue wait p50 / p95 / p99: ${dist(m.queueWaitMs, seconds)}`,
|
|
`- Time to first token p50 / p95 / p99: ${dist(m.ttftMs, seconds)}`,
|
|
`- First activity p50 / p95 / p99: ${dist(m.firstActivityMs, seconds)}`,
|
|
`- Run duration p50 / p95 / p99: ${dist(m.runDurationMs, seconds)}`,
|
|
`- Board API p50 / p95 / p99: ${dist(m.board.ms, millis)}; failed ${m.board.failures} of ${m.board.requests}`,
|
|
...Object.entries(m.board.byRoute).map(([route, d]) => ` - ${route} p50 / p90 / p99: ${millis(d.p50)} / ${millis(d.p90)} / ${millis(d.p99)} (n=${d.n})`),
|
|
`- Event-loop utilization max: ${m.server.eventLoopUtilizationMax ?? "not measured"}; delay p99 max: ${millis(m.server.eventLoopDelayP99MsMax)}; statements/s max: ${m.server.statementsPerSecondMax ?? "not measured"}; pool wait p99 max: ${millis(m.server.poolWaitP99MsMax)}`,
|
|
"- Criteria (final):",
|
|
...step.criteriaFinal.map(criterionLine),
|
|
"",
|
|
);
|
|
}
|
|
if (result.notMeasured.length) lines.push(`Not measured: ${result.notMeasured.join(", ")}.`, "");
|
|
if (result.incompleteReasons.length) lines.push(`Incomplete because: ${result.incompleteReasons.join("; ")}.`, "");
|
|
lines.push(`Drain: ${result.drained ? "every task finished" : `${result.openTasksAtEnd} tasks still open`}.`, "");
|
|
if (result.cleanup) {
|
|
const c = result.cleanup;
|
|
lines.push(`Cleanup in ${seconds(c.durationMs)}: ${c.runsCancelled} runs cancelled, ${c.tasksArchived} tasks archived, ${c.agentsDeleted} agents deleted`
|
|
+ ` (${c.agentsKeptForCostHistory} kept terminated for cost history, ${c.agentsStillTerminating.length} still terminating),`
|
|
+ ` sandbox leases still active: ${c.activeSandboxLeases ?? "not readable"} (${c.sandboxLeasesReleasedDuringWait} released while the cleanup waited), errors: ${c.errors.length}.`);
|
|
for (const error of c.errors) lines.push(`- ${error}`);
|
|
}
|
|
if (result.error) lines.push(`Error: ${result.error}`);
|
|
return lines;
|
|
}
|