From bbed32450c9807d9f7be992f26c4810b13080ae7 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 23:13:07 -0600 Subject: [PATCH] perf(load-model): model recorded native run rates, the recorded event mix and coalesced delta batches (#21549) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `scripts/benchmark-agent-scaling-load.ts` (#21418) estimates how many concurrent agents one server process can carry > - It assumed 20 text deltas/s and 1 other event/s per agent, and every other event was an `item.completed` > - Recorded hosted native runs show about 0.2 deltas/s and 0.5 other events/s, with a mix in which 39% of the other events are server trace spans that take a cheaper path > - This pull request makes the recorded rates and mix the model's default, keeps the old rates as an explicit stress profile, adds a burst profile, and mirrors the delta batching of #21520 > - The benefit is a model that measures what production does, so the next optimizations target the real per-agent cost ## Linked Issues or Issue Description Refs #21418 (load model and plan). Refs #21520 (delta batching that the model now mirrors). **What existing behavior does this improve?** The load model's default rates, its event mix and its delta batching rule, and the plan document's measurements. **Subsystem affected** `scripts/benchmark-agent-scaling-load.ts`, its test `server/src/__tests__/agent-scaling-load-model.test.ts`, and `doc/plans/2026-11-13-thousand-concurrent-agents.md`. **Current behavior** The model defaults to 20 deltas/s and 1 other event/s per agent. Every other event is a native `item.completed` with a chat-progress wake and a checkpoint save. It cuts a delta batch on the timer even while the previous batch commits. **Proposed behavior** - `PROFILE=recorded` (default): 0.2 deltas/s and 0.5 other events/s per agent. `PROFILE=burst`: 2.5 and 3, a run's busiest minute. `PROFILE=stress`: the earlier 20 and 1. API requests stay at 0.3/s. Each rate can still be set on its own. - `EVENT_MIX=recorded` (default) sends other events in the recorded proportions. A `run.performance.span` is an ordinary append with no wake and no checkpoint, as the server's run trace writes it. `tool.execution.*` and `item.completed` are native appends with a chat-progress wake, then a checkpoint save. `EVENT_MIX=` sends one type only. - Delta batching follows #21520: deltas that arrive while a batch commits all join one next batch. - The report adds `profile` and `eventsIssuedByType`. **Reason and benefit** At the recorded rates the per-agent cost is the non-delta runner events and their checkpoint saves, not deltas. The plan and the next optimizations need that picture. **Breaking changes** None for the product. Anyone who runs the model with no options now gets the recorded profile; `PROFILE=stress` gives the earlier load. ## What Changed - Profiles, the recorded event mix (a 100-slot cycle per agent, offset by agent so the fleet sends the mix at every moment), the span append path, and the coalesced delta batches. - The model test now checks the profile in the report and that both server trace spans and runner events with a progress wake are in the mix, for the recorded profile and for the stress profile with one transaction per delta. - The plan has a new section "Load model at recorded rates": the recorded source (40 recent native runs on one hosted workspace, as aggregates), the profiles, the mix and its server paths, fresh runs at 500, 1,000 and 2,000 agents plus burst and stress, and what they show. It records the 3-connection checkpoint pool as a computed per-process limit on a hosted database, and that SQL building in Drizzle costs about as much main-thread time as result parsing on the hosted instance. A new section "Considered and not done" records why `run.performance.span` rows stay (about 3% of the statements per agent, with readers) and why checkpoint saves per critical event only are not done now. ## Verification - `server/src/__tests__/agent-scaling-load-model.test.ts` passes (2 tests). The new assertions fail against the previous script: it reports no `profile` and no `eventsIssuedByType`. - `server/src/__tests__/agent-scaling-load-model.test.ts` also checks each profile's rates with no overrides, and that a rate set on its own wins. - Load model runs, recorded in the plan (60 s, main pool of 10, embedded PostgreSQL on the same machine, each run started below a 1-minute load average of 15): recorded rates at 1,000 agents (event-loop utilization 0.61) and 2,000 (0.85) finish every operation with no failure; 500 agents all in their busiest minute run at 0.95 with p99 lateness under 0.4 s. About 4.4 statements per agent per second at recorded rates, 35% of them on the checkpoint pool. - Earlier runs at load averages of 20–55 had shown one process falling behind at 500 burst agents and at 2,000 recorded; the quieter runs after the model fixes (event order, spans off the runner stream) do not, and the plan says so. The checkpoint pool as a per-process limit is now stated as a computed limit for a hosted round trip (about 400 agents in a busy minute at 1 ms), not as measured. - Greptile findings fixed: a waiting delta batch no longer takes deltas sent after the next runner event, takes every delta queued when it starts, and spans no longer join the runner event stream. ## Risks - Low: a benchmark script, its test and a plan document. No product code changes. - The machine that measured the plan's table was shared; the plan marks the rows that the machine distorted. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../2026-11-13-thousand-concurrent-agents.md | 103 ++++++++++++++++- scripts/benchmark-agent-scaling-load.ts | 108 +++++++++++++++--- .../agent-scaling-load-model.test.ts | 28 ++++- 3 files changed, 216 insertions(+), 23 deletions(-) diff --git a/doc/plans/2026-11-13-thousand-concurrent-agents.md b/doc/plans/2026-11-13-thousand-concurrent-agents.md index c43512d9657..a17a2113b99 100644 --- a/doc/plans/2026-11-13-thousand-concurrent-agents.md +++ b/doc/plans/2026-11-13-thousand-concurrent-agents.md @@ -129,6 +129,104 @@ What it shows: - Lease renewal is under 1 statement/s at 400 agents (15/s at 50 before). - The model does not measure server memory, so it does not show change 1h. +## Load model at recorded rates + +The rates above were assumptions. Recorded hosted native runs show a much +lighter and different load, so the model's default changed on 2026-11-14. + +Source: 40 recent native runs on one hosted workspace (median 26 minutes). +Events stored per run: 0.69/s at the median, 1.5/s at p90. Text deltas: 0.18/s +on average, 1.6–2.7/s in a run's busiest minute; a run streams text for about +8% of its life. Rows by event type: `run.performance.span` 10,030, +`item.delta` 9,502, `tool.execution.progressed` 6,951, `item.completed` 4,705, +`tool.execution.started` and `tool.execution.completed` about 2,150 each. +Controller checkpoint rows: about 1.4 per running run per second. + +| Profile (`PROFILE`) | Deltas/s per agent | Other events/s per agent | Use | +| --- | --- | --- | --- | +| `recorded` (default) | 0.2 | 0.5 | median of the recorded runs | +| `burst` | 2.5 | 3 | a run's busiest minute | +| `stress` | 20 | 1 | the earlier assumption, kept for comparison | + +API requests stay at 0.3/s per agent in every profile. Other events follow the +recorded mix (`EVENT_MIX`), and each type takes the server path it takes in +production: + +| Event type | Share | Server path | +| --- | --- | --- | +| `run.performance.span` | 39% | written by the server's own run trace: one ordinary append, no wake, no checkpoint | +| `tool.execution.progressed` | 27% | runner event with a chat-progress wake: a transaction, then a controller checkpoint save | +| `item.completed` | 18% | the same (a final reply also wakes response presentation) | +| `tool.execution.started`, `tool.execution.completed` | 8% each | the same | + +The model's delta batching now also follows #21520: deltas that arrive while a +batch commits go into one next batch, until the run's next runner event. + +Measured 2026-11-14 on master (after #21511, #21520 and #21528), 60 s window, +main pool of 10 connections, embedded PostgreSQL on the same machine. Each run +started only when the machine's 1-minute load average was below 15; the load +average at the start and the end is shown. The machine is shared, and the +500-agent recorded run had other work start during its window (marked busy). + +| Agents | Profile | Load avg start / end | Event-loop utilization | Statements/s | Per agent per second | Checkpoint pool statements/s | Other events done / offered | Other event lateness p50 / p99 | Delta lateness p50 / p99 | API p50 / p99 | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 500 | recorded, busy | 14.9 / 22.2 | 0.66 | 1,979 | 4.0 | 688 | 14,774 / 15,000 | 3 ms / 5.2 s | 2 ms / 5.7 s | 3 ms / 7.1 s | +| 1,000 | recorded | 14.0 / 13.5 | 0.61 | 4,351 | 4.4 | 1,525 | 30,000 / 30,000 | 1 ms / 10 ms | 1 ms / 252 ms | 2 ms / 9 ms | +| 2,000 | recorded | 13.4 / 15.9 | 0.85 | 8,701 | 4.4 | 3,049 | 60,000 / 60,000 | 2 ms / 2.1 s | 1 ms / 2.2 s | 3 ms / 3.0 s | +| 250 | burst | 12.3 / 16.6 | 0.70 | 6,102 | 24.4 | 2,287 | 45,000 / 45,000 | 1 ms / 122 ms | 2 ms / 281 ms | 2 ms / 199 ms | +| 500 | burst | 13.8 / 11.7 | 0.95 | 12,114 | 24.2 | 4,488 | 90,000 / 90,000 | 4 ms / 171 ms | 9 ms / 318 ms | 5 ms / 268 ms | + +No run had a failed operation, a failed checkpoint save or a lost lease. + +What it shows: + +- At recorded rates one process costs about 4.4 statements per agent per + second, and about 35% of them are on the 3-connection checkpoint pool. The + non-delta runner events and their checkpoint saves are most of the cost; text + deltas are now a small share. +- On loopback PostgreSQL one process carries 2,000 agents at recorded rates + (event-loop utilization 0.85, p99 lateness about 2 s), and 500 agents all in + their busiest minute at the same time (0.95, p99 under 0.4 s). +- The 3-connection checkpoint pool kept up locally: about 4,500 statements/s, + or 1,100 saves/s, at 500 agents in a busy minute. Since #21528 a save is 4 + statements (BEGIN, the owner lock with the transaction id, the write, + COMMIT), and 5 where commit-time notifications are on (a `pg_notify` before + COMMIT). Each save holds a connection for those round trips, so with a hosted + round trip of about 1 ms the same 3 connections allow about 600–750 saves/s, + which is about 330–400 agents in a busy minute. That is a per-process limit + on a hosted database (change 1g sizes this pool); it is computed here, not + measured. +- Earlier runs on 2026-11-14 at load averages of 20–55 showed one process + falling behind at 500 agents in a busy minute and at 2,000 at recorded rates. + With the machine quieter and the model's event order corrected, both kept up, + so those earlier results described the machine, not the server code. +- The model saves a checkpoint without blocking the agent's next event. The + controller acknowledges a runner event only after its checkpoint save, so in + production a slow checkpoint pool also delays the run's next events. The + model therefore understates the effect of that pool. +- On the hosted instance a CPU profile of the main thread (2026-11-14, with + these fixes) shows building SQL text in Drizzle costing about as much + main-thread time as postgres.js result parsing (each about a fifth). A + statement with fixed SQL text, as in #21511, therefore saves more CPU there + than its statement count suggests; the next changes should be judged by CPU + per operation, not only by statements. + +## Considered and not done + +- **Stop writing `run.performance.span` rows.** These spans are the largest + event type by rows, but each is one statement with no wake and no + checkpoint. Recorded runs have about 190–250 span rows per run over a median + of 26 minutes, about 0.12–0.16 rows per run per second: about 3% of the + statements per agent. The top names are `tool.request.input_stream` (median + 56 per run), `tool.execute` (25), `execution.plugin_dispatch` (11) and + `authorization.base` (7). Run forensics, the restart-chaos speed scripts and + exports read these rows, so they stay. +- **Save a controller checkpoint only for critical runner events.** After the + routine checkpoint save becomes one statement, this would save about 0.3 + statements per agent per second at recorded rates. It would change the + acknowledgement order and the restore cursor of the runner's control plane, + which is recovery-critical. Not now; revisit with the writer-thread work. + ## Why more servers do not help today | Assumption | Where | Effect with two steady processes | @@ -212,8 +310,9 @@ turn, so memory per session counts for idle agents too. - The load model runs on loopback PostgreSQL on a developer machine. It measures the modelled sources only and has no hosted round-trip time. -- The per-agent rates (20 deltas/s, 1 other event/s, 0.3 API requests/s) are - assumptions, not a recorded workload. Phase 0 step 2 replaces them. +- The first tables used assumed rates (20 deltas/s, 1 other event/s). The + "Load model at recorded rates" section uses rates from recorded hosted runs; + the API rate (0.3/s) is still an assumption. - Staging's pool size and run-log storage mode were not read. - Tool-gateway calls, recovery sweeps, and browser fan-out are not in the model. diff --git a/scripts/benchmark-agent-scaling-load.ts b/scripts/benchmark-agent-scaling-load.ts index 407dba26e51..07e602bf6d7 100644 --- a/scripts/benchmark-agent-scaling-load.ts +++ b/scripts/benchmark-agent-scaling-load.ts @@ -6,10 +6,15 @@ * * deltas an `item.delta` per text chunk, committed as the server's port * commits them: the first after another event at once, later ones - * in one transaction per DELTA_BATCH_MS or 100 deltas - * (DELTA_BATCH_MS=0: one transaction per delta, as before batching) - * events appendHeartbeatRunEvent for an `item.completed`, then the real - * controller checkpoint save, which overlaps the next append + * together after DELTA_BATCH_MS or 100 deltas, and the deltas that + * arrive while a batch commits all in the next batch + * (DELTA_BATCH_MS=0: one append per delta, as before batching) + * events the recorded mix of other events (EVENT_MIX). A runner event + * (`tool.execution.*`, `item.completed`) is a native source append + * that registers a chat progress wake, then the real controller + * checkpoint save, which overlaps the next append. A + * `run.performance.span` is written by the server's own run trace: + * an ordinary append with no wake and no checkpoint * lease every LEASE_S, one renewal of all leases through the shared * schedule (LEASE_MODE=single: one transaction per run, as before). * In batch mode LEASE_S is at most 20: the schedule renews a remote @@ -33,10 +38,17 @@ * * pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-agent-scaling-load.ts * + * PROFILE sets the per-agent rates: `recorded` (default) is the median of + * recorded hosted native runs, 0.2 deltas/s and 0.5 other events/s; `burst` + * is a run's busiest minute, 2.5 deltas/s and 3 other events/s; `stress` is + * the earlier assumption, 20 deltas/s and 1 other event/s. API_PER_S is 0.3 + * in every profile. Each rate can still be set on its own. + * * Optional: AGENTS (50), SECONDS (20), SOURCES (deltas,events,lease,api), - * DELTAS_PER_S (20), EVENTS_PER_S (1), API_PER_S (0.3), LEASE_S (20), - * DELTA_BYTES (120), EVENT_KIB (2), DELTA_BATCH_MS (250), LEASE_MODE (batch), - * DATABASE_POOL_MAX (driver default 10). + * PROFILE (recorded), DELTAS_PER_S, EVENTS_PER_S, API_PER_S, LEASE_S (20), + * EVENT_MIX (recorded, or one event type for all other events), + * DELTA_BYTES (120), EVENT_KIB (2), SPAN_BYTES (600), DELTA_BATCH_MS (250), + * LEASE_MODE (batch), DATABASE_POOL_MAX (driver default 10). */ import assert from "node:assert/strict"; import { randomBytes, randomUUID } from "node:crypto"; @@ -59,12 +71,38 @@ const express = serverRequire("express") as typeof import("express"); const number = (name: string, fallback: number) => Number(process.env[name] ?? fallback); const AGENTS = number("AGENTS", 50), SECONDS = number("SECONDS", 20); -const RATES = { deltas: number("DELTAS_PER_S", 20), events: number("EVENTS_PER_S", 1), api: number("API_PER_S", 0.3), - lease: 1 / number("LEASE_S", 20) }; +const PROFILES = { + recorded: { deltas: 0.2, events: 0.5, api: 0.3 }, + burst: { deltas: 2.5, events: 3, api: 0.3 }, + stress: { deltas: 20, events: 1, api: 0.3 }, +}; +const PROFILE = (process.env.PROFILE ?? "recorded") as keyof typeof PROFILES; +assert(PROFILE in PROFILES, `Unknown PROFILE ${PROFILE}`); +const RATES = { deltas: number("DELTAS_PER_S", PROFILES[PROFILE].deltas), events: number("EVENTS_PER_S", PROFILES[PROFILE].events), + api: number("API_PER_S", PROFILES[PROFILE].api), lease: 1 / number("LEASE_S", 20) }; +// Rows per event type among the other events of recorded hosted native runs. +const RECORDED_EVENT_MIX = { "run.performance.span": 10_030, "tool.execution.progressed": 6_951, "item.completed": 4_705, + "tool.execution.started": 2_150, "tool.execution.completed": 2_150 }; +const EVENT_MIX: Record = !process.env.EVENT_MIX || process.env.EVENT_MIX === "recorded" + ? RECORDED_EVENT_MIX : { [process.env.EVENT_MIX]: 1 }; +// One cycle of 100 slots, each type in proportion to its rows. Agents start +// at different slots so the fleet sends the mix at every moment. +const EVENT_CYCLE = (() => { + const total = Object.values(EVENT_MIX).reduce((sum, rows) => sum + rows, 0); + const shares = Object.entries(EVENT_MIX).map(([type, rows]) => ({ type, exact: rows / total * 100 })); + const slots = shares.map(share => ({ ...share, count: Math.floor(share.exact) })); + for (const slot of [...slots].sort((a, b) => (b.exact - b.count) - (a.exact - a.count)) + .slice(0, 100 - slots.reduce((sum, slot) => sum + slot.count, 0))) slot.count += 1; + const cycle: string[] = []; + for (let index = 0; cycle.length < 100; index++) { + for (const slot of slots) if (index < slot.count) cycle.push(slot.type); + } + return cycle; +})(); type Source = keyof typeof RATES; const SOURCES = (process.env.SOURCES ?? "deltas,events,lease,api").split(",").map(value => value.trim()).filter(Boolean) as Source[]; for (const source of SOURCES) assert(source in RATES, `Unknown source ${source}`); -const DELTA_BYTES = number("DELTA_BYTES", 120), EVENT_KIB = number("EVENT_KIB", 2); +const DELTA_BYTES = number("DELTA_BYTES", 120), EVENT_KIB = number("EVENT_KIB", 2), SPAN_BYTES = number("SPAN_BYTES", 600); const DELTA_BATCH_MS = number("DELTA_BATCH_MS", 250), DELTA_BATCH_MAX = 100; const LEASE_BATCHED = (process.env.LEASE_MODE ?? "batch") !== "single"; // The shared schedule renews a remote lease on its own 20 s after the last @@ -120,7 +158,9 @@ try { fleet.push({ agentId: agent!.id, issueId: issue!.id, runId, leaseOwner, write, records, checkpoint, seq: 0, checkpointTail: Promise.resolve(), appendTail: Promise.resolve() as Promise, deltaQueue: [] as Array<{ input: AppendHeartbeatRunEventInput; settle: (error?: unknown) => void }>, - deltaTimer: null as ReturnType | null, streamedSinceBoundary: false, token: createLocalAgentJwt(agent!.id, companyId, "process", runId, null)! }); + deltaTimer: null as ReturnType | null, streamedSinceBoundary: false, + openBatch: null as Array<{ input: AppendHeartbeatRunEventInput; settle: (error?: unknown) => void }> | null, + eventSlot: Math.floor(index * EVENT_CYCLE.length / Math.max(1, AGENTS)), token: createLocalAgentJwt(agent!.id, companyId, "process", runId, null)! }); } const app = express(); @@ -149,12 +189,29 @@ try { const flushDeltas = (agent: Agent) => { if (agent.deltaTimer !== null) clearTimeout(agent.deltaTimer); agent.deltaTimer = null; - const batch = agent.deltaQueue; - if (batch.length === 0) return; - agent.deltaQueue = []; + if (agent.deltaQueue.length === 0) return; agent.streamedSinceBoundary = true; - agent.appendTail = agent.appendTail.then(() => appendHeartbeatRunEventBatch(db, batch.map(item => item.input), () => {})) - .then(() => { for (const item of batch) item.settle(); }, error => { for (const item of batch) item.settle(error); }); + // As the port: one batch waits behind the batch that is committing, and + // takes every delta queued when it starts, unless a runner event came first. + if (agent.openBatch) { + agent.openBatch.push(...agent.deltaQueue); + agent.deltaQueue = []; + return; + } + const batch = agent.deltaQueue; + agent.deltaQueue = []; + agent.openBatch = batch; + agent.appendTail = agent.appendTail.then(() => { + if (agent.openBatch === batch) { + agent.openBatch = null; + batch.push(...agent.deltaQueue); + agent.deltaQueue = []; + if (agent.deltaTimer !== null) clearTimeout(agent.deltaTimer); + agent.deltaTimer = null; + } + return appendHeartbeatRunEventBatch(db, batch.map(item => item.input), () => {}) + .then(() => { for (const item of batch) item.settle(); }, error => { for (const item of batch) item.settle(error); }); + }); }; const appendDelta = (agent: Agent) => { const { input } = eventInput(agent, "item.delta", DELTA_BYTES); @@ -173,15 +230,29 @@ try { const append = (agent: Agent, eventType: string, bytes: number) => { const { seq, event, input } = eventInput(agent, eventType, bytes); flushDeltas(agent); + agent.openBatch = null; agent.streamedSinceBoundary = false; const queued = agent.appendTail.then(() => appendHeartbeatRunEvent(db, input)).then(() => ({ seq, event })); agent.appendTail = queued.catch(() => undefined); return queued; }; + // The server's run trace appends its spans directly: not a runner event, so + // no source sequence, no wake, no checkpoint, and no place in the run's + // stream of runner events. + const appendSpan = (agent: Agent) => { + const payload = { schema: "paperclip.run-performance-span.v1", span: "tool.call", parentSpan: "task.turn", + durationMs: 12.5, outcome: "ok", detail: randomBytes(Math.ceil(SPAN_BYTES * 0.75)).toString("base64") }; + return appendHeartbeatRunEvent(db, { companyId, agentId: agent.agentId, runId: agent.runId, + eventType: "run.performance.span", stream: "system", level: "info", message: "runner span: tool.call (12.5ms)", payload }); + }; + const eventsByType: Record = {}; const operations: Record Promise> = { deltas: agent => appendDelta(agent), events: async agent => { - const { seq, event } = await append(agent, "item.completed", EVENT_KIB * 1024); + const eventType = EVENT_CYCLE[(agent.eventSlot++) % EVENT_CYCLE.length]!; + eventsByType[eventType] = (eventsByType[eventType] ?? 0) + 1; + if (eventType === "run.performance.span") { await appendSpan(agent); return; } + const { seq, event } = await append(agent, eventType, EVENT_KIB * 1024); const changed = new Map([[`event:${seq}`, JSON.stringify(event)], ["cursor", JSON.stringify({ ackedSourceSeq: seq })]]); for (const [key, value] of changed) agent.records.set(key, value); // The save is serial per controller and does not block the next frame. @@ -203,6 +274,7 @@ try { for (const source of SOURCES) if (!(LEASE_BATCHED && source === "lease")) await operations[source](fleet[0]!); await fleet[0]!.checkpointTail; assert.equal(checkpointFailures, 0, `checkpoint save failed during warm-up: ${firstCheckpointError}`); + for (const eventType of Object.keys(eventsByType)) delete eventsByType[eventType]; const stats = Object.fromEntries(SOURCES.map(source => [source, { done: 0, failed: 0, late: [] as number[], firstError: "" }])) as Record; @@ -298,7 +370,7 @@ try { donePerSecond: round(stat.done / seconds), lateMs: { p50: quantile(stat.late, 0.5), p99: quantile(stat.late, 0.99), max: stat.late.length ? round(stat.late.reduce((max, value) => Math.max(max, value), -Infinity)) : null } }]; })), - deltaBatchMs: DELTA_BATCH_MS, leaseMode: LEASE_BATCHED ? "batch" : "single", leasesLost, + profile: PROFILE, eventsIssuedByType: eventsByType, deltaBatchMs: DELTA_BATCH_MS, leaseMode: LEASE_BATCHED ? "batch" : "single", leasesLost, checkpointSaves: { failed: checkpointFailures, firstError: firstCheckpointError || undefined }, }, null, 2)); } finally { diff --git a/server/src/__tests__/agent-scaling-load-model.test.ts b/server/src/__tests__/agent-scaling-load-model.test.ts index a6212c1f50a..f7c2b79e981 100644 --- a/server/src/__tests__/agent-scaling-load-model.test.ts +++ b/server/src/__tests__/agent-scaling-load-model.test.ts @@ -13,9 +13,10 @@ describe("agent scaling load model", () => { // The default batches deltas and renews every lease in one tick; the other // mode is the earlier one transaction per delta and per lease. it.each([ - { mode: "batched deltas and shared lease renewal", env: {} }, - { mode: "one transaction per delta and per lease", env: { DELTA_BATCH_MS: "0", LEASE_MODE: "single" } }, - ])("drives every source through the real server code and reports each pool ($mode)", async ({ env }) => { + { mode: "recorded event mix, batched deltas and shared lease renewal", env: {}, profile: "recorded" }, + { mode: "stress profile, one transaction per delta and per lease", + env: { PROFILE: "stress", DELTA_BATCH_MS: "0", LEASE_MODE: "single" }, profile: "stress" }, + ])("drives every source through the real server code and reports each pool ($mode)", async ({ env, profile }) => { const { stdout } = await run(process.execPath, [tsx, script], { cwd: serverRoot, maxBuffer: 16 * 1024 * 1024, env: { ...process.env, AGENTS: "2", SECONDS: "2", DELTAS_PER_S: "5", EVENTS_PER_S: "2", API_PER_S: "1", LEASE_S: "1", ...env }, @@ -24,7 +25,13 @@ describe("agent scaling load model", () => { database: { meteredStatementsPerSecondByPool: Record<"main" | "checkpoint" | "lease", number> }; checkpointSaves: { failed: number }; bySource: Record; + profile: string; + eventsIssuedByType: Record; }; + expect(report.profile).toBe(profile); + // Server trace spans and runner events with a progress wake are both in the mix. + expect(report.eventsIssuedByType["run.performance.span"]).toBeGreaterThan(0); + expect(Object.keys(report.eventsIssuedByType).some(type => type.startsWith("tool.execution.") || type === "item.completed")).toBe(true); expect(Object.keys(report.bySource).sort()).toEqual(["api", "deltas", "events", "lease"]); for (const [source, result] of Object.entries(report.bySource)) { expect(result.failed, source).toBe(0); @@ -41,4 +48,19 @@ describe("agent scaling load model", () => { expect(report.database.meteredStatementsPerSecondByPool.checkpoint).toBeGreaterThan(0); expect(report.database.meteredStatementsPerSecondByPool.lease).toBeGreaterThan(0); }, 240_000); + + it.each([ + { profile: "recorded", env: {}, rates: { deltas: 0.2, events: 0.5, api: 0.3 } }, + { profile: "burst", env: { PROFILE: "burst" }, rates: { deltas: 2.5, events: 3, api: 0.3 } }, + { profile: "stress", env: { PROFILE: "stress" }, rates: { deltas: 20, events: 1, api: 0.3 } }, + { profile: "burst", env: { PROFILE: "burst", DELTAS_PER_S: "1" }, rates: { deltas: 1, events: 3, api: 0.3 } }, + ])("uses the $profile profile's rates unless a rate is set on its own ($env)", async ({ profile, env, rates }) => { + const { stdout } = await run(process.execPath, [tsx, script], { + cwd: serverRoot, maxBuffer: 16 * 1024 * 1024, + env: { ...process.env, AGENTS: "1", SECONDS: "1", SOURCES: "deltas,events,api", ...env }, + }); + const report = JSON.parse(stdout.slice(stdout.indexOf("{"))) as { profile: string; perAgentRatesPerSecond: Record }; + expect(report.profile).toBe(profile); + expect(report.perAgentRatesPerSecond).toEqual(rates); + }, 240_000); });