diff --git a/doc/plans/2026-11-13-thousand-concurrent-agents.md b/doc/plans/2026-11-13-thousand-concurrent-agents.md index c43512d9657..a17a2113b99 100644 --- a/doc/plans/2026-11-13-thousand-concurrent-agents.md +++ b/doc/plans/2026-11-13-thousand-concurrent-agents.md @@ -129,6 +129,104 @@ What it shows: - Lease renewal is under 1 statement/s at 400 agents (15/s at 50 before). - The model does not measure server memory, so it does not show change 1h. +## Load model at recorded rates + +The rates above were assumptions. Recorded hosted native runs show a much +lighter and different load, so the model's default changed on 2026-11-14. + +Source: 40 recent native runs on one hosted workspace (median 26 minutes). +Events stored per run: 0.69/s at the median, 1.5/s at p90. Text deltas: 0.18/s +on average, 1.6–2.7/s in a run's busiest minute; a run streams text for about +8% of its life. Rows by event type: `run.performance.span` 10,030, +`item.delta` 9,502, `tool.execution.progressed` 6,951, `item.completed` 4,705, +`tool.execution.started` and `tool.execution.completed` about 2,150 each. +Controller checkpoint rows: about 1.4 per running run per second. + +| Profile (`PROFILE`) | Deltas/s per agent | Other events/s per agent | Use | +| --- | --- | --- | --- | +| `recorded` (default) | 0.2 | 0.5 | median of the recorded runs | +| `burst` | 2.5 | 3 | a run's busiest minute | +| `stress` | 20 | 1 | the earlier assumption, kept for comparison | + +API requests stay at 0.3/s per agent in every profile. Other events follow the +recorded mix (`EVENT_MIX`), and each type takes the server path it takes in +production: + +| Event type | Share | Server path | +| --- | --- | --- | +| `run.performance.span` | 39% | written by the server's own run trace: one ordinary append, no wake, no checkpoint | +| `tool.execution.progressed` | 27% | runner event with a chat-progress wake: a transaction, then a controller checkpoint save | +| `item.completed` | 18% | the same (a final reply also wakes response presentation) | +| `tool.execution.started`, `tool.execution.completed` | 8% each | the same | + +The model's delta batching now also follows #21520: deltas that arrive while a +batch commits go into one next batch, until the run's next runner event. + +Measured 2026-11-14 on master (after #21511, #21520 and #21528), 60 s window, +main pool of 10 connections, embedded PostgreSQL on the same machine. Each run +started only when the machine's 1-minute load average was below 15; the load +average at the start and the end is shown. The machine is shared, and the +500-agent recorded run had other work start during its window (marked busy). + +| Agents | Profile | Load avg start / end | Event-loop utilization | Statements/s | Per agent per second | Checkpoint pool statements/s | Other events done / offered | Other event lateness p50 / p99 | Delta lateness p50 / p99 | API p50 / p99 | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 500 | recorded, busy | 14.9 / 22.2 | 0.66 | 1,979 | 4.0 | 688 | 14,774 / 15,000 | 3 ms / 5.2 s | 2 ms / 5.7 s | 3 ms / 7.1 s | +| 1,000 | recorded | 14.0 / 13.5 | 0.61 | 4,351 | 4.4 | 1,525 | 30,000 / 30,000 | 1 ms / 10 ms | 1 ms / 252 ms | 2 ms / 9 ms | +| 2,000 | recorded | 13.4 / 15.9 | 0.85 | 8,701 | 4.4 | 3,049 | 60,000 / 60,000 | 2 ms / 2.1 s | 1 ms / 2.2 s | 3 ms / 3.0 s | +| 250 | burst | 12.3 / 16.6 | 0.70 | 6,102 | 24.4 | 2,287 | 45,000 / 45,000 | 1 ms / 122 ms | 2 ms / 281 ms | 2 ms / 199 ms | +| 500 | burst | 13.8 / 11.7 | 0.95 | 12,114 | 24.2 | 4,488 | 90,000 / 90,000 | 4 ms / 171 ms | 9 ms / 318 ms | 5 ms / 268 ms | + +No run had a failed operation, a failed checkpoint save or a lost lease. + +What it shows: + +- At recorded rates one process costs about 4.4 statements per agent per + second, and about 35% of them are on the 3-connection checkpoint pool. The + non-delta runner events and their checkpoint saves are most of the cost; text + deltas are now a small share. +- On loopback PostgreSQL one process carries 2,000 agents at recorded rates + (event-loop utilization 0.85, p99 lateness about 2 s), and 500 agents all in + their busiest minute at the same time (0.95, p99 under 0.4 s). +- The 3-connection checkpoint pool kept up locally: about 4,500 statements/s, + or 1,100 saves/s, at 500 agents in a busy minute. Since #21528 a save is 4 + statements (BEGIN, the owner lock with the transaction id, the write, + COMMIT), and 5 where commit-time notifications are on (a `pg_notify` before + COMMIT). Each save holds a connection for those round trips, so with a hosted + round trip of about 1 ms the same 3 connections allow about 600–750 saves/s, + which is about 330–400 agents in a busy minute. That is a per-process limit + on a hosted database (change 1g sizes this pool); it is computed here, not + measured. +- Earlier runs on 2026-11-14 at load averages of 20–55 showed one process + falling behind at 500 agents in a busy minute and at 2,000 at recorded rates. + With the machine quieter and the model's event order corrected, both kept up, + so those earlier results described the machine, not the server code. +- The model saves a checkpoint without blocking the agent's next event. The + controller acknowledges a runner event only after its checkpoint save, so in + production a slow checkpoint pool also delays the run's next events. The + model therefore understates the effect of that pool. +- On the hosted instance a CPU profile of the main thread (2026-11-14, with + these fixes) shows building SQL text in Drizzle costing about as much + main-thread time as postgres.js result parsing (each about a fifth). A + statement with fixed SQL text, as in #21511, therefore saves more CPU there + than its statement count suggests; the next changes should be judged by CPU + per operation, not only by statements. + +## Considered and not done + +- **Stop writing `run.performance.span` rows.** These spans are the largest + event type by rows, but each is one statement with no wake and no + checkpoint. Recorded runs have about 190–250 span rows per run over a median + of 26 minutes, about 0.12–0.16 rows per run per second: about 3% of the + statements per agent. The top names are `tool.request.input_stream` (median + 56 per run), `tool.execute` (25), `execution.plugin_dispatch` (11) and + `authorization.base` (7). Run forensics, the restart-chaos speed scripts and + exports read these rows, so they stay. +- **Save a controller checkpoint only for critical runner events.** After the + routine checkpoint save becomes one statement, this would save about 0.3 + statements per agent per second at recorded rates. It would change the + acknowledgement order and the restore cursor of the runner's control plane, + which is recovery-critical. Not now; revisit with the writer-thread work. + ## Why more servers do not help today | Assumption | Where | Effect with two steady processes | @@ -212,8 +310,9 @@ turn, so memory per session counts for idle agents too. - The load model runs on loopback PostgreSQL on a developer machine. It measures the modelled sources only and has no hosted round-trip time. -- The per-agent rates (20 deltas/s, 1 other event/s, 0.3 API requests/s) are - assumptions, not a recorded workload. Phase 0 step 2 replaces them. +- The first tables used assumed rates (20 deltas/s, 1 other event/s). The + "Load model at recorded rates" section uses rates from recorded hosted runs; + the API rate (0.3/s) is still an assumption. - Staging's pool size and run-log storage mode were not read. - Tool-gateway calls, recovery sweeps, and browser fan-out are not in the model. diff --git a/scripts/benchmark-agent-scaling-load.ts b/scripts/benchmark-agent-scaling-load.ts index 407dba26e51..07e602bf6d7 100644 --- a/scripts/benchmark-agent-scaling-load.ts +++ b/scripts/benchmark-agent-scaling-load.ts @@ -6,10 +6,15 @@ * * deltas an `item.delta` per text chunk, committed as the server's port * commits them: the first after another event at once, later ones - * in one transaction per DELTA_BATCH_MS or 100 deltas - * (DELTA_BATCH_MS=0: one transaction per delta, as before batching) - * events appendHeartbeatRunEvent for an `item.completed`, then the real - * controller checkpoint save, which overlaps the next append + * together after DELTA_BATCH_MS or 100 deltas, and the deltas that + * arrive while a batch commits all in the next batch + * (DELTA_BATCH_MS=0: one append per delta, as before batching) + * events the recorded mix of other events (EVENT_MIX). A runner event + * (`tool.execution.*`, `item.completed`) is a native source append + * that registers a chat progress wake, then the real controller + * checkpoint save, which overlaps the next append. A + * `run.performance.span` is written by the server's own run trace: + * an ordinary append with no wake and no checkpoint * lease every LEASE_S, one renewal of all leases through the shared * schedule (LEASE_MODE=single: one transaction per run, as before). * In batch mode LEASE_S is at most 20: the schedule renews a remote @@ -33,10 +38,17 @@ * * pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-agent-scaling-load.ts * + * PROFILE sets the per-agent rates: `recorded` (default) is the median of + * recorded hosted native runs, 0.2 deltas/s and 0.5 other events/s; `burst` + * is a run's busiest minute, 2.5 deltas/s and 3 other events/s; `stress` is + * the earlier assumption, 20 deltas/s and 1 other event/s. API_PER_S is 0.3 + * in every profile. Each rate can still be set on its own. + * * Optional: AGENTS (50), SECONDS (20), SOURCES (deltas,events,lease,api), - * DELTAS_PER_S (20), EVENTS_PER_S (1), API_PER_S (0.3), LEASE_S (20), - * DELTA_BYTES (120), EVENT_KIB (2), DELTA_BATCH_MS (250), LEASE_MODE (batch), - * DATABASE_POOL_MAX (driver default 10). + * PROFILE (recorded), DELTAS_PER_S, EVENTS_PER_S, API_PER_S, LEASE_S (20), + * EVENT_MIX (recorded, or one event type for all other events), + * DELTA_BYTES (120), EVENT_KIB (2), SPAN_BYTES (600), DELTA_BATCH_MS (250), + * LEASE_MODE (batch), DATABASE_POOL_MAX (driver default 10). */ import assert from "node:assert/strict"; import { randomBytes, randomUUID } from "node:crypto"; @@ -59,12 +71,38 @@ const express = serverRequire("express") as typeof import("express"); const number = (name: string, fallback: number) => Number(process.env[name] ?? fallback); const AGENTS = number("AGENTS", 50), SECONDS = number("SECONDS", 20); -const RATES = { deltas: number("DELTAS_PER_S", 20), events: number("EVENTS_PER_S", 1), api: number("API_PER_S", 0.3), - lease: 1 / number("LEASE_S", 20) }; +const PROFILES = { + recorded: { deltas: 0.2, events: 0.5, api: 0.3 }, + burst: { deltas: 2.5, events: 3, api: 0.3 }, + stress: { deltas: 20, events: 1, api: 0.3 }, +}; +const PROFILE = (process.env.PROFILE ?? "recorded") as keyof typeof PROFILES; +assert(PROFILE in PROFILES, `Unknown PROFILE ${PROFILE}`); +const RATES = { deltas: number("DELTAS_PER_S", PROFILES[PROFILE].deltas), events: number("EVENTS_PER_S", PROFILES[PROFILE].events), + api: number("API_PER_S", PROFILES[PROFILE].api), lease: 1 / number("LEASE_S", 20) }; +// Rows per event type among the other events of recorded hosted native runs. +const RECORDED_EVENT_MIX = { "run.performance.span": 10_030, "tool.execution.progressed": 6_951, "item.completed": 4_705, + "tool.execution.started": 2_150, "tool.execution.completed": 2_150 }; +const EVENT_MIX: Record = !process.env.EVENT_MIX || process.env.EVENT_MIX === "recorded" + ? RECORDED_EVENT_MIX : { [process.env.EVENT_MIX]: 1 }; +// One cycle of 100 slots, each type in proportion to its rows. Agents start +// at different slots so the fleet sends the mix at every moment. +const EVENT_CYCLE = (() => { + const total = Object.values(EVENT_MIX).reduce((sum, rows) => sum + rows, 0); + const shares = Object.entries(EVENT_MIX).map(([type, rows]) => ({ type, exact: rows / total * 100 })); + const slots = shares.map(share => ({ ...share, count: Math.floor(share.exact) })); + for (const slot of [...slots].sort((a, b) => (b.exact - b.count) - (a.exact - a.count)) + .slice(0, 100 - slots.reduce((sum, slot) => sum + slot.count, 0))) slot.count += 1; + const cycle: string[] = []; + for (let index = 0; cycle.length < 100; index++) { + for (const slot of slots) if (index < slot.count) cycle.push(slot.type); + } + return cycle; +})(); type Source = keyof typeof RATES; const SOURCES = (process.env.SOURCES ?? "deltas,events,lease,api").split(",").map(value => value.trim()).filter(Boolean) as Source[]; for (const source of SOURCES) assert(source in RATES, `Unknown source ${source}`); -const DELTA_BYTES = number("DELTA_BYTES", 120), EVENT_KIB = number("EVENT_KIB", 2); +const DELTA_BYTES = number("DELTA_BYTES", 120), EVENT_KIB = number("EVENT_KIB", 2), SPAN_BYTES = number("SPAN_BYTES", 600); const DELTA_BATCH_MS = number("DELTA_BATCH_MS", 250), DELTA_BATCH_MAX = 100; const LEASE_BATCHED = (process.env.LEASE_MODE ?? "batch") !== "single"; // The shared schedule renews a remote lease on its own 20 s after the last @@ -120,7 +158,9 @@ try { fleet.push({ agentId: agent!.id, issueId: issue!.id, runId, leaseOwner, write, records, checkpoint, seq: 0, checkpointTail: Promise.resolve(), appendTail: Promise.resolve() as Promise, deltaQueue: [] as Array<{ input: AppendHeartbeatRunEventInput; settle: (error?: unknown) => void }>, - deltaTimer: null as ReturnType | null, streamedSinceBoundary: false, token: createLocalAgentJwt(agent!.id, companyId, "process", runId, null)! }); + deltaTimer: null as ReturnType | null, streamedSinceBoundary: false, + openBatch: null as Array<{ input: AppendHeartbeatRunEventInput; settle: (error?: unknown) => void }> | null, + eventSlot: Math.floor(index * EVENT_CYCLE.length / Math.max(1, AGENTS)), token: createLocalAgentJwt(agent!.id, companyId, "process", runId, null)! }); } const app = express(); @@ -149,12 +189,29 @@ try { const flushDeltas = (agent: Agent) => { if (agent.deltaTimer !== null) clearTimeout(agent.deltaTimer); agent.deltaTimer = null; - const batch = agent.deltaQueue; - if (batch.length === 0) return; - agent.deltaQueue = []; + if (agent.deltaQueue.length === 0) return; agent.streamedSinceBoundary = true; - agent.appendTail = agent.appendTail.then(() => appendHeartbeatRunEventBatch(db, batch.map(item => item.input), () => {})) - .then(() => { for (const item of batch) item.settle(); }, error => { for (const item of batch) item.settle(error); }); + // As the port: one batch waits behind the batch that is committing, and + // takes every delta queued when it starts, unless a runner event came first. + if (agent.openBatch) { + agent.openBatch.push(...agent.deltaQueue); + agent.deltaQueue = []; + return; + } + const batch = agent.deltaQueue; + agent.deltaQueue = []; + agent.openBatch = batch; + agent.appendTail = agent.appendTail.then(() => { + if (agent.openBatch === batch) { + agent.openBatch = null; + batch.push(...agent.deltaQueue); + agent.deltaQueue = []; + if (agent.deltaTimer !== null) clearTimeout(agent.deltaTimer); + agent.deltaTimer = null; + } + return appendHeartbeatRunEventBatch(db, batch.map(item => item.input), () => {}) + .then(() => { for (const item of batch) item.settle(); }, error => { for (const item of batch) item.settle(error); }); + }); }; const appendDelta = (agent: Agent) => { const { input } = eventInput(agent, "item.delta", DELTA_BYTES); @@ -173,15 +230,29 @@ try { const append = (agent: Agent, eventType: string, bytes: number) => { const { seq, event, input } = eventInput(agent, eventType, bytes); flushDeltas(agent); + agent.openBatch = null; agent.streamedSinceBoundary = false; const queued = agent.appendTail.then(() => appendHeartbeatRunEvent(db, input)).then(() => ({ seq, event })); agent.appendTail = queued.catch(() => undefined); return queued; }; + // The server's run trace appends its spans directly: not a runner event, so + // no source sequence, no wake, no checkpoint, and no place in the run's + // stream of runner events. + const appendSpan = (agent: Agent) => { + const payload = { schema: "paperclip.run-performance-span.v1", span: "tool.call", parentSpan: "task.turn", + durationMs: 12.5, outcome: "ok", detail: randomBytes(Math.ceil(SPAN_BYTES * 0.75)).toString("base64") }; + return appendHeartbeatRunEvent(db, { companyId, agentId: agent.agentId, runId: agent.runId, + eventType: "run.performance.span", stream: "system", level: "info", message: "runner span: tool.call (12.5ms)", payload }); + }; + const eventsByType: Record = {}; const operations: Record Promise> = { deltas: agent => appendDelta(agent), events: async agent => { - const { seq, event } = await append(agent, "item.completed", EVENT_KIB * 1024); + const eventType = EVENT_CYCLE[(agent.eventSlot++) % EVENT_CYCLE.length]!; + eventsByType[eventType] = (eventsByType[eventType] ?? 0) + 1; + if (eventType === "run.performance.span") { await appendSpan(agent); return; } + const { seq, event } = await append(agent, eventType, EVENT_KIB * 1024); const changed = new Map([[`event:${seq}`, JSON.stringify(event)], ["cursor", JSON.stringify({ ackedSourceSeq: seq })]]); for (const [key, value] of changed) agent.records.set(key, value); // The save is serial per controller and does not block the next frame. @@ -203,6 +274,7 @@ try { for (const source of SOURCES) if (!(LEASE_BATCHED && source === "lease")) await operations[source](fleet[0]!); await fleet[0]!.checkpointTail; assert.equal(checkpointFailures, 0, `checkpoint save failed during warm-up: ${firstCheckpointError}`); + for (const eventType of Object.keys(eventsByType)) delete eventsByType[eventType]; const stats = Object.fromEntries(SOURCES.map(source => [source, { done: 0, failed: 0, late: [] as number[], firstError: "" }])) as Record; @@ -298,7 +370,7 @@ try { donePerSecond: round(stat.done / seconds), lateMs: { p50: quantile(stat.late, 0.5), p99: quantile(stat.late, 0.99), max: stat.late.length ? round(stat.late.reduce((max, value) => Math.max(max, value), -Infinity)) : null } }]; })), - deltaBatchMs: DELTA_BATCH_MS, leaseMode: LEASE_BATCHED ? "batch" : "single", leasesLost, + profile: PROFILE, eventsIssuedByType: eventsByType, deltaBatchMs: DELTA_BATCH_MS, leaseMode: LEASE_BATCHED ? "batch" : "single", leasesLost, checkpointSaves: { failed: checkpointFailures, firstError: firstCheckpointError || undefined }, }, null, 2)); } finally { diff --git a/server/src/__tests__/agent-scaling-load-model.test.ts b/server/src/__tests__/agent-scaling-load-model.test.ts index a6212c1f50a..f7c2b79e981 100644 --- a/server/src/__tests__/agent-scaling-load-model.test.ts +++ b/server/src/__tests__/agent-scaling-load-model.test.ts @@ -13,9 +13,10 @@ describe("agent scaling load model", () => { // The default batches deltas and renews every lease in one tick; the other // mode is the earlier one transaction per delta and per lease. it.each([ - { mode: "batched deltas and shared lease renewal", env: {} }, - { mode: "one transaction per delta and per lease", env: { DELTA_BATCH_MS: "0", LEASE_MODE: "single" } }, - ])("drives every source through the real server code and reports each pool ($mode)", async ({ env }) => { + { mode: "recorded event mix, batched deltas and shared lease renewal", env: {}, profile: "recorded" }, + { mode: "stress profile, one transaction per delta and per lease", + env: { PROFILE: "stress", DELTA_BATCH_MS: "0", LEASE_MODE: "single" }, profile: "stress" }, + ])("drives every source through the real server code and reports each pool ($mode)", async ({ env, profile }) => { const { stdout } = await run(process.execPath, [tsx, script], { cwd: serverRoot, maxBuffer: 16 * 1024 * 1024, env: { ...process.env, AGENTS: "2", SECONDS: "2", DELTAS_PER_S: "5", EVENTS_PER_S: "2", API_PER_S: "1", LEASE_S: "1", ...env }, @@ -24,7 +25,13 @@ describe("agent scaling load model", () => { database: { meteredStatementsPerSecondByPool: Record<"main" | "checkpoint" | "lease", number> }; checkpointSaves: { failed: number }; bySource: Record; + profile: string; + eventsIssuedByType: Record; }; + expect(report.profile).toBe(profile); + // Server trace spans and runner events with a progress wake are both in the mix. + expect(report.eventsIssuedByType["run.performance.span"]).toBeGreaterThan(0); + expect(Object.keys(report.eventsIssuedByType).some(type => type.startsWith("tool.execution.") || type === "item.completed")).toBe(true); expect(Object.keys(report.bySource).sort()).toEqual(["api", "deltas", "events", "lease"]); for (const [source, result] of Object.entries(report.bySource)) { expect(result.failed, source).toBe(0); @@ -41,4 +48,19 @@ describe("agent scaling load model", () => { expect(report.database.meteredStatementsPerSecondByPool.checkpoint).toBeGreaterThan(0); expect(report.database.meteredStatementsPerSecondByPool.lease).toBeGreaterThan(0); }, 240_000); + + it.each([ + { profile: "recorded", env: {}, rates: { deltas: 0.2, events: 0.5, api: 0.3 } }, + { profile: "burst", env: { PROFILE: "burst" }, rates: { deltas: 2.5, events: 3, api: 0.3 } }, + { profile: "stress", env: { PROFILE: "stress" }, rates: { deltas: 20, events: 1, api: 0.3 } }, + { profile: "burst", env: { PROFILE: "burst", DELTAS_PER_S: "1" }, rates: { deltas: 1, events: 3, api: 0.3 } }, + ])("uses the $profile profile's rates unless a rate is set on its own ($env)", async ({ profile, env, rates }) => { + const { stdout } = await run(process.execPath, [tsx, script], { + cwd: serverRoot, maxBuffer: 16 * 1024 * 1024, + env: { ...process.env, AGENTS: "1", SECONDS: "1", SOURCES: "deltas,events,api", ...env }, + }); + const report = JSON.parse(stdout.slice(stdout.indexOf("{"))) as { profile: string; perAgentRatesPerSecond: Record }; + expect(report.profile).toBe(profile); + expect(report.perAgentRatesPerSecond).toEqual(rates); + }, 240_000); });