perf(load-model): model recorded native run rates, the recorded event mix and coalesced delta batches (#21549)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `scripts/benchmark-agent-scaling-load.ts` (#21418) estimates how
many concurrent agents one server process can carry
> - It assumed 20 text deltas/s and 1 other event/s per agent, and every
other event was an `item.completed`
> - Recorded hosted native runs show about 0.2 deltas/s and 0.5 other
events/s, with a mix in which 39% of the other events are server trace
spans that take a cheaper path
> - This pull request makes the recorded rates and mix the model's
default, keeps the old rates as an explicit stress profile, adds a burst
profile, and mirrors the delta batching of #21520
> - The benefit is a model that measures what production does, so the
next optimizations target the real per-agent cost

## Linked Issues or Issue Description

Refs #21418 (load model and plan). Refs #21520 (delta batching that the
model now mirrors).

**What existing behavior does this improve?**
The load model's default rates, its event mix and its delta batching
rule, and the plan document's measurements.

**Subsystem affected**
`scripts/benchmark-agent-scaling-load.ts`, its test
`server/src/__tests__/agent-scaling-load-model.test.ts`, and
`doc/plans/2026-11-13-thousand-concurrent-agents.md`.

**Current behavior**
The model defaults to 20 deltas/s and 1 other event/s per agent. Every
other event is a native `item.completed` with a chat-progress wake and a
checkpoint save. It cuts a delta batch on the timer even while the
previous batch commits.

**Proposed behavior**
- `PROFILE=recorded` (default): 0.2 deltas/s and 0.5 other events/s per
agent. `PROFILE=burst`: 2.5 and 3, a run's busiest minute.
`PROFILE=stress`: the earlier 20 and 1. API requests stay at 0.3/s. Each
rate can still be set on its own.
- `EVENT_MIX=recorded` (default) sends other events in the recorded
proportions. A `run.performance.span` is an ordinary append with no wake
and no checkpoint, as the server's run trace writes it.
`tool.execution.*` and `item.completed` are native appends with a
chat-progress wake, then a checkpoint save. `EVENT_MIX=<type>` sends one
type only.
- Delta batching follows #21520: deltas that arrive while a batch
commits all join one next batch.
- The report adds `profile` and `eventsIssuedByType`.

**Reason and benefit**
At the recorded rates the per-agent cost is the non-delta runner events
and their checkpoint saves, not deltas. The plan and the next
optimizations need that picture.

**Breaking changes**
None for the product. Anyone who runs the model with no options now gets
the recorded profile; `PROFILE=stress` gives the earlier load.

## What Changed

- Profiles, the recorded event mix (a 100-slot cycle per agent, offset
by agent so the fleet sends the mix at every moment), the span append
path, and the coalesced delta batches.
- The model test now checks the profile in the report and that both
server trace spans and runner events with a progress wake are in the
mix, for the recorded profile and for the stress profile with one
transaction per delta.
- The plan has a new section "Load model at recorded rates": the
recorded source (40 recent native runs on one hosted workspace, as
aggregates), the profiles, the mix and its server paths, fresh runs at
500, 1,000 and 2,000 agents plus burst and stress, and what they show.
It records the 3-connection checkpoint pool as a computed per-process
limit on a hosted database, and that SQL building in Drizzle costs about
as much main-thread time as result parsing on the hosted instance. A new
section "Considered and not done" records why `run.performance.span`
rows stay (about 3% of the statements per agent, with readers) and why
checkpoint saves per critical event only are not done now.

## Verification

- `server/src/__tests__/agent-scaling-load-model.test.ts` passes (2
tests). The new assertions fail against the previous script: it reports
no `profile` and no `eventsIssuedByType`.
- `server/src/__tests__/agent-scaling-load-model.test.ts` also checks
each profile's rates with no overrides, and that a rate set on its own
wins.
- Load model runs, recorded in the plan (60 s, main pool of 10, embedded
PostgreSQL on the same machine, each run started below a 1-minute load
average of 15): recorded rates at 1,000 agents (event-loop utilization
0.61) and 2,000 (0.85) finish every operation with no failure; 500
agents all in their busiest minute run at 0.95 with p99 lateness under
0.4 s. About 4.4 statements per agent per second at recorded rates, 35%
of them on the checkpoint pool.
- Earlier runs at load averages of 20–55 had shown one process falling
behind at 500 burst agents and at 2,000 recorded; the quieter runs after
the model fixes (event order, spans off the runner stream) do not, and
the plan says so. The checkpoint pool as a per-process limit is now
stated as a computed limit for a hosted round trip (about 400 agents in
a busy minute at 1 ms), not as measured.
- Greptile findings fixed: a waiting delta batch no longer takes deltas
sent after the next runner event, takes every delta queued when it
starts, and spans no longer join the runner event stream.

## Risks

- Low: a benchmark script, its test and a plan document. No product code
changes.
- The machine that measured the plan's table was shared; the plan marks
the rows that the machine distorted.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 23:13:07 -06:00
1 parent 6dd0349d52
commit bbed32450c
3 files changed
+216 -23

No files matched your search

@@ -129,6 +129,104 @@ What it shows:
- Lease renewal is under 1 statement/s at 400 agents (15/s at 50 before).
- The model does not measure server memory, so it does not show change 1h.
## Load model at recorded rates
The rates above were assumptions. Recorded hosted native runs show a much
lighter and different load, so the model's default changed on 2026-11-14.
Source: 40 recent native runs on one hosted workspace (median 26 minutes).
Events stored per run: 0.69/s at the median, 1.5/s at p90. Text deltas: 0.18/s
on average, 1.6–2.7/s in a run's busiest minute; a run streams text for about
8% of its life. Rows by event type: `run.performance.span` 10,030,
`item.delta` 9,502, `tool.execution.progressed` 6,951, `item.completed` 4,705,
`tool.execution.started` and `tool.execution.completed` about 2,150 each.
Controller checkpoint rows: about 1.4 per running run per second.
| Profile (`PROFILE`) | Deltas/s per agent | Other events/s per agent | Use |
| --- | --- | --- | --- |
| `recorded` (default) | 0.2 | 0.5 | median of the recorded runs |
| `burst` | 2.5 | 3 | a run's busiest minute |
| `stress` | 20 | 1 | the earlier assumption, kept for comparison |
API requests stay at 0.3/s per agent in every profile. Other events follow the
recorded mix (`EVENT_MIX`), and each type takes the server path it takes in
production:
| Event type | Share | Server path |
| --- | --- | --- |
| `run.performance.span` | 39% | written by the server's own run trace: one ordinary append, no wake, no checkpoint |
| `tool.execution.progressed` | 27% | runner event with a chat-progress wake: a transaction, then a controller checkpoint save |
| `item.completed` | 18% | the same (a final reply also wakes response presentation) |
| `tool.execution.started`, `tool.execution.completed` | 8% each | the same |
The model's delta batching now also follows #21520: deltas that arrive while a
batch commits go into one next batch, until the run's next runner event.
Measured 2026-11-14 on master (after #21511, #21520 and #21528), 60 s window,
main pool of 10 connections, embedded PostgreSQL on the same machine. Each run
started only when the machine's 1-minute load average was below 15; the load
average at the start and the end is shown. The machine is shared, and the
500-agent recorded run had other work start during its window (marked busy).
| Agents | Profile | Load avg start / end | Event-loop utilization | Statements/s | Per agent per second | Checkpoint pool statements/s | Other events done / offered | Other event lateness p50 / p99 | Delta lateness p50 / p99 | API p50 / p99 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 500 | recorded, busy | 14.9 / 22.2 | 0.66 | 1,979 | 4.0 | 688 | 14,774 / 15,000 | 3 ms / 5.2 s | 2 ms / 5.7 s | 3 ms / 7.1 s |
| 1,000 | recorded | 14.0 / 13.5 | 0.61 | 4,351 | 4.4 | 1,525 | 30,000 / 30,000 | 1 ms / 10 ms | 1 ms / 252 ms | 2 ms / 9 ms |
| 2,000 | recorded | 13.4 / 15.9 | 0.85 | 8,701 | 4.4 | 3,049 | 60,000 / 60,000 | 2 ms / 2.1 s | 1 ms / 2.2 s | 3 ms / 3.0 s |
| 250 | burst | 12.3 / 16.6 | 0.70 | 6,102 | 24.4 | 2,287 | 45,000 / 45,000 | 1 ms / 122 ms | 2 ms / 281 ms | 2 ms / 199 ms |
| 500 | burst | 13.8 / 11.7 | 0.95 | 12,114 | 24.2 | 4,488 | 90,000 / 90,000 | 4 ms / 171 ms | 9 ms / 318 ms | 5 ms / 268 ms |
No run had a failed operation, a failed checkpoint save or a lost lease.
What it shows:
- At recorded rates one process costs about 4.4 statements per agent per
second, and about 35% of them are on the 3-connection checkpoint pool. The
non-delta runner events and their checkpoint saves are most of the cost; text
deltas are now a small share.
- On loopback PostgreSQL one process carries 2,000 agents at recorded rates
(event-loop utilization 0.85, p99 lateness about 2 s), and 500 agents all in
their busiest minute at the same time (0.95, p99 under 0.4 s).
- The 3-connection checkpoint pool kept up locally: about 4,500 statements/s,
or 1,100 saves/s, at 500 agents in a busy minute. Since #21528 a save is 4
statements (BEGIN, the owner lock with the transaction id, the write,
COMMIT), and 5 where commit-time notifications are on (a `pg_notify` before
COMMIT). Each save holds a connection for those round trips, so with a hosted
round trip of about 1 ms the same 3 connections allow about 600–750 saves/s,
which is about 330–400 agents in a busy minute. That is a per-process limit
on a hosted database (change 1g sizes this pool); it is computed here, not
measured.
- Earlier runs on 2026-11-14 at load averages of 20–55 showed one process
falling behind at 500 agents in a busy minute and at 2,000 at recorded rates.
With the machine quieter and the model's event order corrected, both kept up,
so those earlier results described the machine, not the server code.
- The model saves a checkpoint without blocking the agent's next event. The
controller acknowledges a runner event only after its checkpoint save, so in
production a slow checkpoint pool also delays the run's next events. The
model therefore understates the effect of that pool.
- On the hosted instance a CPU profile of the main thread (2026-11-14, with
these fixes) shows building SQL text in Drizzle costing about as much
main-thread time as postgres.js result parsing (each about a fifth). A
statement with fixed SQL text, as in #21511, therefore saves more CPU there
than its statement count suggests; the next changes should be judged by CPU
per operation, not only by statements.
## Considered and not done
- **Stop writing `run.performance.span` rows.** These spans are the largest
event type by rows, but each is one statement with no wake and no
checkpoint. Recorded runs have about 190–250 span rows per run over a median
of 26 minutes, about 0.12–0.16 rows per run per second: about 3% of the
statements per agent. The top names are `tool.request.input_stream` (median
56 per run), `tool.execute` (25), `execution.plugin_dispatch` (11) and
`authorization.base` (7). Run forensics, the restart-chaos speed scripts and
exports read these rows, so they stay.
- **Save a controller checkpoint only for critical runner events.** After the
routine checkpoint save becomes one statement, this would save about 0.3
statements per agent per second at recorded rates. It would change the
acknowledgement order and the restore cursor of the runner's control plane,
which is recovery-critical. Not now; revisit with the writer-thread work.
## Why more servers do not help today
| Assumption | Where | Effect with two steady processes |
@@ -212,8 +310,9 @@ turn, so memory per session counts for idle agents too.
- The load model runs on loopback PostgreSQL on a developer machine. It
measures the modelled sources only and has no hosted round-trip time.
- The per-agent rates (20 deltas/s, 1 other event/s, 0.3 API requests/s) are
assumptions, not a recorded workload. Phase 0 step 2 replaces them.
- The first tables used assumed rates (20 deltas/s, 1 other event/s). The
"Load model at recorded rates" section uses rates from recorded hosted runs;
the API rate (0.3/s) is still an assumption.
- Staging's pool size and run-log storage mode were not read.
- Tool-gateway calls, recovery sweeps, and browser fan-out are not in the
model.
+90 -18
View File
@@ -6,10 +6,15 @@
*
* deltas an `item.delta` per text chunk, committed as the server's port
* commits them: the first after another event at once, later ones
* in one transaction per DELTA_BATCH_MS or 100 deltas
* (DELTA_BATCH_MS=0: one transaction per delta, as before batching)
* events appendHeartbeatRunEvent for an `item.completed`, then the real
* controller checkpoint save, which overlaps the next append
* together after DELTA_BATCH_MS or 100 deltas, and the deltas that
* arrive while a batch commits all in the next batch
* (DELTA_BATCH_MS=0: one append per delta, as before batching)
* events the recorded mix of other events (EVENT_MIX). A runner event
* (`tool.execution.*`, `item.completed`) is a native source append
* that registers a chat progress wake, then the real controller
* checkpoint save, which overlaps the next append. A
* `run.performance.span` is written by the server's own run trace:
* an ordinary append with no wake and no checkpoint
* lease every LEASE_S, one renewal of all leases through the shared
* schedule (LEASE_MODE=single: one transaction per run, as before).
* In batch mode LEASE_S is at most 20: the schedule renews a remote
@@ -33,10 +38,17 @@
*
* pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-agent-scaling-load.ts
*
* PROFILE sets the per-agent rates: `recorded` (default) is the median of
* recorded hosted native runs, 0.2 deltas/s and 0.5 other events/s; `burst`
* is a run's busiest minute, 2.5 deltas/s and 3 other events/s; `stress` is
* the earlier assumption, 20 deltas/s and 1 other event/s. API_PER_S is 0.3
* in every profile. Each rate can still be set on its own.
*
* Optional: AGENTS (50), SECONDS (20), SOURCES (deltas,events,lease,api),
* DELTAS_PER_S (20), EVENTS_PER_S (1), API_PER_S (0.3), LEASE_S (20),
* DELTA_BYTES (120), EVENT_KIB (2), DELTA_BATCH_MS (250), LEASE_MODE (batch),
* DATABASE_POOL_MAX (driver default 10).
* PROFILE (recorded), DELTAS_PER_S, EVENTS_PER_S, API_PER_S, LEASE_S (20),
* EVENT_MIX (recorded, or one event type for all other events),
* DELTA_BYTES (120), EVENT_KIB (2), SPAN_BYTES (600), DELTA_BATCH_MS (250),
* LEASE_MODE (batch), DATABASE_POOL_MAX (driver default 10).
*/
import assert from "node:assert/strict";
import { randomBytes, randomUUID } from "node:crypto";
@@ -59,12 +71,38 @@ const express = serverRequire("express") as typeof import("express");
const number = (name: string, fallback: number) => Number(process.env[name] ?? fallback);
const AGENTS = number("AGENTS", 50), SECONDS = number("SECONDS", 20);
const RATES = { deltas: number("DELTAS_PER_S", 20), events: number("EVENTS_PER_S", 1), api: number("API_PER_S", 0.3),
lease: 1 / number("LEASE_S", 20) };
const PROFILES = {
recorded: { deltas: 0.2, events: 0.5, api: 0.3 },
burst: { deltas: 2.5, events: 3, api: 0.3 },
stress: { deltas: 20, events: 1, api: 0.3 },
};
const PROFILE = (process.env.PROFILE ?? "recorded") as keyof typeof PROFILES;
assert(PROFILE in PROFILES, `Unknown PROFILE ${PROFILE}`);
const RATES = { deltas: number("DELTAS_PER_S", PROFILES[PROFILE].deltas), events: number("EVENTS_PER_S", PROFILES[PROFILE].events),
api: number("API_PER_S", PROFILES[PROFILE].api), lease: 1 / number("LEASE_S", 20) };
// Rows per event type among the other events of recorded hosted native runs.
const RECORDED_EVENT_MIX = { "run.performance.span": 10_030, "tool.execution.progressed": 6_951, "item.completed": 4_705,
"tool.execution.started": 2_150, "tool.execution.completed": 2_150 };
const EVENT_MIX: Record<string, number> = !process.env.EVENT_MIX || process.env.EVENT_MIX === "recorded"
? RECORDED_EVENT_MIX : { [process.env.EVENT_MIX]: 1 };
// One cycle of 100 slots, each type in proportion to its rows. Agents start
// at different slots so the fleet sends the mix at every moment.
const EVENT_CYCLE = (() => {
const total = Object.values(EVENT_MIX).reduce((sum, rows) => sum + rows, 0);
const shares = Object.entries(EVENT_MIX).map(([type, rows]) => ({ type, exact: rows / total * 100 }));
const slots = shares.map(share => ({ ...share, count: Math.floor(share.exact) }));
for (const slot of [...slots].sort((a, b) => (b.exact - b.count) - (a.exact - a.count))
.slice(0, 100 - slots.reduce((sum, slot) => sum + slot.count, 0))) slot.count += 1;
const cycle: string[] = [];
for (let index = 0; cycle.length < 100; index++) {
for (const slot of slots) if (index < slot.count) cycle.push(slot.type);
}
return cycle;
})();
type Source = keyof typeof RATES;
const SOURCES = (process.env.SOURCES ?? "deltas,events,lease,api").split(",").map(value => value.trim()).filter(Boolean) as Source[];
for (const source of SOURCES) assert(source in RATES, `Unknown source ${source}`);
const DELTA_BYTES = number("DELTA_BYTES", 120), EVENT_KIB = number("EVENT_KIB", 2);
const DELTA_BYTES = number("DELTA_BYTES", 120), EVENT_KIB = number("EVENT_KIB", 2), SPAN_BYTES = number("SPAN_BYTES", 600);
const DELTA_BATCH_MS = number("DELTA_BATCH_MS", 250), DELTA_BATCH_MAX = 100;
const LEASE_BATCHED = (process.env.LEASE_MODE ?? "batch") !== "single";
// The shared schedule renews a remote lease on its own 20 s after the last
@@ -120,7 +158,9 @@ try {
fleet.push({ agentId: agent!.id, issueId: issue!.id, runId, leaseOwner, write, records, checkpoint, seq: 0,
checkpointTail: Promise.resolve(), appendTail: Promise.resolve() as Promise<unknown>,
deltaQueue: [] as Array<{ input: AppendHeartbeatRunEventInput; settle: (error?: unknown) => void }>,
deltaTimer: null as ReturnType<typeof setTimeout> | null, streamedSinceBoundary: false, token: createLocalAgentJwt(agent!.id, companyId, "process", runId, null)! });
deltaTimer: null as ReturnType<typeof setTimeout> | null, streamedSinceBoundary: false,
openBatch: null as Array<{ input: AppendHeartbeatRunEventInput; settle: (error?: unknown) => void }> | null,
eventSlot: Math.floor(index * EVENT_CYCLE.length / Math.max(1, AGENTS)), token: createLocalAgentJwt(agent!.id, companyId, "process", runId, null)! });
}
const app = express();
@@ -149,12 +189,29 @@ try {
const flushDeltas = (agent: Agent) => {
if (agent.deltaTimer !== null) clearTimeout(agent.deltaTimer);
agent.deltaTimer = null;
const batch = agent.deltaQueue;
if (batch.length === 0) return;
agent.deltaQueue = [];
if (agent.deltaQueue.length === 0) return;
agent.streamedSinceBoundary = true;
agent.appendTail = agent.appendTail.then(() => appendHeartbeatRunEventBatch(db, batch.map(item => item.input), () => {}))
.then(() => { for (const item of batch) item.settle(); }, error => { for (const item of batch) item.settle(error); });
// As the port: one batch waits behind the batch that is committing, and
// takes every delta queued when it starts, unless a runner event came first.
if (agent.openBatch) {
agent.openBatch.push(...agent.deltaQueue);
agent.deltaQueue = [];
return;
}
const batch = agent.deltaQueue;
agent.deltaQueue = [];
agent.openBatch = batch;
agent.appendTail = agent.appendTail.then(() => {
if (agent.openBatch === batch) {
agent.openBatch = null;
batch.push(...agent.deltaQueue);
agent.deltaQueue = [];
if (agent.deltaTimer !== null) clearTimeout(agent.deltaTimer);
agent.deltaTimer = null;
}
return appendHeartbeatRunEventBatch(db, batch.map(item => item.input), () => {})
.then(() => { for (const item of batch) item.settle(); }, error => { for (const item of batch) item.settle(error); });
});
};
const appendDelta = (agent: Agent) => {
const { input } = eventInput(agent, "item.delta", DELTA_BYTES);
@@ -173,15 +230,29 @@ try {
const append = (agent: Agent, eventType: string, bytes: number) => {
const { seq, event, input } = eventInput(agent, eventType, bytes);
flushDeltas(agent);
agent.openBatch = null;
agent.streamedSinceBoundary = false;
const queued = agent.appendTail.then(() => appendHeartbeatRunEvent(db, input)).then(() => ({ seq, event }));
agent.appendTail = queued.catch(() => undefined);
return queued;
};
// The server's run trace appends its spans directly: not a runner event, so
// no source sequence, no wake, no checkpoint, and no place in the run's
// stream of runner events.
const appendSpan = (agent: Agent) => {
const payload = { schema: "paperclip.run-performance-span.v1", span: "tool.call", parentSpan: "task.turn",
durationMs: 12.5, outcome: "ok", detail: randomBytes(Math.ceil(SPAN_BYTES * 0.75)).toString("base64") };
return appendHeartbeatRunEvent(db, { companyId, agentId: agent.agentId, runId: agent.runId,
eventType: "run.performance.span", stream: "system", level: "info", message: "runner span: tool.call (12.5ms)", payload });
};
const eventsByType: Record<string, number> = {};
const operations: Record<Source, (agent: Agent) => Promise<unknown>> = {
deltas: agent => appendDelta(agent),
events: async agent => {
const { seq, event } = await append(agent, "item.completed", EVENT_KIB * 1024);
const eventType = EVENT_CYCLE[(agent.eventSlot++) % EVENT_CYCLE.length]!;
eventsByType[eventType] = (eventsByType[eventType] ?? 0) + 1;
if (eventType === "run.performance.span") { await appendSpan(agent); return; }
const { seq, event } = await append(agent, eventType, EVENT_KIB * 1024);
const changed = new Map([[`event:${seq}`, JSON.stringify(event)], ["cursor", JSON.stringify({ ackedSourceSeq: seq })]]);
for (const [key, value] of changed) agent.records.set(key, value);
// The save is serial per controller and does not block the next frame.
@@ -203,6 +274,7 @@ try {
for (const source of SOURCES) if (!(LEASE_BATCHED && source === "lease")) await operations[source](fleet[0]!);
await fleet[0]!.checkpointTail;
assert.equal(checkpointFailures, 0, `checkpoint save failed during warm-up: ${firstCheckpointError}`);
for (const eventType of Object.keys(eventsByType)) delete eventsByType[eventType];
const stats = Object.fromEntries(SOURCES.map(source => [source, { done: 0, failed: 0, late: [] as number[], firstError: "" }])) as
Record<Source, { done: number; failed: number; late: number[]; firstError: string }>;
@@ -298,7 +370,7 @@ try {
donePerSecond: round(stat.done / seconds), lateMs: { p50: quantile(stat.late, 0.5), p99: quantile(stat.late, 0.99),
max: stat.late.length ? round(stat.late.reduce((max, value) => Math.max(max, value), -Infinity)) : null } }];
})),
deltaBatchMs: DELTA_BATCH_MS, leaseMode: LEASE_BATCHED ? "batch" : "single", leasesLost,
profile: PROFILE, eventsIssuedByType: eventsByType, deltaBatchMs: DELTA_BATCH_MS, leaseMode: LEASE_BATCHED ? "batch" : "single", leasesLost,
checkpointSaves: { failed: checkpointFailures, firstError: firstCheckpointError || undefined },
}, null, 2));
} finally {
@@ -13,9 +13,10 @@ describe("agent scaling load model", () => {
// The default batches deltas and renews every lease in one tick; the other
// mode is the earlier one transaction per delta and per lease.
it.each([
{ mode: "batched deltas and shared lease renewal", env: {} },
{ mode: "one transaction per delta and per lease", env: { DELTA_BATCH_MS: "0", LEASE_MODE: "single" } },
])("drives every source through the real server code and reports each pool ($mode)", async ({ env }) => {
{ mode: "recorded event mix, batched deltas and shared lease renewal", env: {}, profile: "recorded" },
{ mode: "stress profile, one transaction per delta and per lease",
env: { PROFILE: "stress", DELTA_BATCH_MS: "0", LEASE_MODE: "single" }, profile: "stress" },
])("drives every source through the real server code and reports each pool ($mode)", async ({ env, profile }) => {
const { stdout } = await run(process.execPath, [tsx, script], {
cwd: serverRoot, maxBuffer: 16 * 1024 * 1024,
env: { ...process.env, AGENTS: "2", SECONDS: "2", DELTAS_PER_S: "5", EVENTS_PER_S: "2", API_PER_S: "1", LEASE_S: "1", ...env },
@@ -24,7 +25,13 @@ describe("agent scaling load model", () => {
database: { meteredStatementsPerSecondByPool: Record<"main" | "checkpoint" | "lease", number> };
checkpointSaves: { failed: number };
bySource: Record<string, { offered: number; done: number; failed: number }>;
profile: string;
eventsIssuedByType: Record<string, number>;
};
expect(report.profile).toBe(profile);
// Server trace spans and runner events with a progress wake are both in the mix.
expect(report.eventsIssuedByType["run.performance.span"]).toBeGreaterThan(0);
expect(Object.keys(report.eventsIssuedByType).some(type => type.startsWith("tool.execution.") || type === "item.completed")).toBe(true);
expect(Object.keys(report.bySource).sort()).toEqual(["api", "deltas", "events", "lease"]);
for (const [source, result] of Object.entries(report.bySource)) {
expect(result.failed, source).toBe(0);
@@ -41,4 +48,19 @@ describe("agent scaling load model", () => {
expect(report.database.meteredStatementsPerSecondByPool.checkpoint).toBeGreaterThan(0);
expect(report.database.meteredStatementsPerSecondByPool.lease).toBeGreaterThan(0);
}, 240_000);
it.each([
{ profile: "recorded", env: {}, rates: { deltas: 0.2, events: 0.5, api: 0.3 } },
{ profile: "burst", env: { PROFILE: "burst" }, rates: { deltas: 2.5, events: 3, api: 0.3 } },
{ profile: "stress", env: { PROFILE: "stress" }, rates: { deltas: 20, events: 1, api: 0.3 } },
{ profile: "burst", env: { PROFILE: "burst", DELTAS_PER_S: "1" }, rates: { deltas: 1, events: 3, api: 0.3 } },
])("uses the $profile profile's rates unless a rate is set on its own ($env)", async ({ profile, env, rates }) => {
const { stdout } = await run(process.execPath, [tsx, script], {
cwd: serverRoot, maxBuffer: 16 * 1024 * 1024,
env: { ...process.env, AGENTS: "1", SECONDS: "1", SOURCES: "deltas,events,api", ...env },
});
const report = JSON.parse(stdout.slice(stdout.indexOf("{"))) as { profile: string; perAgentRatesPerSecond: Record<string, number> };
expect(report.profile).toBe(profile);
expect(report.perAgentRatesPerSecond).toEqual(rates);
}, 240_000);
});