Files
PaperClipAI/scripts
DottaandClaude Fable 5.2 05fef9f27b perf(server): add a load model of per-agent server cost and the plan for 1,000 concurrent agents (#21418)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every agent run of a workspace
> - A hosted workspace slows down with about 50 agents running at the
same time
> - The target is 1,000 agents at the same time, served by several
server processes
> - This pull request adds the plan and a local load model that measures
what each running agent costs the server
> - The benefit is a measured ranking of the causes, and a repeatable
before/after number for each fix

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The server controls agent runs. With about 50 runs active, requests wait
and run events fall behind.

**Current behavior**

Each text delta from an agent is one database transaction. In the load
model, 50 agents at 20 deltas per second issue about 3,900 statements
per second, and the process completes about half of the offered deltas.
Agent authentication slows from 7 ms to 464 ms because it shares the
connection pool.

**Proposed behavior**

The plan gives three phases: measure, decrease the cost per agent, then
let several server processes own runs. Each fix is one pull request with
a test and a before/after number from the load model.

**Reason and benefit**

The ranking was an estimate from code. The load model replaces the
estimate with a measurement and gives each fix a number to improve.

## What Changed

- Added `doc/plans/2026-11-13-thousand-concurrent-agents.md`: finish
line, evidence, load model result, phases, and owner decisions. It now
also has a "Load model after Phase 1" section. That section has a table
at 50, 100, 200 and 400 agents and a row at 200 agents with deltas and
leases as before Phase 1. The 400-agent delta count was measured before
the count fix and is marked as not reliable.
- Added `scripts/benchmark-agent-scaling-load.ts`: N simulated agents
call the real event append, controller checkpoint save, lease renewal,
and agent authentication code against an embedded PostgreSQL. It reports
statements for each database pool, event-loop utilization, and lateness
for each source.
- Deltas: by default the model commits deltas as the server port does
after #21460. The first text after another event commits at once. Later
deltas commit in one transaction (`appendHeartbeatRunEventBatch`) per
250 ms or 100 deltas, and the stream does not wait for the receipts.
`DELTA_BATCH_MS=0` keeps one transaction per delta, as before batching.
- Leases: by default the model renews every lease through the shared
schedule from #21444 (`startNativeSessionExecutionLeaseRenewal`,
`renewNativeExecutionLeasesNow`). `LEASE_MODE=single` keeps one
transaction per run, as before.
- The commit counter is read before the checkpoint drain, so commits
that come after the window do not count in it.
- Per-source counts freeze when the window closes. An operation that
settles later, while queued work drains, is not done.
- Batch lease mode needs `LEASE_S` of at most 20. The shared schedule
renews a remote lease on its own 20 s after its last renewal, and the
model does not count that work, so the script rejects a longer interval.
- Added `server/src/__tests__/agent-scaling-load-model.test.ts`. It runs
the model end to end in both modes: the default (batched deltas and
shared lease renewal) and `DELTA_BATCH_MS=0 LEASE_MODE=single`. It
checks that every source finishes without failures and that each pool is
counted.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/agent-scaling-load-model.test.ts` on a tree that contains
#21460 and #21444: 2 of 2 pass (both modes). The test also checks that
no source reports more done than offered.
- `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-agent-scaling-load.ts` with `AGENTS=50 SECONDS=20`
and higher agent counts. The results are in the plan.
- `pnpm check:module-boundaries` passes.
- No product code changed.

## Risks

Low risk. The pull request adds a document and a script that no product
code imports. The load model runs on loopback PostgreSQL, so its
absolute numbers are not a prediction for a hosted database. The plan
states this limit.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and
code execution. Subagents on the same model read the code paths.
- The two-mode load model test and the merge with master: Claude Opus
5.5 (`claude-opus-5-5`), Claude Code, with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 16:06:40 -06:00
..