mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
d8a5db36e697c3ccb6e61aef7dcaddec1dd42293
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
bbed32450c |
perf(load-model): model recorded native run rates, the recorded event mix and coalesced delta batches (#21549)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `scripts/benchmark-agent-scaling-load.ts` (#21418) estimates how many concurrent agents one server process can carry > - It assumed 20 text deltas/s and 1 other event/s per agent, and every other event was an `item.completed` > - Recorded hosted native runs show about 0.2 deltas/s and 0.5 other events/s, with a mix in which 39% of the other events are server trace spans that take a cheaper path > - This pull request makes the recorded rates and mix the model's default, keeps the old rates as an explicit stress profile, adds a burst profile, and mirrors the delta batching of #21520 > - The benefit is a model that measures what production does, so the next optimizations target the real per-agent cost ## Linked Issues or Issue Description Refs #21418 (load model and plan). Refs #21520 (delta batching that the model now mirrors). **What existing behavior does this improve?** The load model's default rates, its event mix and its delta batching rule, and the plan document's measurements. **Subsystem affected** `scripts/benchmark-agent-scaling-load.ts`, its test `server/src/__tests__/agent-scaling-load-model.test.ts`, and `doc/plans/2026-11-13-thousand-concurrent-agents.md`. **Current behavior** The model defaults to 20 deltas/s and 1 other event/s per agent. Every other event is a native `item.completed` with a chat-progress wake and a checkpoint save. It cuts a delta batch on the timer even while the previous batch commits. **Proposed behavior** - `PROFILE=recorded` (default): 0.2 deltas/s and 0.5 other events/s per agent. `PROFILE=burst`: 2.5 and 3, a run's busiest minute. `PROFILE=stress`: the earlier 20 and 1. API requests stay at 0.3/s. Each rate can still be set on its own. - `EVENT_MIX=recorded` (default) sends other events in the recorded proportions. A `run.performance.span` is an ordinary append with no wake and no checkpoint, as the server's run trace writes it. `tool.execution.*` and `item.completed` are native appends with a chat-progress wake, then a checkpoint save. `EVENT_MIX=<type>` sends one type only. - Delta batching follows #21520: deltas that arrive while a batch commits all join one next batch. - The report adds `profile` and `eventsIssuedByType`. **Reason and benefit** At the recorded rates the per-agent cost is the non-delta runner events and their checkpoint saves, not deltas. The plan and the next optimizations need that picture. **Breaking changes** None for the product. Anyone who runs the model with no options now gets the recorded profile; `PROFILE=stress` gives the earlier load. ## What Changed - Profiles, the recorded event mix (a 100-slot cycle per agent, offset by agent so the fleet sends the mix at every moment), the span append path, and the coalesced delta batches. - The model test now checks the profile in the report and that both server trace spans and runner events with a progress wake are in the mix, for the recorded profile and for the stress profile with one transaction per delta. - The plan has a new section "Load model at recorded rates": the recorded source (40 recent native runs on one hosted workspace, as aggregates), the profiles, the mix and its server paths, fresh runs at 500, 1,000 and 2,000 agents plus burst and stress, and what they show. It records the 3-connection checkpoint pool as a computed per-process limit on a hosted database, and that SQL building in Drizzle costs about as much main-thread time as result parsing on the hosted instance. A new section "Considered and not done" records why `run.performance.span` rows stay (about 3% of the statements per agent, with readers) and why checkpoint saves per critical event only are not done now. ## Verification - `server/src/__tests__/agent-scaling-load-model.test.ts` passes (2 tests). The new assertions fail against the previous script: it reports no `profile` and no `eventsIssuedByType`. - `server/src/__tests__/agent-scaling-load-model.test.ts` also checks each profile's rates with no overrides, and that a rate set on its own wins. - Load model runs, recorded in the plan (60 s, main pool of 10, embedded PostgreSQL on the same machine, each run started below a 1-minute load average of 15): recorded rates at 1,000 agents (event-loop utilization 0.61) and 2,000 (0.85) finish every operation with no failure; 500 agents all in their busiest minute run at 0.95 with p99 lateness under 0.4 s. About 4.4 statements per agent per second at recorded rates, 35% of them on the checkpoint pool. - Earlier runs at load averages of 20–55 had shown one process falling behind at 500 burst agents and at 2,000 recorded; the quieter runs after the model fixes (event order, spans off the runner stream) do not, and the plan says so. The checkpoint pool as a per-process limit is now stated as a computed limit for a hosted round trip (about 400 agents in a busy minute at 1 ms), not as measured. - Greptile findings fixed: a waiting delta batch no longer takes deltas sent after the next runner event, takes every delta queued when it starts, and spans no longer join the runner event stream. ## Risks - Low: a benchmark script, its test and a plan document. No product code changes. - The machine that measured the plan's table was shared; the plan marks the rows that the machine distorted. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
05fef9f27b |
perf(server): add a load model of per-agent server cost and the plan for 1,000 concurrent agents (#21418)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process controls every agent run of a workspace > - A hosted workspace slows down with about 50 agents running at the same time > - The target is 1,000 agents at the same time, served by several server processes > - This pull request adds the plan and a local load model that measures what each running agent costs the server > - The benefit is a measured ranking of the causes, and a repeatable before/after number for each fix ## Linked Issues or Issue Description **What existing behavior does this improve?** The server controls agent runs. With about 50 runs active, requests wait and run events fall behind. **Current behavior** Each text delta from an agent is one database transaction. In the load model, 50 agents at 20 deltas per second issue about 3,900 statements per second, and the process completes about half of the offered deltas. Agent authentication slows from 7 ms to 464 ms because it shares the connection pool. **Proposed behavior** The plan gives three phases: measure, decrease the cost per agent, then let several server processes own runs. Each fix is one pull request with a test and a before/after number from the load model. **Reason and benefit** The ranking was an estimate from code. The load model replaces the estimate with a measurement and gives each fix a number to improve. ## What Changed - Added `doc/plans/2026-11-13-thousand-concurrent-agents.md`: finish line, evidence, load model result, phases, and owner decisions. It now also has a "Load model after Phase 1" section. That section has a table at 50, 100, 200 and 400 agents and a row at 200 agents with deltas and leases as before Phase 1. The 400-agent delta count was measured before the count fix and is marked as not reliable. - Added `scripts/benchmark-agent-scaling-load.ts`: N simulated agents call the real event append, controller checkpoint save, lease renewal, and agent authentication code against an embedded PostgreSQL. It reports statements for each database pool, event-loop utilization, and lateness for each source. - Deltas: by default the model commits deltas as the server port does after #21460. The first text after another event commits at once. Later deltas commit in one transaction (`appendHeartbeatRunEventBatch`) per 250 ms or 100 deltas, and the stream does not wait for the receipts. `DELTA_BATCH_MS=0` keeps one transaction per delta, as before batching. - Leases: by default the model renews every lease through the shared schedule from #21444 (`startNativeSessionExecutionLeaseRenewal`, `renewNativeExecutionLeasesNow`). `LEASE_MODE=single` keeps one transaction per run, as before. - The commit counter is read before the checkpoint drain, so commits that come after the window do not count in it. - Per-source counts freeze when the window closes. An operation that settles later, while queued work drains, is not done. - Batch lease mode needs `LEASE_S` of at most 20. The shared schedule renews a remote lease on its own 20 s after its last renewal, and the model does not count that work, so the script rejects a longer interval. - Added `server/src/__tests__/agent-scaling-load-model.test.ts`. It runs the model end to end in both modes: the default (batched deltas and shared lease renewal) and `DELTA_BATCH_MS=0 LEASE_MODE=single`. It checks that every source finishes without failures and that each pool is counted. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-scaling-load-model.test.ts` on a tree that contains #21460 and #21444: 2 of 2 pass (both modes). The test also checks that no source reports more done than offered. - `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-agent-scaling-load.ts` with `AGENTS=50 SECONDS=20` and higher agent counts. The results are in the plan. - `pnpm check:module-boundaries` passes. - No product code changed. ## Risks Low risk. The pull request adds a document and a script that no product code imports. The load model runs on loopback PostgreSQL, so its absolute numbers are not a prediction for a hosted database. The plan states this limit. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and code execution. Subagents on the same model read the code paths. - The two-mode load model test and the merge with master: Claude Opus 5.5 (`claude-opus-5-5`), Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |