mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process controls every running agent, and its main thread limits how many agents can run at the same time > - Each agent has a controller journal. The controller saves a checkpoint of this journal after each control event and before each frame it sends > - Each checkpoint converted every command and every event of the journal to JSON again, also when only one record changed. The cost grew with the journal length (up to 500 commands and 4,096 events) > - This pull request keeps the JSON of each record and uses it again while the record does not change > - The benefit is that a checkpoint converts only the records that changed, so the main thread does less work for each agent ## Linked Issues or Issue Description Refs #21411 (this change sits on top of its checkpoint queue). **What existing behavior does this improve?** The controller journal checkpoint in `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`. Each flush calls `captureCheckpoint`, which runs `JSON.stringify` on every command and every recovery event of the journal. **Current behavior** A flush after one new event converts the full journal to JSON. With 4,096 events, one flush made about 5 MB of new JSON strings and used about 14 ms of CPU on the event loop. A hosted CPU profile with about 50 agents showed about 3% of the main thread in this path. **Proposed behavior** A flush converts only the records that changed since the last flush. The other records use the JSON that the controller made before. The saved records are byte-identical to the records of the full conversion. **Reason and benefit** The target is 1,000 agents in one controller process. Checkpoint CPU must not grow with journal length. After this change, one flush with 4,096 events makes about 2.5 KB of new JSON and uses about 2 ms of CPU. ## What Changed - Add `CheckpointRecordCache` to `durable-prp-control-plane.ts`. It keeps the JSON and the part name of each command and event record, keyed by the record object. It uses the JSON again only when the record has the same keys, in the same order, with the same values (`===`). Commands change some fields in place (`status`, `result`, `controllerSeq`), and this comparison finds these changes. - `captureCheckpoint` accepts the cache as an optional fourth argument. Only `setCheckpointPersistence` and `flushCheckpoint` pass it. All other callers (decode, legacy import, bootstrap inspection) do the full conversion as before. - `checkpointChanges` uses loops instead of array copies. Records that did not change keep the same string, so the comparison is by reference. - `queueCommand` stores a copy of the command payload (`structuredClone`). The journal must own its nested objects, because the cache compares nested objects by reference. - `durableRecoveryInternals` exposes `captureCheckpoint` and `createCheckpointRecordCache` for tests. - Regions changed: `checkpointChanges` and the new cache above `captureCheckpoint`; `captureCheckpoint`; `durableRecoveryInternals`; one field of `DurablePrpControlPlane`; the `captureCheckpoint` calls in `setCheckpointPersistence` and `flushCheckpoint`; the `payload` field in `queueCommand`. The `#event` path is not changed. - No change to what a checkpoint saves, when a flush is awaited, or the order of event commit, acknowledgement and checkpoint. The "at most one waiting checkpoint" behavior from #21411 is not changed. ## Verification - New `durable-prp-checkpoint-records.test.ts`: - A journal with 499 commands and 4,000 events gets one new command. The test counts `JSON.stringify` calls on journal records during the flush. Before: 4,499. After: 0 calls on existing records, at most 2 in total (the size check of the new command and its checkpoint record). The delta contains only `command:500`. The test then restores a new controller from the checkpoint and compares its records with a full conversion without the cache. I saw this test fail on `origin/master`. - A property-style test with 5 seeds and 600 random steps each: add, settle and remove commands (with `controllerSeq` renumbering), add, replay and evict events, streaming and usage events, `structuredClone` and spread copies of the state, header counters, the same value written again, and a key-order change. After each step, the records with the cache must be byte-identical to the records without the cache, in the same order. I checked that this test fails when the field comparison is removed (cache by object only). - Existing `durable-prp-control-plane.test.ts` passes. On this loaded machine, some 5-second timing tests time out with and without this change (they also time out on `origin/master`), and pass when I run them alone. - Benchmark: `PAPERCLIP_CHECKPOINT_FLUSH_BENCHMARK=1 pnpm exec vitest run src/control-plane/durable-prp-checkpoint.benchmark.test.ts` (from `packages/paperclip-runner`). Each case commits one new event and flushes with a no-op save, 200 times. CPU time is noisy (load average about 40), the byte and record counts are exact. | Journal | CPU per flush, before | CPU per flush, after | JSON bytes per flush, before | after | Records converted per flush, before | after | | --- | --- | --- | --- | --- | --- | --- | | 10 commands, 10 events | 207 µs | 64–80 µs | 20,975 | 2,465 | 20 | 1 | | 500 commands, 10 events | 1,874 µs | 505–511 µs | 375,068 | 2,466 | 510 | 1 | | 10 commands, 4,096 events | 14,013 µs | 1,655–2,343 µs | 5,174,786 | 2,473 | 4,106 | 1 | - `pnpm check:module-boundaries` passes. `tsc -p packages/paperclip-runner/tsconfig.json --noEmit` passes. ## Risks - The cache assumes that nested `payload`, `result` and `envelope` objects of journal records are not edited in place after they enter the journal. I checked the module: results are stored with `structuredClone`, envelopes with `structuredClone`, and events are replaced, not edited. This change also copies the payload in `queueCommand`. If a later change edits a nested object in place, the checkpoint could keep old JSON. The property test shows top-level edits are found. - Some per-record work is still proportional to the journal (a Map of part names and a field comparison for each record). It is about 0.4 µs for each record. Removing it needs dirty tracking in every state mutator, which this change does not do. - No change to the saved format, so old and new servers read the same checkpoints during a deploy overlap. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>