mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process stores every event of every running native agent in `heartbeat_run_events` > - Each native source event append counted all retained events of its source (`count(*)`, `max(source_seq)`) to compute the acknowledgement cursor > - So the cost of each append grew with the length of the run, and long runs made every text delta slower > - This pull request keeps the gap-free cursor of each source on the run row and moves it in the same UPDATE that reserves the event sequence > - The benefit is that an in-order append stops reading other events, so its cost does not grow with run length ## Linked Issues or Issue Description Refs #21131 (earlier reduction of statements per event). Item 1b of the agent scaling plan in #21418. **What existing behavior does this improve?** The append of a native source event (`appendNativeSourceEvent` in `server/src/services/heartbeat-run-events.ts`). It ran a `retained` CTE that counted every event of the source on every append. **Subsystem affected** Server run event ingest (`heartbeat_run_events`, `heartbeat_runs`). **Current behavior** On a run that already holds 50,000 events, one in-order append takes about 10–13 ms (p50) on a local PostgreSQL. On a run with 400 events it takes about 3.3 ms. **Proposed behavior** The append reads the stored cursor in the run lock statement that it already sends. An in-order event moves the cursor and does not count events. On the same 50,000-event run an append takes about 1.7–3.3 ms (p50). **Reason and benefit** Text deltas are about 85% of the server's statements in the load model. Their cost must not grow with run length. **Breaking changes** None. The new column is nullable. The previous release ignores it. ## What Changed - Add the nullable column `heartbeat_runs.native_source_cursors` (`jsonb`, `{"<source instance id>": <seq>}`). Migration `0479` is `ADD COLUMN` only and classifies as `online_compatible`. - The run lock read of a native append also reads the cursor of its source. - When the event sequence is at most cursor + 1, the append statement has no `retained` CTE. It moves the cursor in the UPDATE that reserves the sequence, and the receipt uses the cursor. - Every other case (no cursor, a gap, or a cursor behind the stored events) runs the earlier recount. In the same UPDATE it stores the cursor when the source has no gap, and removes it when the source has a gap. So the event that fills a gap recounts. - The cursor is a lower bound. A writer that does not keep it (the previous release during a deploy overlap, or `NativeRunCoordinatorStore`) can only make it lag. The next out-of-order event then recounts and corrects it. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-event-atomic-append.test.ts src/__tests__/heartbeat-run-event-sequencing.test.ts` passes (19 tests). - Red/green: before the change, 3 tests failed: the two new tests and the updated lock-read statement test. After the change, all pass. - New: "keeps the source cursor on the run so an in-order append does not count the run's events" checks that the append statement has no `count(` and that a replay returns the cursor. - New: "recounts a source whose run has no cursor or a cursor behind its stored events" covers rows saved before the change, rows from a writer that does not keep the cursor, a gap, and the event that fills the gap. - Focused related tests: `paperclip-control-plane-port.test.ts`, `native-observer-replay`, `native-event-boundary-golden`, `chat-native-progress-work`, `live-recovery-work`, `native-interaction-bridge`. - `pnpm check:module-boundaries` passes. `pnpm verify` returned exit 2 (broader CI coverage required) because of local benchmark scripts that are not in this pull request. Required CI supplies the broad coverage. - Append cost on a run with many events (local embedded PostgreSQL, 1,000 in-order appends, busy machine, load average about 20–40): | Events already in run | Before p50 / p90 | After p50 / p90 | | --- | --- | --- | | 50,000 (run 1) | 10.4 / 15.8 ms | 1.7 / 3.7 ms | | 50,000 (run 2) | 12.9 / 21.6 ms | 3.3 / 5.9 ms | | 400 | 3.3 / 6.2 ms | 3.7 / 7.7 ms (noise) | - Statements per event do not change: BEGIN, run lock read, one append statement, COMMIT (4). The append statement no longer scans the source's events. - Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418, `AGENTS=50 SECONDS=20`): the machine was too busy for a stable result. Deltas only: before 20,000 of 20,000 done at 3,999 statements/s. After: 13,796 and 14,817 of 20,000 in two runs. All sources: before 6,346 of 20,000 deltas done. After: 20,000 and 2,083 in two runs. A 20-second window gives only 400 events per run, so it cannot show this change. The long-run figures above are the evidence. ## Risks - Low. The column is nullable and the previous release does not read it. Runs that are in flight during a deploy have no cursor. Their first append recounts once and stores the cursor. - A stale cursor can only make the acknowledgement lag, never pass the stored events. Event rows are deleted only with their company or agent. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>