2 Commits

Include renames

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	3bf7af6c77	perf(plugins): read the lifecycle inbox from per-resource cursors (#20576) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugins get agent, project and agent runtime changes from the `resource_lifecycle_events` journal. Each plugin acknowledges events one resource at a time. > - #20405 made the inbox page read linear in the company journal size. The journal has no retention, so each page still reads every event of the company to find the unacknowledged ones: about 1.7 s per 1M events. > - The delivery worker reads a page again after each acknowledgment. The idle-sleep check also reads the whole journal when all events are acknowledged. `acknowledge` checks every older event of the resource. > - On a hosted instance, the page read ran 181 times in one window at about 1 s each. > - This pull request adds a per-plugin, per-resource cursor. Reads start at the cursor, so they do not read acknowledged history again. > - The benefit is that page, idle-check and acknowledgment cost follow the number of resources and pending events, not the journal size. Delivery order and results stay the same. ## Linked Issues or Issue Description Refs #20405 **What happened?** The plugin lifecycle inbox page query (`select ... from "resource_lifecycle_events" ... DISTINCT ON ...`) ran 181 times at about 1 s each on a hosted database in one window. Each acknowledgment causes another full read of the company journal. The idle-sleep check (`hasPending`) reads the full journal when all events are acknowledged. `acknowledge` reads the full history of the resource. **Expected behavior** A page read, the idle-sleep check and an acknowledgment must not read acknowledged history again. Their cost must not grow with the journal size. **Steps to reproduce** 1. Give a company 400 agents with 2,500 `agent_runtime` events each (1M events). 2. Acknowledge the first 2,000 events of each agent for one plugin. 3. Call `pluginLifecycleInbox(db, pluginId).list(companyId, 100, undefined, ["agent", "project", "agent_runtime"])`. 4. The read takes about 1.7 s. `hasPending` takes about 240 ms. **Paperclip version or commit** `8d2bf78323` (master). **Deployment mode** Hosted (PostgreSQL). Local repro uses embedded PostgreSQL 18. ## What Changed - `packages/db/src/schema/plugin_lifecycle_resource_cursors.ts`: new table `plugin_lifecycle_resource_cursors`. It has one row per plugin, company, resource type and resource id. `acknowledged_through_id` means: every event of that resource at or below this id is acknowledged by the plugin. Plugin and company deletion cascade. - Migration `0462_plugin_lifecycle_resource_cursors`: creates the table and carries existing acknowledgments forward (see Risks for the previous-release story). - `server/src/services/plugin-lifecycle-inbox.ts`: - `list` and `hasPending` use a backward loose index scan on `resource_lifecycle_events_resource_order_idx`. It visits each resource stream once and gets its latest event id. A stream whose latest event is at or below its cursor is settled and is not read again. For other streams, the read gets one head: the unacknowledged creation, else the first unacknowledged transition above the cursor. Then it applies `afterId`, id order and the limit, as before. - `acknowledge` reads only the events between the cursor and the acknowledged event. It still writes the per-event `plugin_lifecycle_acknowledgments` row. It moves the cursor only over a contiguous acknowledged prefix. A creation recorded after a pending transition does not move the cursor. - An event counts as acknowledged when it is at or below the cursor, or when it has an acknowledgment row. Thus rows written by an earlier release, and gaps in old history, keep their current behavior. - `server/src/__tests__/resource-lifecycle-events.test.ts`: new database tests for the migration carry-forward, gaps, late creations, cursor reads, the rollback rows and the plugin-delete cascade. All tests from #20405 still pass without change. - `scripts/benchmark-plugin-lifecycle-inbox.ts`: runs the migration backfill on the fixture and times it. Adds an acknowledgment case. Runs `VACUUM ANALYZE`, because hosted tables are autovacuumed. - `doc/DATABASE.md`: describes the cursor, its read cost and the migration. - The delivery worker (`plugin-lifecycle-delivery.ts`) is not changed. ## Verification Thesis: the cost is a read of the whole company journal (page, idle check) or the whole resource history (acknowledgment). If a per-resource cursor marks the acknowledged prefix, each read can skip that prefix. Then the cost follows the number of resource streams and pending events. The rows must stay equal. Local SQL timings: embedded PostgreSQL 18.1 on Apple Silicon, synthetic data, warm cache, one warmup, 7 samples, baseline and treatment order alternate per sample. The machine was shared with other jobs (load average about 35), so spread is wide. Values are median (min to max) in ms. Rows are equal (`assert.deepEqual`) for every list and probe case. Fixture A: 400 agents x 2,500 runtime events, 2,000 acknowledged per agent. 1,020,440 events, 800,420 acknowledgments. | Case | Before | After | Speedup | | --- | --- | --- | --- | | page of 100, all types | 1,700 (1,633-2,438) | 11.5 (9.5-12.2) | 148x | | page of 100, agent and project | 3.3 (2.5-5.4) | 4.2 (3.6-6.7) | 0.8x (slower) | | idle-sleep check | 238 (174-367) | 2.7 (2.1-3.4) | 88x | | acknowledge one head | 15.3 (9.4-30.1) | 20.3 (12.9-29.9) | neutral (noise) | | migration backfill | - | 9.3 s | - | Fixture B: 40 agents x 25,000 runtime events, 24,000 acknowledged per agent (long history). 1,020,080 events. | Case | Before | After | Speedup | | --- | --- | --- | --- | | page of 100, all types | 1,943 (1,642-4,979) | 4.4 (2.9-5.6) | 444x | | page of 100, agent and project | 3.5 (2.1-5.7) | 4.6 (2.8-12.0) | 0.8x (slower) | | idle-sleep check | 234 (220-243) | 2.8 (1.9-6.5) | 85x | | acknowledge one head | 64.6 (48.5-71.6) | 5.0 (2.6-7.1) | 13x | | migration backfill | - | 4.6 s | - | Fixture C: 5,000 agents x 200 runtime events, 190 acknowledged per agent (many resources). 1,025,040 events. | Case | Before | After | Speedup | | --- | --- | --- | --- | | page of 100, all types | 1,867 (1,649-1,975) | 76.9 (66.2-497) | 24x | | page of 100, agent and project | 10.3 (8.7-12.4) | 21.6 (20.2-23.2) | 0.5x (slower) | | idle-sleep check | 220 (201-298) | 2.8 (2.0-5.3) | 78x | | acknowledge one head | 4.9 (3.4-8.6) | 6.0 (5.2-7.6) | neutral | | migration backfill | - | 15.6 s | - | `EXPLAIN (ANALYZE, BUFFERS)` for the all-types page on fixture A: before 2,488 ms, 13,597 shared hits plus temp files; after 7.0 ms, 7,926 shared hits. Commands: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=400 --per-agent=2500 --acked=2000 --samples=7 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=40 --per-agent=25000 --acked=24000 --samples=7 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=8d2bf78323 --agents=5000 --per-agent=200 --acked=190 --samples=7 cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts src/__tests__/plugin-lifecycle-delivery.test.ts src/__tests__/project-instance-secrets.test.ts src/__tests__/agent-runtime-hooks.test.ts pnpm --filter @paperclipai/db check:migrations ``` - `resource-lifecycle-events.test.ts`, `plugin-lifecycle-delivery.test.ts` and `project-instance-secrets.test.ts`: 86 passed. `agent-runtime-hooks.test.ts`: 22 passed. - `tsc --noEmit -p server`: pass. `check:migrations`: pass. ## Risks - **Schema change.** Migration `0462` adds one table and two foreign keys to `plugins` and `companies`. It adds no index on an existing table. The backfill reads the acknowledgment table once. It took 4.6 s to 15.6 s on 1M-event fixtures with 0.8M to 1M acknowledgments. - **Migration order.** The number `0462` is reserved after open PRs that use `0459` to `0461` (#20571, #20568, #20573). Merge this PR after them. Before merge, rebase onto master and chain the snapshot after the latest migration. - **Previous-release story.** - Upgrade: the backfill stores, per plugin and resource, the end of the contiguous acknowledged prefix. A cursor never passes an unacknowledged event, so no event is lost. Acknowledged events above a gap keep their rows, so no event is delivered again. A test runs the migration statement on rows that only an earlier release wrote and checks that `list` returns the same rows before and after. - Rolling deploy: an earlier release that still runs writes only acknowledgment rows. The new reads honor those rows. - Rollback: the new code still writes every acknowledgment row, so an earlier release sees the same progress. - **Ordering assumption.** The cursor is safe because writers record a resource's events while they hold that resource's row lock (`paperclip_record_agent_runtime` locks the agent row; agent status and project writers document the lock). Thus a later commit for the same resource always has a higher id. A writer that breaks this contract could commit an event below the cursor, and that event would not be delivered. A lower id from a different resource is still delivered (existing test). - **Slower path.** A page that asks only for `agent` and `project` streams is 1 to 11 ms slower. These streams are short, so the old full read was cheap, and the new read visits each stream once. The cost is bounded by the number of resources. - **Many resources.** The page cost grows with the number of resource streams of the requested types. At 5,000 agents a page is about 77 ms, compared with about 1.9 s before. - The hosted planner can choose a different plan. The hosted improvement is not yet measured. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 12:58:32 -06:00
DottaandClaude Opus 5.5	c57cd1bdb9	perf(server): read plugin lifecycle inbox heads in one pass (#20405) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugins get agent, project and agent runtime changes from the `resource_lifecycle_events` journal. Each plugin acknowledges events one resource at a time. > - The inbox page query checked each unacknowledged event against all older events of the same resource. Each check walked the acknowledged history of that resource. > - Agent runtime streams get a new event for each configuration change, so one resource can have thousands of acknowledged events. The cost grew as pending events times acknowledged history. > - On a hosted instance, three copies of this read ran for 601 seconds each, and one earlier copy ran for 19 minutes. The delivery worker runs it again after each acknowledgment. > - This pull request selects the head of each resource in one pass and keeps the same result. > - The benefit is that the page read is linear in the company journal size. In local tests it is 250x to 660x faster, and the result rows are equal. ## Linked Issues or Issue Description **What happened?** The plugin lifecycle inbox page query (`select "id", "company_id", "resource_type", "resource_id", "action", "created_at" from "resource_lifecycle_events" where ...`) ran for 601 seconds on a hosted database. Three copies were active at the same time. **Expected behavior** A lifecycle page read must stay fast when a resource has a long acknowledged history and a pending backlog. **Steps to reproduce** 1. Give a company 40 agents with 1,500 `agent_runtime` events each. 2. Acknowledge the first 1,200 events of each agent for one plugin. 3. Call `pluginLifecycleInbox(db, pluginId).list(companyId, 100, undefined, ["agent", "project", "agent_runtime"])`. 4. The read takes about 10 seconds for 60,000 events of that company. At 200,000 events it takes about 6 minutes. **Paperclip version or commit** `02bfe71a6b` (master). **Deployment mode** Hosted (PostgreSQL). Local repro uses embedded PostgreSQL 18. ## What Changed - `server/src/services/plugin-lifecycle-inbox.ts`: `list` now selects one head per resource with `DISTINCT ON (resource_type, resource_id)`. The head is the unacknowledged creation, else the oldest unacknowledged event. Then the query applies `afterId`, id order and the limit. This is the same set that the old `firstForResource` predicate admits. `acknowledge` keeps the old predicate for its single-event check. - `plugin-lifecycle-inbox.ts`: new `hasPending(companyId, types)`. It returns whether any unacknowledged event exists. Each resource with an unacknowledged event has a head, so this gives the same answer as `list(companyId, 1).length > 0`. - `server/src/services/plugin-lifecycle-delivery.ts`: the idle-sleep journal check uses `hasPending`. This keeps its early exit. A one-row `list` page would read the full journal. - `server/src/__tests__/resource-lifecycle-events.test.ts`: new database test for heads across acknowledged history, a creation recorded after transitions, acknowledged gaps, shared ids across resource types, per-plugin progress, another company, `afterId` pages, limits, `hasPending`, and the next head after an acknowledgment. The test also passes on the old query, so it locks the contract. - `scripts/benchmark-plugin-lifecycle-inbox.ts`: new benchmark. It loads the baseline service from git, runs both on the same fixture, alternates order, asserts equal rows, and prints plans. - `doc/DATABASE.md`: describes the read cost and the benchmark command. - No schema change, index or migration. ## Verification Thesis: the old read cost is (unacknowledged candidates) x (acknowledged older events per resource), because each candidate walks the acknowledged prefix of its resource. One pass that keeps the per-resource head removes that product and returns equal rows. Baseline plan at 220,080 events: 40,020 candidates each walked about 3,999 older rows, which gave 160,039,980 acknowledgment probes and 562,140,833 buffer hits. The treatment plan reads the journal once with a hash anti join and sorts the 40,020 unacknowledged rows (3,925 buffer hits). Local SQL timings (embedded PostgreSQL 18, Apple Silicon; event counts include a 20,000-event second company, synthetic data, warm cache, one warmup, order alternates): | Fixture (events / acknowledged) | Case | Before median | After median | Samples (after) | Speedup | | --- | --- | --- | --- | --- | --- | | 40,060 / 16,040 | page of 100, all types | 3,263 ms | 13.1 ms | 3 (12.0-14.1) | 250x | | 80,080 / 48,060 | page of 100, all types | 10,137 ms | 23.9 ms | 5 (22.7-32.6) | 424x | | 220,080 / 160,060 | page of 100, all types | 355,718 ms | 538 ms | 3 (72-785) | 662x | | 1,020,080 / 960,060 | page of 100, all types | not run (est. > 30 min) | 1,682 ms | 7 (1,203-2,384) | - | | 80,080 / 48,060 | page of 100, agent and project only | 1.5 ms | 1.6 ms | 5 | neutral | | 80,080 / 48,060 | idle-sleep check | 14.5 ms | 13.8 ms | 5 | neutral | | 1,020,080 / 960,060 | idle-sleep check | not run | 406 ms | 7 | - | - Result rows are equal (`assert.deepEqual` on all mapped rows, in order) for each case where both versions ran. - The agent runtime revision read in `plugin-agent-runtime-state.ts` and `agent-file-store.ts` (`select id ... order by id desc limit 1`) uses a backward index-only scan and reads 5 buffers at 1M events. It is not changed. Commands: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=02bfe71a6b --agents=40 --per-agent=1500 --acked=1200 --samples=5 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=02bfe71a6b --agents=40 --per-agent=25000 --acked=24000 --samples=7 --treatment-only cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts src/__tests__/plugin-lifecycle-delivery.test.ts src/__tests__/agent-runtime-hooks.test.ts ``` - `resource-lifecycle-events.test.ts`: 28 passed. `plugin-lifecycle-delivery.test.ts`: 40 passed (includes idle-sleep "none" and "present"). `agent-runtime-hooks.test.ts`: 22 passed. - `tsc --noEmit -p server`: pass. ## Risks - The page read is still linear in the company journal size, because acknowledgments are per plugin and the journal has no retention yet. At 1M events a page takes about 1.7 s locally. Each acknowledgment wakes another full pass. A per-plugin cursor or retention can remove this later. - The planner can choose a different plan on the hosted database. The local plans use a hash anti join and one sort. The hosted improvement is not yet measured. - `acknowledge` still checks one event against older events of its resource. That check is linear in that resource's history per call, as before. - Low behavior risk: the new test passes on both the old and new queries. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 17:37:14 -06:00