2 Commits

Include renames

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	faaae003a9	perf(chat): find the conversation head from open delivery work (#20564) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External chat messages go into a durable delivery queue. Each conversation drains in provider order. > - Each drain step reads the head of one conversation. The head query filters one endpoint's deliveries by state and thread. > - On a hosted instance, this query ran about 10,000 times at about 300 ms each. The database was busy, and request reads waited behind background work. > - PostgreSQL read every processed delivery of the endpoint and its `normalized_event` value to find a few open rows. > - This pull request finds the endpoint's open delivery IDs first and fetches only those rows. > - The benefit is less background database load. The selected head and the delivery order do not change. ## Linked Issues or Issue Description **What existing behavior does this improve?** The inbound conversation head query (`earliestOpenConversationDelivery`). **Subsystem affected** Server chat inbound delivery queue. **Current behavior** The query has `endpoint_id = $1 and (state in ('received', 'retry', 'processing') or (state = 'processed' and <pending inbound wake>))`, a thread filter on `normalized_event`, and a sort on `normalized_event` keys. No index matches the `or`. PostgreSQL scans all deliveries of the endpoint. For each row it reads `normalized_event` to test the thread. It also reads all of `chat_actions` to build the pending-wake set. A busy endpoint keeps all of its processed history, so the cost grows with that history. **Proposed behavior** The query selects `id in (<open delivery IDs of the endpoint>)`. That set is a `union all` of two small reads: - received, retry and processing deliveries of the endpoint, through the existing `chat_deliveries_work_idx`; - processed deliveries that an issued or processing `inbound_wakeup` action of the same endpoint (and company) still owns. PostgreSQL fetches only those rows by primary key, then applies the same reaction, thread and order rules. **Reason and benefit** The cost no longer grows with processed history. On a local fixture with 60,000 processed deliveries, the head query reads 1,309 shared buffers instead of 262,630. **Breaking changes** None. There is no schema change, migration, API change, timer or poll change. Related work: #17971 changed the pending-wake predicate of the same queue. #20400 added the index for the other chat delivery statement in the same report (`select state ... normalized_event->'githubAutomatic'->'context'->>'deliveryId'`). That statement already uses the #20400 index on master, so this pull request does not change it. ## What Changed - `delivery-ledger.ts`: add `openConversationDeliveryIds(endpointId)`. It returns the endpoint's open delivery IDs. It selects the same set as the old state predicate for that endpoint. - `conversation-queue.ts`: the conversation head query uses `id in (openConversationDeliveryIds(endpointId))` in place of the state `or`. The thread filter, reaction exclusion, order, limit and lifecycle dependency read do not change. - `inbound-queue-discovery.test.ts`: add a real PostgreSQL test. It compares the new set with the old predicate on three endpoints. It covers each state, issued and processing wakes, duplicate wakes, wakes of non-processed deliveries, processed, other-kind and null-delivery actions, an action on a different endpoint of the same company, and reactions. It also checks that the real captured head query returns the oracle's first row. - `benchmark-inbound-queue-discovery.ts`: add `--endpoint-history=<rows>` (a busy endpoint with 4 KiB payloads) and `--only=<fixture>`. Add a work budget: the treatment head query must visit at most 100 delivery rows on the history fixtures. - `doc/DATABASE.md`: describe the head query and the benchmark command. ## Verification Local SQL measurements. PostgreSQL 17.10 (Postgres.app, macOS arm64), disposable database, synthetic data, default settings with 256 MB shared buffers. The host was under heavy load from other work, so timings are noisy. Shared buffers and visited rows are stable. Each arm has two warmups and five alternating samples (median [min..max]). Baseline: `origin/master` at `8d2bf78323`. | Fixture | Query | Baseline | Treatment | Shared buffers | Delivery rows visited | Output | | --- | --- | ---: | ---: | ---: | ---: | --- | | 60,000 processed, 4 KiB | head | 290.09 ms [220.21..554.19] | 7.34 ms [5.85..7.68] | 262,630 → 1,309 | 60,002 → 4 | Equal | | 20,000 processed, 4 KiB | head | 276.02 ms [211.77..322.12] | 34.61 ms [4.45..194.15] | 67,618 → 486 | 20,002 → 4 | Equal | | 1,500 processed, 32 KiB | head | 15.88 ms [13.66..21.79] | 2.29 ms [1.64..7.75] | 10,602 → 80 | 1,502 → 4 | Equal | | 20 processed | head | 1.49 ms [0.92..3.66] | 1.85 ms [1.01..1.97] | 2 → 45 | 22 → 968 | Equal | - The deadline and ready selectors do not change. Their buffers are identical in both arms; their timings move only with host noise. - The small control is neutral. On a near-empty table, PostgreSQL picks a sequential scan for the open-state branch. This costs about 45 buffers. - The 60,000-row run passes `--require-work-budget`. The baseline exceeds the budget (`baselineExceedsWorkBudget: true`). This is the red/green proof. There is no timing threshold in CI. - On the treatment, the remaining head cost is the scan of `chat_actions` for pending wakes of the endpoint. The baseline also reads all of `chat_actions` for its hashed subplan. - `pnpm exec vitest run src/__tests__/inbound-queue-discovery.test.ts` (server): 6 tests pass, including the new equivalence test. - `chat-channels.integration.test.ts` filtered with `-t "reorder|orders|FIFO|conversation drain|lease renewal|receipt order|edit|lifecycle"`: 148 assembled chat tests pass. They cover Slack, Teams, Telegram, Discord and GitHub provider-order drains, head-of-line barriers, lease loss, lifecycle edits and wake retries. - `pnpm exec tsc --noEmit -p tsconfig.json` (server) passes. Reproduce: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-inbound-queue-discovery.ts --baseline-ref=8d2bf78323 --endpoint-history=60000 --only=endpoint-history --require-work-budget ``` Set `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` to use an existing disposable PostgreSQL server. The benchmark uses synthetic data only. ## Risks - Query plans depend on data and settings. These local numbers do not prove hosted request latency. A follow-up statistics window after deploy must confirm the gain. - The pending-wake branch still scans the endpoint's `chat_actions`. A partial index on pending `inbound_wakeup` actions would remove that scan. I did not add it, to avoid a migration. - Low behavior risk. The new set equals the old predicate for each endpoint, and the order expression is unchanged. Tests compare both on the same rows. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edit) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 10:15:29 -06:00
DottaandPaperclip	a91f78e7e0	perf(chat): reduce inbound queue discovery cost (#17971) ... ## Thinking Path > - Paperclip helps people manage companies of AI agents. > - External chat messages enter a durable delivery queue. > - Queue reads must preserve pending wake ownership and conversation order. > - Retained processed history increases the estimated cost of a correlated pending-action test. > - PostgreSQL 17 enables JIT for these small selectors and reads extra JSON in deadline discovery. > - This pull request uses a scoped pending-action set for open-work discovery. > - The benefit is less database work with the same selected deliveries and deadlines. ## Linked Issues or Issue Description **What existing behavior does this improve?** Inbound delivery deadline discovery and conversation-head selection. **Subsystem affected** Server chat queue discovery. **Current behavior** A correlated `EXISTS` inside an `OR` has a high cost estimate on retained completed history. PostgreSQL 17 already hashes its pending-action read, but the estimate still enables JIT. Deadline discovery also reads JSON from completed deliveries that have no pending wake. **Proposed behavior** Use `(company_id, endpoint_id, delivery_id)` membership in the pending-action set for the two open-work selectors. Keep ready-only evaluation on its original scoped `EXISTS`. Keep retry, lease, ordering, lifecycle and endpoint rules. **Reason and benefit** The local retained-history fixture reduces deadline selection time. The small control and the final head selection are neutral. Ready-only selection remains an unchanged control. **Breaking changes** None. No API, schema, migration, stored history, worker, timer or notification changes. Related work: #17647 and #15927 change inbound wake-notice scheduling and selection. This PR changes the separate conversation queue ownership predicate. ## What Changed - Use a fully scoped pending-action set for open-work discovery. Exclude null delivery IDs to preserve exact boolean results. Keep duplicate actions from multiplying deliveries. - Preserve the original ready-only query. - Add five real PostgreSQL tests and a reproducible baseline/treatment benchmark. Capture the complete real service SQL, parameters, positional rows and execution plans. - Document the query cause, rejected hypotheses, test results and limits in `doc/plans/2026-10-27-inbound-queue-query-proof.md`. ## Verification Pinned pre-fix baseline: `09ad9adce2cf4742944353a3707a2f908e9bb35e`. Local fixture: Linux ARM64 PostgreSQL 17.11, Node 26.4.0. It has 1500 completed deliveries and actions, two open deliveries, and 32 KiB of synthetic incompressible JSON per historical delivery. A separate control has 20 completed rows. The normal settings use JIT on with a 100000 cost threshold, 128 MiB shared buffers and 4 MiB work_mem. The final quiet retained-history pair measured these local SQL medians. Each arm has two warmups and five alternating samples. The small control is neutral. | Actual selector | Baseline | Treatment | Complete positional output | | --- | ---: | ---: | --- | | Next inbound deadline | 18.78 ms | 1.58 ms | Equal | | Conversation head | 5.71 ms | 6.18 ms | Equal | | Ordinary ready control | 9.23 ms | 8.33 ms | Equal ; unchanged | The deadline plan uses fewer shared buffers. Head physical reads remain unchanged. Deadline discovery avoids JIT on this fixture. The pristine head estimate is below the JIT threshold in both arms, and its timing is neutral. With JIT disabled, deadline discovery still improves; head and ready timings overlap. - Red: the unchanged baseline fails the explicit local fixture work budget for JIT and deadline buffer reads. Green: the treatment passes. There is no timing threshold in ordinary CI. - Five new PostgreSQL tests and six existing assembled chat tests pass. They cover scoped/null/duplicate pending actions, boolean results, payloads, head barriers, reactions, paused/attention/AgentMail exclusions, retry/processing deadlines, provider ordering, lease loss, lifecycle retry and edit/admission races. - Actual old/new ready selectors have equal error outcomes for unrelated malformed retry timestamps. An initial AND-only oracle comparison was inconclusive and is not defect evidence. - `pnpm -r typecheck` passes. - `pnpm build` passes. - Final head `f3cac298d8a8a5377127b0db7057c52019bbe426` passes the complete [CI workflow](https://github.com/paperclipai/paperclip/actions/runs/40625611887). Greptile is 5/5 on that head with no unresolved review threads. The earlier end-to-end cleanup lock-timeout failure passed on this revision. - The complete local suite was not run. The host's known native PostgreSQL startup failure exhausts SysV semaphores. Focused integration tests use a disposable Docker PostgreSQL constructor through the existing test seam. Complete final-head CI is required. Reproduce the real SQL comparison: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-inbound-queue-discovery.ts --baseline-ref=09ad9adce2cf4742944353a3707a2f908e9bb35e --require-work-budget --retained-only ``` Use `--jit-off` for the diagnostic control. The default command uses the repository's bundled PostgreSQL version and records it in output. The reported 17.11 run used the existing test constructor seam with a disposable `postgres:17` container. Raw samples, complete queries, parameters and plans are emitted as JSON. The benchmark uses disposable synthetic data and does not read a live database URL. ## Risks - Query plans depend on data and PostgreSQL settings. These local measurements do not prove deployed speed or request latency. - Head selection still reads historical JSON. Action history remains stored and scanned. This is not a constant-time queue implementation. - The changed false branch has no retry timestamp cast. Ready-only parsing, claims and retries stay on their existing query. - Background SQL totals can overlap. Do not add these savings into an HTTP latency claim. The largest possible saving for an affected selection is bounded by its observed SQL time, about 49–53 ms on average in discovery. ## Model Used OpenAI Codex, GPT-6 family, with tool use and code execution. The runtime does not expose the exact model ID or context-window size. Three parallel agents own separate performance fixes; this PR contains only the inbound queue fix. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 07:19:22 -05:00