mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 12:04:41 +01:00
DottaandPaperclip
475f7df2e7
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat publications connect a run to its authorized originating conversation. > - The milestone selector evaluates detailed lineage and final-answer predicates. > - A local plan evaluated these predicates across many unrelated run and conversation pairs. > - This pull request materializes recent runs that already match an active or waiting conversation. > - The existing query still makes each publication decision. > - The benefit is less JSON work when most run history is unrelated to active chat. ## Linked Issues or Issue Description **What happened?** The milestone candidate query repeatedly reads large run context values for unrelated conversations. A synthetic baseline discarded 60,962 run and conversation pairs through the complex join filter. **Expected behavior** Apply the existing same-company conversation binding before expensive lineage checks. Preserve every candidate column, order, publication decision and continuation rule. **Steps to reproduce** Run the two synthetic PostgreSQL comparisons below against baseline `db70c5acd92f50fa37fb4f0dea3da0e7685089a2`. They capture complete SQL from the actual service implementations. Related work: #17606 changes chat lifecycle scheduling. #15802 deferred milestone cadence. This SQL-only change leaves scheduling, startup reconciliation and recovery intact. It has no dependency on those PRs. ## What Changed - Materialize recent runs with an active or waiting same-company conversation, using the unchanged native and legacy binding expression. - Retain every outer lineage, continuation, final-response, privacy, cursor, ordering and idempotency check. - Add a real PostgreSQL historical-binding regression and selective/dense performance controls. - Document the measured gain and the dense-history tradeoff. ## Verification - Dedicated binding, publication reconciliation and app coordination suites passed all 27 tests across three files with real PostgreSQL where required. - The selective fixture has 2,615 runs, 55 conversation issues and about 192 KiB of pseudorandom context padding per run. Five alternating measured samples follow two warmups on PostgreSQL 17.11. - Final-harness selective milestone median: 1,495.85 ms before, 130.96 ms after, 11.42 times faster. Ranges: 1,483–1,716 ms and 125.59–135.85 ms. All 60 complete ordered rows match, including 15 explicit historical-binding cases. - Dense control: 736.71 ms before, 809.08 ms after, with overlapping ranges. All 200 complete ordered rows match. This is inconclusive performance evidence with possible materialization overhead. - The unchanged native progress control is neutral in both fixtures. It returns 41 equal complete ordered rows. - The added large-window control has 12,215 runs, including 12,000 recent native history rows. A caller limit of one produces the real service page size of 25. First-page medians 762.65 to 800.63 ms overlap and are inconclusive. The 90% cursor improves 234.84 to 54.36 ms; the 99.99% tail improves 352.10 to 12.88 ms; the finished cursor improves 340.00 to 58.58 ms. Complete ordered results match: 25, 25, seven and zero rows. Same-run later-conversation rows remain present. - Large-window CTE materialization itself takes about 4.6–10 ms with zero temp spill. This does not establish a materialization regression or justify a larger production query rewrite. - Reproduce the added window control with `--cursor-window` and a distinct report path. - Reproduce selective: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-milestones.ts --baseline-ref=db70c5acd92f50fa37fb4f0dea3da0e7685089a2 --docker-postgres=17 --report=/tmp/chat-milestones.json`. - Add `--dense-chat` and a distinct report path for the dense control. The harness applies normal migrations to a disposable container and uses no hosted database. - The existing real-service test at `server/src/__tests__/chat-channels.integration.test.ts:34584` calls enqueue with limit=1 four times, crosses 30 resolver-rejected candidates, expects admissions [1,1,1,0], and asserts the exact three publication keys plus internal-run exclusion. It passed and supplies the full composition/publication-set proof requested in review. - Assembled chat integration passed 37 selected tests. Together with the 27 focused tests, 64 checks passed. The selector covers bounded draining, exact continuations, private summaries, final-answer precedence, lifecycle idempotency and native progress races. - Final head `1b3a6f50c9e7ec712dcbaf82c7fc3df141ec3f28`: [full CI passed](https://github.com/paperclipai/paperclip/actions/runs/40534934737), with 55 successful checks, two skipped and one neutral. Greptile is 5/5 on this head; zero unresolved review threads. Paperclip Review is 4/5, neutral, explicitly with no must-fix correctness or security defect. Its remaining payload-memory/dense-overhead caution is retained here as a measured residual tradeoff; an identity-only projection needs production evidence and separate comparison. - After the Drizzle metadata correction, both generated service queries and every parameter remain byte-identical to the selective and dense measured versions. The real PostgreSQL binding regression passed again. - Full repository typecheck and build passed on the integrated final source. The local full test command was stopped after native PostgreSQL initialization failed repeatedly, producing 16 failures and skipped database suites. It is incomplete, not a pass. Focused real-database suites used disposable Docker PostgreSQL. Current-head CI runs the full gates separately. ## Risks - A materialized source can add overhead when most recent runs match active conversations. The dense control does not establish a gain. - Historical JSON comparisons retain string and whitespace semantics. Test fixtures disable the normalization trigger only while inserting retained historical shapes in disposable databases. - Synthetic SQL measurements exclude provider delivery and do not establish deployed latency, actual compute sleep or fleet-wide cost reduction. - No schema migration, index, startup job, polling change or deployment is included. ## Model Used OpenAI Codex, configured model `gpt-6.1-sol` with xhigh reasoning, tool use and code execution. Independent blast-radius review also used `gpt-6-astra` and `gpt-6.1-sol`. The runtime context window was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>