DottaandPaperclip
475f7df2e7
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat publications connect a run to its authorized originating conversation. > - The milestone selector evaluates detailed lineage and final-answer predicates. > - A local plan evaluated these predicates across many unrelated run and conversation pairs. > - This pull request materializes recent runs that already match an active or waiting conversation. > - The existing query still makes each publication decision. > - The benefit is less JSON work when most run history is unrelated to active chat. ## Linked Issues or Issue Description **What happened?** The milestone candidate query repeatedly reads large run context values for unrelated conversations. A synthetic baseline discarded 60,962 run and conversation pairs through the complex join filter. **Expected behavior** Apply the existing same-company conversation binding before expensive lineage checks. Preserve every candidate column, order, publication decision and continuation rule. **Steps to reproduce** Run the two synthetic PostgreSQL comparisons below against baseline `db70c5acd92f50fa37fb4f0dea3da0e7685089a2`. They capture complete SQL from the actual service implementations. Related work: #17606 changes chat lifecycle scheduling. #15802 deferred milestone cadence. This SQL-only change leaves scheduling, startup reconciliation and recovery intact. It has no dependency on those PRs. ## What Changed - Materialize recent runs with an active or waiting same-company conversation, using the unchanged native and legacy binding expression. - Retain every outer lineage, continuation, final-response, privacy, cursor, ordering and idempotency check. - Add a real PostgreSQL historical-binding regression and selective/dense performance controls. - Document the measured gain and the dense-history tradeoff. ## Verification - Dedicated binding, publication reconciliation and app coordination suites passed all 27 tests across three files with real PostgreSQL where required. - The selective fixture has 2,615 runs, 55 conversation issues and about 192 KiB of pseudorandom context padding per run. Five alternating measured samples follow two warmups on PostgreSQL 17.11. - Final-harness selective milestone median: 1,495.85 ms before, 130.96 ms after, 11.42 times faster. Ranges: 1,483–1,716 ms and 125.59–135.85 ms. All 60 complete ordered rows match, including 15 explicit historical-binding cases. - Dense control: 736.71 ms before, 809.08 ms after, with overlapping ranges. All 200 complete ordered rows match. This is inconclusive performance evidence with possible materialization overhead. - The unchanged native progress control is neutral in both fixtures. It returns 41 equal complete ordered rows. - The added large-window control has 12,215 runs, including 12,000 recent native history rows. A caller limit of one produces the real service page size of 25. First-page medians 762.65 to 800.63 ms overlap and are inconclusive. The 90% cursor improves 234.84 to 54.36 ms; the 99.99% tail improves 352.10 to 12.88 ms; the finished cursor improves 340.00 to 58.58 ms. Complete ordered results match: 25, 25, seven and zero rows. Same-run later-conversation rows remain present. - Large-window CTE materialization itself takes about 4.6–10 ms with zero temp spill. This does not establish a materialization regression or justify a larger production query rewrite. - Reproduce the added window control with `--cursor-window` and a distinct report path. - Reproduce selective: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-milestones.ts --baseline-ref=db70c5acd92f50fa37fb4f0dea3da0e7685089a2 --docker-postgres=17 --report=/tmp/chat-milestones.json`. - Add `--dense-chat` and a distinct report path for the dense control. The harness applies normal migrations to a disposable container and uses no hosted database. - The existing real-service test at `server/src/__tests__/chat-channels.integration.test.ts:34584` calls enqueue with limit=1 four times, crosses 30 resolver-rejected candidates, expects admissions [1,1,1,0], and asserts the exact three publication keys plus internal-run exclusion. It passed and supplies the full composition/publication-set proof requested in review. - Assembled chat integration passed 37 selected tests. Together with the 27 focused tests, 64 checks passed. The selector covers bounded draining, exact continuations, private summaries, final-answer precedence, lifecycle idempotency and native progress races. - Final head `1b3a6f50c9e7ec712dcbaf82c7fc3df141ec3f28`: [full CI passed](https://github.com/paperclipai/paperclip/actions/runs/40534934737), with 55 successful checks, two skipped and one neutral. Greptile is 5/5 on this head; zero unresolved review threads. Paperclip Review is 4/5, neutral, explicitly with no must-fix correctness or security defect. Its remaining payload-memory/dense-overhead caution is retained here as a measured residual tradeoff; an identity-only projection needs production evidence and separate comparison. - After the Drizzle metadata correction, both generated service queries and every parameter remain byte-identical to the selective and dense measured versions. The real PostgreSQL binding regression passed again. - Full repository typecheck and build passed on the integrated final source. The local full test command was stopped after native PostgreSQL initialization failed repeatedly, producing 16 failures and skipped database suites. It is incomplete, not a pass. Focused real-database suites used disposable Docker PostgreSQL. Current-head CI runs the full gates separately. ## Risks - A materialized source can add overhead when most recent runs match active conversations. The dense control does not establish a gain. - Historical JSON comparisons retain string and whitespace semantics. Test fixtures disable the normalization trigger only while inserting retained historical shapes in disposable databases. - Synthetic SQL measurements exclude provider delivery and do not establish deployed latency, actual compute sleep or fleet-wide cost reduction. - No schema migration, index, startup job, polling change or deployment is included. ## Model Used OpenAI Codex, configured model `gpt-6.1-sol` with xhigh reasoning, tool use and code execution. Independent blast-radius review also used `gpt-6-astra` and `gpt-6.1-sol`. The runtime context window was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
9.3 KiB
Chat milestone query performance
The milestone selector first materializes runs that match an active or waiting conversation in the same company. It then applies the existing publication origin, continuation, status and final-answer checks. This prevents PostgreSQL from evaluating expensive JSON lineage predicates across unrelated run/conversation pairs.
The change preserves boundRunIssueIdSql() and all publication authorization,
idempotency, ordering and pagination logic. It adds no migration or index. The
one-second recovery cadence, startup reconciliation, commit signals, shutdown
and native progress selector are unchanged. The separate lifecycle scheduling
work in PR #17606 is outside
this change.
Reproduce the measurements
Run from the repository root with Docker available:
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-milestones.ts --baseline-ref=db70c5acd92f50fa37fb4f0dea3da0e7685089a2 --docker-postgres=17 --report=/tmp/chat-milestones-report.json
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-milestones.ts --baseline-ref=db70c5acd92f50fa37fb4f0dea3da0e7685089a2 --docker-postgres=17 --dense-chat --report=/tmp/chat-milestones-dense-report.json
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-milestones.ts --baseline-ref=db70c5acd92f50fa37fb4f0dea3da0e7685089a2 --docker-postgres=17 --cursor-window --report=/tmp/chat-milestones-large-window-report.json
Without --docker-postgres=17, the harness uses disposable embedded PostgreSQL.
Neither mode uses an existing application database. The Docker mode starts a
container on an ephemeral localhost port, applies the repository migrations and
removes the container after the run.
The harness imports the baseline service from the pinned commit and captures the complete SQL and parameters generated by both real service implementations. It stops before publication writes. The report includes full positional result equality, all samples, plans, database version/settings, source hashes and the treatment diff. A detached, uncommitted treatment is identified by its file hashes and diff in addition to its HEAD.
Local results
Measured on PostgreSQL 17.11 with 128 MB shared buffers and 4 MB work memory. The synthetic fixture contains 2,615 runs, 55 conversation issues and contexts containing a base64 string made from 144 KiB of pseudorandom bytes, approximately 192 KiB of text. Each query has two warmup passes and five measured passes; baseline and treatment order alternates. These are local SQL measurements, excluding provider delivery and application processing.
| Fixture and query | Baseline median, range | Treatment median, range | Complete ordered rows |
|---|---|---|---|
| Selective bindings: milestones | 1,496 ms, 1,483–1,716 | 131 ms, 125.59–135.85 | 60 equal |
| Selective bindings: native progress control | 17.15 ms, 16.35–18.38 | 17.43 ms, 16.77–19.78 | 41 equal |
| Dense bindings: milestones | 737 ms, 629–1,291 | 809 ms, 463–1,288 | 200 equal |
| Dense bindings: native progress control | 23.93 ms, 22.72–25.47 | 24.33 ms, 20.33–29.37 | 41 equal |
The selective fixture improved by 11.42×. Its original merge join discarded 60,962 run/conversation pairs through a complex lineage filter. The treatment materialized 101 matching runs first and discarded no pairs through that join filter. Shared-hit blocks fell from 3,276,955 to 64,903. Neither plan spilled to temporary files.
The dense control binds about 2,500 formerly unrelated history rows to active conversations through the first-class issue column, preserving existing native progress receipts. It materializes 1,768 runs. Its treatment median was about 10% slower, with heavily overlapping sample ranges. A subsequent standalone EXPLAIN also varied substantially. Dense performance is inconclusive, with possible materialization overhead; it does not establish a speedup. Both native progress controls are neutral.
The gain depends on binding selectivity and the database plan. These measurements do not establish a universal speedup, improved request latency or deployed database cost. Compare fresh production query-statistics windows after any separately authorized rollout.
Large-window cursor control
The --cursor-window control uses the same service implementation with 12,215
runs, including 12,000 recent native runs with first-class task bindings and
approximately 8 KiB encoded contexts. It requests a limit of one; the service's
minimum candidate page size is 25. The fixture adds a second conversation for
the same task so both late nonempty pages must preserve the other conversation
on the cursor's own run.
Every statement is captured from enqueueChatRunMilestones, including its
actual second-page cursor. Capture advances the real keyset loop with a
non-publishable placeholder page and stops before any publication write. The
finished cursor comes from the last actual candidate on the short final page.
All statements then run against the real PostgreSQL fixture, with two warmups
and five alternating measurements.
| Service page | Baseline median, range | Treatment median, range | Complete ordered rows |
|---|---|---|---|
| First page | 762.65 ms, 678.43–807.93 | 800.63 ms, 597.38–838.01 | 25 equal |
| Cursor at 90% of history | 234.84 ms, 226.42–241.19 | 54.36 ms, 53.39–63.62 | 25 equal |
| Cursor at 99.99% of history | 352.10 ms, 321.27–377.10 | 12.88 ms, 12.13–29.92 | 7 equal |
| Cursor after the final candidate | 340.00 ms, 324.40–381.30 | 58.58 ms, 56.55–78.92 | 0 equal |
The first page is neutral/inconclusive; its sample ranges overlap. Later pages improved by 4.32×, 27.33× and 5.80× respectively. The unchanged native progress control was neutral at 22.79 versus 22.72 ms, with 25 equal rows.
The treatment still materializes 12,101 matching runs before its outer cursor filter. The materialization node took 4.6–10.0 ms in these plans, and no measured page spilled temporary data. These observations do not measure peak memory or prove how every JSON value is stored or decompressed. They do show that this bounded, native-heavy fixture did not reproduce the feared large-window cursor regression. No identity-only rewrite or additional cursor predicate was needed for this change. The dense and first-page performance limits remain disclosed.
Correctness and blast radius
The safety condition is that the new prefilter repeats a predicate already required by the existing conversation join. It uses the same company, conversation-state and native/legacy binding expression within the same SQL statement. All authoritative checks still run afterward.
The selective benchmark compares every returned column in exact order and checks 15 explicit historical binding cases. These include whitespace trimming, task-ID fallback, numeric/object/array JSON values, uppercase and malformed identifiers, native and first-class issue precedence, cross-company values and null/scalar/array contexts. Dense results compare the complete first 200 ordered rows; they do not claim that all history fits on that page.
Migration 0313's insert trigger normally normalizes recognized JSON identifiers
into the first-class issue binding. The fixture disables that trigger only
while inserting historical row shapes in its disposable database and restores
it in finally. Runtime triggers are unchanged.
Run the dedicated regression and existing coordination suites:
pnpm --filter @paperclipai/server exec vitest run src/__tests__/chat-run-milestone-candidates.test.ts src/services/chat-publication-reconciliation.test.ts src/__tests__/app-private-hostname-gate.test.ts
These three suites passed 27 tests during validation, with a disposable PostgreSQL 17 constructor override for the database test.
The service integration checks cover bounded draining past rejected
candidates, exact continuation origins, private summaries, explicit final-answer
precedence, idempotent lifecycle updates, ownership-attention messages and native
progress reauthorization after a question, final answer or revoked reach wins.
The relevant cases live in server/src/__tests__/chat-channels.integration.test.ts.
These checks prove service behavior using synthetic provider runtimes; they do
not prove live provider delivery or deployed sleep/wake behavior.
The following selection passed 37 tests with disposable PostgreSQL 17; 1,529 unrelated cases were skipped:
pnpm --filter @paperclipai/server exec vitest run src/__tests__/chat-channels.integration.test.ts \
-t "drains bounded Slack chat-origin milestones|coalesces one Slack run's lifecycle|edits a Slack working reply|publishes a Telegram failure for an exact confirmation continuation|replaces working with one safe ownership-attention|does not let a late|keeps an internal Telegram run summary private|publishes only a resolver-selected Telegram reply|settles an interrupted native run once|preserves an interrupted run's selected final|skips a locked native progress|reauthorizes skipped native progress|coalesces closed native progress|publishes native progress with a typed issue binding|enforces the native progress cadence|does not replay stale native progress|keeps native progress behind question|reuses a pre-run FIFO notice|link-only interaction continuation|successful GitHub link-question continuation|publishes a closed-choice question, settles its Slack card"