mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 08:22:20 +02:00
DottaandPaperclip
a7504784e5
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat connectors publish safe progress from native runs. > - A recurring query joins each run to its conversation through JSON context. > - Large snapshots make repeated JSON reads expensive. > - Issue-scoped runs already have a typed issue binding. > - This pull request uses that binding and retains the legacy JSON fallback. ## Linked Issues or Issue Description **What existing behavior does this improve?** The native chat-progress candidate query in `enqueueSafeNativeChatProgress`. **Subsystem affected** `server/` database queries and chat publication. **Current behavior** The conversation join extracts `issueId` from each run's JSON snapshot. A local plan probes the run repeatedly and reads a large snapshot for each probe. **Proposed behavior** Use the run's typed `issue_id` when present. Read the JSON key when that column is null. Keep the existing company, endpoint, event, publication, cursor, and cadence conditions. **Reason and benefit** Avoid repeated snapshot reads for scoped runs. Related: #15802 changed delivery queue scheduling; it left native progress projection on its existing cadence. ## What Changed - Change only the native-progress conversation join. Leave the separate milestone query, which already uses its expression index, unchanged. - Add a disposable PostgreSQL benchmark that captures the real service SQL and compares complete rows on one shared fixture. It includes a legacy null binding, unrelated tenants, 2,400 historical decoys, and 144 KiB of random bytes per snapshot (about 192 KiB after base64 encoding). - Compare cardinality across independent fixtures. Their generated conversation and event IDs are not deterministic. - Add PostgreSQL regressions for a typed issue binding without the JSON key and for the legacy NULL-column fallback. ## Verification - Full-size paired local measurement: native baseline 57.67 ms median, treatment 19.35 ms median, **2.98× faster**. Both return the same 41 complete rows. Five timed samples follow warmups, with baseline and treatment alternated. The unchanged milestone query is a control. - An independent full-size run also passed complete row equality and measured 65.22 → 24.08 ms (2.71×). - The small 1 KiB control passed the same 41-row equality check and was neutral: 15.20 → 15.45 ms median. The improvement targets large snapshots, not small payloads. A repeat of the final JavaScript entry point also passed equality and measured 2.97× while the full test suite was running; its wider timing spread is not the primary performance result. - Four PostgreSQL cases pass: typed issue scope without a legacy JSON key, NULL-column JSON fallback, native-progress cadence, and paused-endpoint no-replay behavior. The remaining tests in that file were excluded by the name filter. - Reproduce with `node --import ./cli/node_modules/tsx/dist/loader.mjs scripts/benchmark-chat-native-progress.mjs --phase=baseline` before applying the query change. Run the same command with `--phase=treatment` after applying it. The tool stores SQL, parameters, plans, all samples, and equality evidence under `.scratch/`. - Full workspace typecheck and final server typecheck passed locally. Four focused PostgreSQL cases passed. The duplicate unsharded local full-suite run was stopped after the same commit passed all CI test shards; the full local tests/build sequence was not completed. CI Build and the aggregate verification gate passed on that commit. - Current-head CI is green at `82c9ef9` (55 successful checks, two valid skips). An unchanged runtime-supervision test failed in the first attempt; its isolated local reproduction passed, and one failed-job rerun passed on the same commit. The local reproduction runs on macOS and does not exercise the Linux process-identity branch. - Greptile reviewed `82c9ef9` at 5/5 with no unresolved threads. ## Risks - The typed issue binding takes precedence when present; older rows retain JSON fallback. Company and publication ownership checks remain in the query. - Measurements use local embedded PostgreSQL 18 with synthetic data. The hosted motivation used PostgreSQL 17. Absolute timings and plans can differ on other compute and data distributions. - The milestone benchmark returns an empty candidate set and is a performance control; the nonempty native comparison is the correctness and performance target. - No schema migration or provider network request is added. This PR does not deploy a query change to a live database. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, tool use, and code execution. The exact serving model ID and context-window size are not exposed. Z-AI GLM 5.3 Prime through OpenRouter (`openrouter/z-ai/glm-5.3-prime`) assisted the initial query treatment and benchmark. Codex repaired the comparison, narrowed the production change, and ran the full-size proof. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>