mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 20:21:50 +01:00
DottaandPaperclip
7d8bab773d
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Mine includes tasks the person created, was assigned, or participated in. > - Its unread badge ranks the visible personal top 100 before checking unread state. > - Correlated participation and privacy predicates can trigger expensive PostgreSQL JIT work. > - This change joins native participant IDs before ranking a narrow candidate projection. > - A partial actor index avoids scanning company activity history for each badge. > - Local representative scoped reads improve while dense unrestricted overhead remains explicit. ## Linked Issues or Issue Description **What existing behavior does this improve?** The Mine unread badge in the company sidebar. **Current behavior:** A fresh five-minute window recorded 402 ranking calls and 31.7 seconds of SQL time. Correlated predicates can spend more time compiling than executing. **Proposed behavior:** Build the same company-scoped participant set, use native issue keys, apply fresh authority before the top-100 limit, then retain the original canonical rank and second unread query. **Reason and benefit:** Reduce full JIT compilation and avoid an unindexed company-history activity scan. Realistic activity-only participation is included in the measurement. **Breaking changes:** No API or policy change. This PR adds one concurrent partial index migration. Related Mine rules in #16356 and #17756 are preserved. ## What Changed - Join distinct native participant IDs; guard exact legacy text before UUID conversion. - Skip redundant activity participation already covered by creation/assignment. Sort a narrow eligible candidate projection before canonical activity ranking. - Share existing pending-request and user-activity predicates with ordinary Mine reads. - Add partial `activity_log(company_id, actor_id, entity_id)` for user-authored issue activity. Register its idempotent concurrent migration with the existing top-level/invalid-index-repair runner. - Add real PostgreSQL coverage for visible top 100, unread/archive/source semantics, exact IDs, company scope and fresh grant/private-project access. - Add a pinned service benchmark with separate pre-index/indexed databases, identical data, full positional equality, ordinary-call timings, plans and real migration/cancellation proof. ## Verification - Fourteen focused PostgreSQL 17 tests pass (16.37 s), including assembled badge routes and grant revocation. Full workspace typecheck and build pass on the final production sources. - The earlier full local suite is incomplete: native PostgreSQL startup exhausted system semaphores, independently reproduced with empty initdb. All sixteen failed cases passed with task-owned Docker PostgreSQL 17. This is not counted as a full local-suite pass. Full Linux CI passes on final head `f44e46ce580e8345209d7165f57bd8bf696c517c`. Greptile and Paperclip Review are both 5/5 with zero unresolved comments. - Baseline `70c864633744cd96c0dc7394d631fc7119cd17d2`; source and DDL hashes retained. Two warmups and five alternating samples per arm/cell. Capture and EXPLAIN occur outside timed ordinary service calls. - All twelve 300k-history cells match complete positional rows in both SQL phases and final returned IDs. Half the personal tasks qualify through activity alone; the actor index contains matching and other-human rows. | Local service case | Before median | After median | Before / after | | --- | ---: | ---: | ---: | | dense / unrestricted / JIT on | 18.33 ms | 25.06 ms | 0.73x | | dense / scoped / JIT on | 909.30 ms | 221.10 ms | 4.11x | | sparse / unrestricted / JIT on | 13.64 ms | 5.84 ms | 2.34x | | sparse / scoped / JIT on | 719.07 ms | 10.79 ms | 66.64x | Small paths add at most 0.6 ms at the median. Dense unrestricted paths regress by 6.7–7.7 ms; dense scoped JIT-off is approximately neutral (218.34→223.24 ms). Sparse JIT-off paths improve too. This is not a uniform speedup. The actual baseline fails the compiler-work assertion. The final query avoids full JIT optimization/inlining and exceeds 1.5x where the old plan triggers those passes. Small/sparse controls meet the 25% or 2 ms allowance. Dense no-JIT cases are explicit diagnostics. The optional 60 ms absolute goal still fails at 221.10 ms on dense scoped reads: substantial privacy-predicate execution remains after compilation is removed. The failure is retained rather than counted as green. The actual normal migration runner upgraded 300k activity rows in 668.79 ms including inspection, index build/validation and journal work. Rerun took 21.83 ms; a deliberately cancelled build left an invalid index, and retry repaired it in 187.71 ms. Concurrent reads/writes completed; the final index was 188416 bytes. No migration was run on the hosted database. The million-row mixed-participation sparse control also passes all four visibility/JIT pairs with complete equality: unrestricted JIT-on 13.44→5.76 ms; scoped JIT-on 807.76→11.59 ms. The actor branch visits 36 indexed rows rather than scanning company history. Actual migration+runner took 255.06 ms; rerun 20.32 ms; cancellation repair/retry 241.73 ms. Concurrent reads and writes completed without failures. Reproduce: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-mine-unread-badge.ts --baseline-ref=70c864633744cd96c0dc7394d631fc7119cd17d2 --history-decoys=240000 --verify-migration --assert-performance --evidence-dir=<directory> ``` Add `--fixture=sparse --history-decoys=940000` for the million-row control. The default harness starts disposable embedded PostgreSQL and ignores DATABASE_URL. Recorded evidence used PostgreSQL 17.11 through a scratch constructor provider because the host native runtime has semaphore limits. All normal database allocation/migration/client paths are retained. Source-only EXISTS/lateral alternatives were measured and rejected after company scans or sparse regressions. A time bound would lose historical participation, and existing indexes cannot select a user's activity. This justifies the exceptional partial index migration. The exact sample ranges, plans, failures, source hashes, migration footprint and feedback-loop timing are in `doc/plans/2026-10-27-mine-unread-badge-query.md`. ## Risks - Dense unrestricted reads are slightly slower, and dense scoped reads retain substantial authority work. The PR removes a compilation/lookup bottleneck rather than all badge latency. - The concurrent build scans activity history and can wait for long-lived transactions. Local build/cancellation timings do not guarantee the duration on larger or slower hosted databases. User-authored issue activity gains one narrow index to maintain. - Participation, visibility, ranking and legacy ID semantics must remain exact; the real database regressions exercise those boundaries. - Hosted improvement is unverified. The affected ranking SQL averaged 78.8 ms while HTTP badge requests averaged 752 ms without correlated traces. No merge or deployment is included. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and local code execution. The runtime does not expose the exact model ID or context window to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>