2 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	d5766e4074	perf(chat): scope task questions and batch inbox queue checks (#18422) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Standing chats show recent messages and current task questions. > - Recent-message pagination bounds history work, but it does not bound question work. > - The question read built the full company attention feed before it selected the chat's tasks. > - Each feed read also checked seeded queue membership with one insert per item. > - This pull request scopes question collection in the attention owner and batches queue membership checks. > - Chats keep their current question cards and initial layout while doing less work. ## Linked Issues or Issue Description Follow-up to #18358 and #18323. Related work: #18312. **What happened?** A hosted standing chat remained slow after recent-message pagination shipped. On commit `4eaa435bf4`, an embedded-browser reload made the conversation usable at about 8.6 seconds. Recent comments finished at about 1.9 seconds. The pending-question request held the initial reveal until about 8.4 seconds. That request called the complete company attention feed and then filtered its results. The separate badge summary also did broad work. **Expected behavior** Chat question cost should depend on the relevant tasks. Reading an existing queue must not require a separate insert for every member. Current access and decision policy must remain in force. **Steps to reproduce** 1. Create a standing chat with questions on tasks assigned to its agent. 2. Add many unrelated tasks and pending questions to the company. 3. Open the chat from a fresh application cache. 4. Compare recent-comment arrival with the pending-question response and content reveal. **Paperclip version or commit** Baseline: `909e29dd071b1aeeb24c2933af049da90a6066d3`. Its affected services match the observed hosted revision `4eaa435bf4`. ## What Changed - Add an internal task-question projection to the attention service. Filter pending interactions by company, task IDs and question kind before card enrichment. - Keep the same access, audience, dismissal, queue, snooze, retention, archive and training-example logic. Ranks are local to the returned question list. - Use the projection from the conversation receipt service. Keep current chat audience and session checks there. - Batch seeded membership inserts in groups of 500. Keep the queue lock, conflict protection, audit events and activity logging for each new membership. Existing members produce no new audit records. - Add real PostgreSQL regression cases and a reproducible paired benchmark. No schema migration or UI reveal change is needed. ## Verification - `pnpm -r typecheck` passed. The final server typecheck also passed after the queue change. - `pnpm build` passed. - Real PostgreSQL service coverage: 57 attention and chat-handoff tests passed. Seven queue-route tests passed, including concurrent reads, mixed existing/new items, duplicates, disabled seeds and audit preservation. - Red/green proof: the unrelated-attention regression fails with the baseline conversation implementation and passes with the scoped read. - `node scripts/check-module-boundaries.mjs` and `git diff --check` passed. - Full `pnpm test:run` was attempted and stopped with exit 130 after embedded PostgreSQL startup failures and 17 failures in the untouched native-session executor suite. It is not a local pass. The focused Docker/PostgreSQL results above completed before Docker Desktop became unavailable. A direct native PostgreSQL probe identified the host semaphore limit (`could not create semaphores: No space left on device`). No shared kernel settings or other test clusters were changed. [Full CI](https://github.com/paperclipai/paperclip/actions/runs/41162314995) passed on `2809f096b3fb2c01b44a96a1b82a27809c9359df`, including all eight browser shards. Final status: 55 successful checks/statuses, three skipped and one neutral. Greptile is 5/5 with no findings or unresolved threads. The fresh local browser retest is blocked because Docker Desktop cannot start and native PostgreSQL cannot allocate semaphores. The original hosted reproduction above is complete; no candidate browser pass is claimed. Local service benchmark: PostgreSQL 17.11, two relevant tasks, five alternating pairs per scenario, after warmup. These are local service timings, not a claim about deployed latency. | Fixture | Read | Baseline median (range) | Candidate median (range) | SQL calls | | --- | --- | --- | --- | --- | | 1,000 unrelated tasks/questions | Chat questions | 670 ms (632–1,905) | 12 ms (11–15) | 1,076 → 22 | | 1,000 unrelated tasks/questions | Company badge summary | 641 ms (622–717) | 167 ms (161–171) | 1,068 → 69 | | Small company | Chat questions | 38 ms (37–52) | 13 ms (12–16) | 52 → 22 | | Small company | Company badge summary | 34 ms (33–39) | 34 ms (32–40) | 44 → 43 | All question fields and ordering matched, except rank is relative to the scoped list. Summary fields matched except the generation timestamp. The interaction plan changed from 1,002 selected rows to two rows through the existing company/issue/status index. The benchmark loads baseline attention, conversation and queue modules together so the queue change cannot contaminate the baseline. ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-questions.ts \ --baseline-ref=909e29dd07 --unrelated=1000 ``` ## Risks - Scoped and complete feeds must share policy. This change reuses the same attention collection and finalization path and tests dismissals, named audiences, snoozing, archives and private access. - Batched inserts must audit only new members. The concurrent PostgreSQL test checks unique membership and one event/activity record per new item across batch boundaries. - Full inbox summaries still collect the company feed. The small-company summary timing is unchanged. Other chat requests and transcript hydration can still contribute to hosted latency. - The first reveal still waits for question cards to preserve layout. This follow-up has not been deployed, so its hosted effect remains unverified. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, code execution and browser profiling. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 07:37:47 -05:00
DottaandPaperclip	6dcf4160dd	perf(chat): load recent messages and page history on scroll (#18358) ... ## Thinking Path > - Paperclip lets people manage work done by agents. > - A standing agent chat can hold months of conversation. > - The chat route forced full-history mode, bypassing compact task loading. > - A cold visit fetched lifetime run summaries and hydrated historical transcripts. > - This PR starts with 20 recent messages and loads earlier pages on upward scroll. > - It preserves the reader's position, current questions, and draft while history grows. ## Linked Issues or Issue Description **What happened?** Standing agent chats forced full-history loading. A long conversation fetched full activity and run history before it became readable. Earlier comments also used an explicit load control. **Expected behavior** Open a small recent conversation quickly. Load older messages when the reader scrolls upward. Show an inline loading state. Keep the visible message and composer mounted through paging, live updates, and Back navigation. **Steps to reproduce** 1. Open a standing agent chat with hundreds of messages and completed runs. 2. Use a fresh browser query cache. 3. Observe full activity and run reads plus transcript requests for historical runs. 4. Compare the recent window and upward paging after this change. Related work: #18312 and #18323 restored compact task and collection loading. #14848 added reopening of visited agent chats. A GitHub search found no open PR for this chat-history optimization. ## What Changed - Return 20 contiguous recent comments for standing chats, with a stable oldest cursor. Keep the existing compact task exchange contract. Use the existing company/issue/time index for cursor reads. - Keep chats compact even if their task view previously expanded history. Retain observed messages through live catch-up and cache-backed return visits. - Keep fetched history pages, acknowledged sends, and the observed recent cursor separate. Reconcile live replies without losing history after a send or deleted cursor. - Hydrate metadata only for newly revealed message/run IDs; preserve prior batches while paging. Slow-send recovery refreshes the recent window. Receipts failures have an independent retry. - Hydrate run summaries only for loaded messages and the latest turn. Validate batches of at most 100 run IDs and keep the existing issue/company access rules. - Reuse hydrated terminal native streams when only history membership changes. Preserve retry, resume, mode changes, reconnect recovery, and interrupted recovery reads. - Load older messages, decision receipts, and task handoffs through upward scrolling. Support wheel, touch, and keyboard input. Keep pending questions visible. - Reserve a status slot while loading. Preserve a logical message anchor and the mounted composer. Pause failed page reads until an inline Retry. - Add database, route, controller, and scroll regressions, plus an opt-in production-build browser rig for long chats on desktop and mobile. Update design and feature/access documentation. ## Verification - The real PostgreSQL cold-open suite passes all 12 tests, including contiguous pages with equal timestamps, complete history, company denial, and scoped run IDs. - Activity route and OpenAPI suites pass 53 tests. The route run allowed local cold-import overhead; CI retains the repository's normal timeouts. - Three new paging regressions fail against the unchanged source. The candidate's focused paging/window suite passes 26 tests. Broader UI verification is recorded below when complete. - Desktop and mobile browser acceptance covers recent-only startup, per-frame reveal geometry, delayed and failed older pages, stable reading anchors and drafts, Back, older comment links, and one persisted live reply. - Full workspace typecheck and build passed locally. The final UI source also passes its typecheck and production build. - Affected UI suites pass: 212 controller/history/paging tests, 352 thread/scroller tests, 26 native-stream tests, plus metadata batching and live-cache hydration coverage. The issue mutation/access route suite passes 179 tests. - All 22 final desktop/mobile browser journeys pass, including delayed image growth. The isolated second-page request proof also passes six repeated desktop/mobile runs. A separate 6,000-message, 3,000-run fixture passes cold-open and delayed-paging checks on both screen sizes, still opening 20 messages and loading one 50-message page. - Final matched production timings use baseline `aa58fa1073` (`index-BaRiMbxd.js`) and candidate UI source `f9a26b9783` (`index-DPWTbsFF.js`, identical application source at final head `bf71ffd5af`). Chromium 151.0.7922.34, 1440×900, 4× CPU throttle, no added network latency, browser HTTP cache disabled, fresh application cache for each cold visit. Both variants block service workers and use the same warm local API server and immutable 600-message/300-run fixture. Only document HTML is intercepted for both variants; API requests are real. The common API revision is `109981f8c1`; subsequent commits change only UI and tests. | Usable conversation | Baseline median (range) | Candidate median (range) | | --- | --- | --- | | Cold direct URL | 6.57 s (5.69–7.45) | 3.07 s (2.50–3.28) | | Warm in-app return | 3.76 s (3.54–4.76) | 0.67 s (0.58–1.46) | | Back return | 2.53 s (2.12–4.20) | 0.51 s (0.47–1.18) | Five pairs alternate order. Every paired difference favors the candidate for all three journeys. Readiness requires a revealed, fully visible latest reply above the composer and an editable composer. Return timing starts after the Tasks route renders. Cold API requests fall from 953–954 to 71; API wire bytes fall from 517–559 KB to about 49 KB. Separate CPU profiles and request waterfalls confirm the removal of lifetime event hydration and are excluded from the five timing pairs. Both variants have the same expected missing optional plan document (404) and no browser exceptions. Control recent-window API reads have medians of 37 ms / 32 ms, with shared-host outliers up to 376 ms. The 6,000-message recent-window SQL uses `issue_comments_company_issue_created_at_idx` (22 index rows read for a 21-row probe). Geometry assertions retain a 2-pixel tolerance for the reading anchor and first-reveal layout. These are local synthetic measurements, not deployed staging latency. Earlier timing probes are excluded because they did not isolate service workers or composer clearance. Final head `bf71ffd5afb5684ba9105a3127f617e7adf5139c` has 55 successful checks, three optional skips and one neutral result, with no failures or pending checks. Greptile is 5/5 with no open review findings. ### Failure record Early local attempts exposed an unconditional scroller wrapper that changed the old task layout, and automatic retries on chat page failures. Both were corrected. The original task scroller structure stays in place. Metadata request assertions wait for initial connection recovery to settle, then verify only the new page’s reads; retry/resume/reconnect revalidation remains intentional. The initial review caught a send-before-scroll cache defect; its controller regression failed before the fix. Later rapid-paging checks caught scroll readiness being gated by metadata after the first reveal; readiness now stays enabled once revealed. Live replies use the separate acknowledged-message cache, then catch up from the server cursor. Visual review exposed a pagination wrapper bypassing existing composer clearance. The body is again a direct scroller child; cold-open browser checks require the full latest reply above the composer. A resize-observer regression failed before the fix and now passes. Controlled delayed inline images prove both pinned following and a stable reading anchor on desktop/mobile. The first performance fixture lacked valid native event envelopes and caused missing-log fallbacks. It is excluded from latency claims. The final fixture uses saved native assistant events and conversation session generations. The live test uses a temporary fixture-agent key, attributes the reply to an existing synthetic run, and revokes the key. It starts no model or agent process. Local database attempts failed during a Docker shutdown. The retained disposable database was restored and its suite passed. Early fixture errors included a missing conversation identity field and session generation. Mobile test setup also selected an offscreen sidebar link and reused queued human input. Optional announcements could intercept input; the fixture now dismisses them before the journeys. The scroll observer now waits for the delivered wheel event and settled animation frames, including when the viewport is already at the top. A concurrently running live test briefly changed another probe’s row count; the final acceptance and timing runs were sequential on separate fixtures. The final rig uses the visible mobile navigation and a saved agent reply. Anchor observation waits for the user's scroll to settle before measuring the effect of live updates. The full local `pnpm test:run` attempt hit repeated embedded PostgreSQL initialization failures and failures in untouched native-session executor suites; it was stopped after the repeated environment failures. The targeted real-PostgreSQL suite passed, and final-head CI is required before handoff. The first CI run also caught an exact mock expectation missing the added company scope; the expectation is updated and its 179-test suite passes. The final-head first CI attempt also stalled an assigned child task in the existing plan-approval scenario, then hit a heartbeat-run row lock timeout during fixture cleanup. The failed shard and aggregate check passed on an unchanged rerun of the same commit. Scenario assertions and timeouts were unchanged. The failure and rerun are retained in [CI run 41116120430](https://github.com/paperclipai/paperclip/actions/runs/41116120430). ## Risks - Historical transcript diagnostics no longer load on a cold chat open. Saved messages and conversation events remain available; diagnostic run views keep their full records. - Cursor paging must not omit comments with equal timestamps. Database tests cover timestamp/UUID ordering and the complete set. - Prepended or late content can move the reader. Browser checks cover retained row identity, viewport offsets, mounted drafts, and Back. Previously loaded pages stay in the query cache for its existing lifetime. - Local synthetic evidence does not establish deployed staging latency. This PR does not merge or deploy the change. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact backend deployment ID and context window were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected checks; limits are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 15:12:24 -05:00