DottaandPaperclip
6dcf4160dd
## Thinking Path > - Paperclip lets people manage work done by agents. > - A standing agent chat can hold months of conversation. > - The chat route forced full-history mode, bypassing compact task loading. > - A cold visit fetched lifetime run summaries and hydrated historical transcripts. > - This PR starts with 20 recent messages and loads earlier pages on upward scroll. > - It preserves the reader's position, current questions, and draft while history grows. ## Linked Issues or Issue Description **What happened?** Standing agent chats forced full-history loading. A long conversation fetched full activity and run history before it became readable. Earlier comments also used an explicit load control. **Expected behavior** Open a small recent conversation quickly. Load older messages when the reader scrolls upward. Show an inline loading state. Keep the visible message and composer mounted through paging, live updates, and Back navigation. **Steps to reproduce** 1. Open a standing agent chat with hundreds of messages and completed runs. 2. Use a fresh browser query cache. 3. Observe full activity and run reads plus transcript requests for historical runs. 4. Compare the recent window and upward paging after this change. Related work: #18312 and #18323 restored compact task and collection loading. #14848 added reopening of visited agent chats. A GitHub search found no open PR for this chat-history optimization. ## What Changed - Return 20 contiguous recent comments for standing chats, with a stable oldest cursor. Keep the existing compact task exchange contract. Use the existing company/issue/time index for cursor reads. - Keep chats compact even if their task view previously expanded history. Retain observed messages through live catch-up and cache-backed return visits. - Keep fetched history pages, acknowledged sends, and the observed recent cursor separate. Reconcile live replies without losing history after a send or deleted cursor. - Hydrate metadata only for newly revealed message/run IDs; preserve prior batches while paging. Slow-send recovery refreshes the recent window. Receipts failures have an independent retry. - Hydrate run summaries only for loaded messages and the latest turn. Validate batches of at most 100 run IDs and keep the existing issue/company access rules. - Reuse hydrated terminal native streams when only history membership changes. Preserve retry, resume, mode changes, reconnect recovery, and interrupted recovery reads. - Load older messages, decision receipts, and task handoffs through upward scrolling. Support wheel, touch, and keyboard input. Keep pending questions visible. - Reserve a status slot while loading. Preserve a logical message anchor and the mounted composer. Pause failed page reads until an inline Retry. - Add database, route, controller, and scroll regressions, plus an opt-in production-build browser rig for long chats on desktop and mobile. Update design and feature/access documentation. ## Verification - The real PostgreSQL cold-open suite passes all 12 tests, including contiguous pages with equal timestamps, complete history, company denial, and scoped run IDs. - Activity route and OpenAPI suites pass 53 tests. The route run allowed local cold-import overhead; CI retains the repository's normal timeouts. - Three new paging regressions fail against the unchanged source. The candidate's focused paging/window suite passes 26 tests. Broader UI verification is recorded below when complete. - Desktop and mobile browser acceptance covers recent-only startup, per-frame reveal geometry, delayed and failed older pages, stable reading anchors and drafts, Back, older comment links, and one persisted live reply. - Full workspace typecheck and build passed locally. The final UI source also passes its typecheck and production build. - Affected UI suites pass: 212 controller/history/paging tests, 352 thread/scroller tests, 26 native-stream tests, plus metadata batching and live-cache hydration coverage. The issue mutation/access route suite passes 179 tests. - All 22 final desktop/mobile browser journeys pass, including delayed image growth. The isolated second-page request proof also passes six repeated desktop/mobile runs. A separate 6,000-message, 3,000-run fixture passes cold-open and delayed-paging checks on both screen sizes, still opening 20 messages and loading one 50-message page. - Final matched production timings use baseline `aa58fa1073` (`index-BaRiMbxd.js`) and candidate UI source `f9a26b9783` (`index-DPWTbsFF.js`, identical application source at final head `bf71ffd5af`). Chromium 151.0.7922.34, 1440×900, 4× CPU throttle, no added network latency, browser HTTP cache disabled, fresh application cache for each cold visit. Both variants block service workers and use the same warm local API server and immutable 600-message/300-run fixture. Only document HTML is intercepted for both variants; API requests are real. The common API revision is `109981f8c1`; subsequent commits change only UI and tests. | Usable conversation | Baseline median (range) | Candidate median (range) | | --- | --- | --- | | Cold direct URL | 6.57 s (5.69–7.45) | 3.07 s (2.50–3.28) | | Warm in-app return | 3.76 s (3.54–4.76) | 0.67 s (0.58–1.46) | | Back return | 2.53 s (2.12–4.20) | 0.51 s (0.47–1.18) | Five pairs alternate order. Every paired difference favors the candidate for all three journeys. Readiness requires a revealed, fully visible latest reply above the composer and an editable composer. Return timing starts after the Tasks route renders. Cold API requests fall from 953–954 to 71; API wire bytes fall from 517–559 KB to about 49 KB. Separate CPU profiles and request waterfalls confirm the removal of lifetime event hydration and are excluded from the five timing pairs. Both variants have the same expected missing optional plan document (404) and no browser exceptions. Control recent-window API reads have medians of 37 ms / 32 ms, with shared-host outliers up to 376 ms. The 6,000-message recent-window SQL uses `issue_comments_company_issue_created_at_idx` (22 index rows read for a 21-row probe). Geometry assertions retain a 2-pixel tolerance for the reading anchor and first-reveal layout. These are local synthetic measurements, not deployed staging latency. Earlier timing probes are excluded because they did not isolate service workers or composer clearance. Final head `bf71ffd5afb5684ba9105a3127f617e7adf5139c` has 55 successful checks, three optional skips and one neutral result, with no failures or pending checks. Greptile is 5/5 with no open review findings. ### Failure record Early local attempts exposed an unconditional scroller wrapper that changed the old task layout, and automatic retries on chat page failures. Both were corrected. The original task scroller structure stays in place. Metadata request assertions wait for initial connection recovery to settle, then verify only the new page’s reads; retry/resume/reconnect revalidation remains intentional. The initial review caught a send-before-scroll cache defect; its controller regression failed before the fix. Later rapid-paging checks caught scroll readiness being gated by metadata after the first reveal; readiness now stays enabled once revealed. Live replies use the separate acknowledged-message cache, then catch up from the server cursor. Visual review exposed a pagination wrapper bypassing existing composer clearance. The body is again a direct scroller child; cold-open browser checks require the full latest reply above the composer. A resize-observer regression failed before the fix and now passes. Controlled delayed inline images prove both pinned following and a stable reading anchor on desktop/mobile. The first performance fixture lacked valid native event envelopes and caused missing-log fallbacks. It is excluded from latency claims. The final fixture uses saved native assistant events and conversation session generations. The live test uses a temporary fixture-agent key, attributes the reply to an existing synthetic run, and revokes the key. It starts no model or agent process. Local database attempts failed during a Docker shutdown. The retained disposable database was restored and its suite passed. Early fixture errors included a missing conversation identity field and session generation. Mobile test setup also selected an offscreen sidebar link and reused queued human input. Optional announcements could intercept input; the fixture now dismisses them before the journeys. The scroll observer now waits for the delivered wheel event and settled animation frames, including when the viewport is already at the top. A concurrently running live test briefly changed another probe’s row count; the final acceptance and timing runs were sequential on separate fixtures. The final rig uses the visible mobile navigation and a saved agent reply. Anchor observation waits for the user's scroll to settle before measuring the effect of live updates. The full local `pnpm test:run` attempt hit repeated embedded PostgreSQL initialization failures and failures in untouched native-session executor suites; it was stopped after the repeated environment failures. The targeted real-PostgreSQL suite passed, and final-head CI is required before handoff. The first CI run also caught an exact mock expectation missing the added company scope; the expectation is updated and its 179-test suite passes. The final-head first CI attempt also stalled an assigned child task in the existing plan-approval scenario, then hit a heartbeat-run row lock timeout during fixture cleanup. The failed shard and aggregate check passed on an unchanged rerun of the same commit. Scenario assertions and timeouts were unchanged. The failure and rerun are retained in [CI run 41116120430](https://github.com/paperclipai/paperclip/actions/runs/41116120430). ## Risks - Historical transcript diagnostics no longer load on a cold chat open. Saved messages and conversation events remain available; diagnostic run views keep their full records. - Cursor paging must not omit comments with equal timestamps. Database tests cover timestamp/UUID ordering and the complete set. - Prepended or late content can move the reader. Browser checks cover retained row identity, viewport offsets, mounted drafts, and Back. Previously loaded pages stay in the query cache for its existing lifetime. - Local synthetic evidence does not establish deployed staging latency. This PR does not merge or deploy the change. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact backend deployment ID and context window were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected checks; limits are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
Issue-detail performance baseline
Run the repeatable baseline from the repository root:
pnpm exec playwright test --config tests/perf/issue-detail/playwright.config.ts
The rig starts an isolated seeded Paperclip instance, runs five samples for each scenario/profile, and writes median-ready raw data plus a Markdown table to test-results/issue-detail-perf/.
Scenarios:
- S1 warm in-app navigation: loads the Issues list, clears the waterfall, and clicks the seeded issue.
- S2 cold open: creates a fresh browser context and deep-links to the seeded issue.
Profiles:
- Unthrottled.
- Fast 4G network with 4x CPU slowdown.
Override the sample count (minimum five) or port when needed:
PAPERCLIP_ISSUE_PERF_RUNS=7 PAPERCLIP_ISSUE_PERF_PORT=3210 pnpm exec playwright test --config tests/perf/issue-detail/playwright.config.ts
Outputs include baseline.md, baseline.json, and a Chrome trace for the first run of each scenario/profile. Open *.trace.json in Chrome DevTools Performance to inspect the issue-detail:* user-timing marks.
Set PAPERCLIP_ISSUE_PERF_PROFILE=1 to capture actual Chrome CPU samples and
conversation-readiness transitions in one diagnostic sample per scenario/profile.
Import the resulting *.cpuprofile in DevTools and inspect *.readiness.json for
the task, comments, metadata, activity, runs, runtime, plan and transcript gates.
Each gate records changes with detail.ready; use its last transition before
content paint when a gate becomes pending after mount. These diagnostic runs
include profiler overhead. Run without the flag for before/after latency claims.
Readiness marks stay in the local browser and contain only fixed gate names and
booleans; the harness enables them before navigation.
For bundle comparisons, build the UI and serve the production assets from an
isolated local instance with PAPERCLIP_UI_DEV_MIDDLEWARE=false. The default
managed server above uses development middleware, which does not reproduce
production chunk loading. Keep the server, database, browser profile and sample
count the same for both builds, and avoid concurrent builds or test suites while
measuring latency.
Copy the first run's output outside test-results/issue-detail-perf, then reuse
its persisted task with PAPERCLIP_ISSUE_PERF_SEED_FILE=/path/to/before/baseline.json
and PAPERCLIP_ISSUE_PERF_BASE_URL pointing to that disposable instance. The
harness validates the task identity before measuring. Network spans in the trace
use recorded request start/end timestamps relative to task navigation; they
include completed requests observed during the sample, even after content paint.
Task layout stability
compact-startup.spec.ts keeps a synthetic local reproduction with 24 messages,
a plan, three documents and three saved PR cards. It holds the enriched task
response, records each visible frame's card count and geometry, and rejects
cards inserted after the conversation becomes readable. It also checks document,
older-comment and unknown anchors without full comments/activity/run reads.
Run it with the same loopback-only URL override below. The fixture URL is saved
in test-results/compact-startup-fixture.json for manual inspection and reuse.
It also checks a delayed Coordinator preference with Classic enabled, index-based
Coordinator list navigation, and the responder identity through reassignment,
comment acknowledgement and legacy runtime discovery. The last case uses a
deterministic local process, records identity continuity on each frame, and saves
startup/running screenshots. It does not call a model provider.
Each send fixture uses a unique task title so repeated runs cannot reuse a
previous task and its cancelled run through ordinary task deduplication.
layout-stability.spec.ts exercises the real task route, including Inbox
navigation, with 160 Markdown comments and a resolved interaction. It delays
initial comments, tests failures and retry, tracks a reading anchor across
media growth, composer resizing, properties toggles, older-page loading and
Back navigation, and covers mobile reduced motion and explicit comment links.
The eight scenarios also exercise same-task hash navigation without remounting
on desktop and mobile, and stalled native/log reads that expose Retry after
the 15-second request deadline.
plan-startup.spec.ts also opens tasks with and without a saved plan while
blocking the separate plan endpoint. Both must reveal the saved conversation
from the task response without issuing that redundant read.
It also checks task-list navigation, where a list-only snapshot needs one normal
plan fetch and must still show a saved plan when present.
Videos and per-frame layout.json measurements are written under each test's
output directory. The two-pixel assertion measures a logical row's viewport
offset; total scroll offset legitimately changes when history is prepended.
Ordinary task opens defer the concealed history and composer until their initial data is ready. The skeleton and scroll/composer containers retain their geometry. Comment-link navigation prepares content eagerly so the target can be positioned before reveal. Revealed content stays mounted during refresh. The draft scenario holds a background task read, checks the same editor remains mounted, reloads the saved draft, sends one reply, and independently checks its persisted comment.
To run against an existing disposable local test-drive instance:
PAPERCLIP_ISSUE_PERF_BASE_URL=http://127.0.0.1:3102 \
pnpm exec playwright test --config tests/perf/issue-detail/playwright.config.ts layout-stability.spec.ts
The suite creates its own company and fixtures. The URL override accepts only loopback origins and rejects remote hosts. Use a disposable local instance, not a shared or production instance. Without the override, the harness starts its own isolated instance as before. Live provider walkthroughs additionally require a configured native Paperclip runner and Codex authentication; deterministic browser fixtures do not substitute for watching an actual provider run.
For the live walkthrough, first assign a disposable task to a native Paperclip
runner configured with the Codex provider. Its scratch project should contain
a small sum.mjs fixture. The script sends a paced job, scrolls up, disconnects
and reconnects, steers a follow-up, and records through completion:
PAPERCLIP_LAYOUT_LIVE_URL=http://127.0.0.1:3102/LAY/issues/LAY-8 \
node tests/perf/issue-detail/live-feed.walkthrough.mjs
Inspect test-results/task-layout/live-acceptance/ for the video, screenshots,
per-frame positions, layout shifts, browser timing counters, and reading-anchor
measurement. The live script rejects non-local URLs and uses real provider
capacity. Review the video as well as the assertions: layout-shift scores alone
do not capture programmatic scrolling or replacement flashes.
Long-running agent chats
The separate agent-chat performance rig covers recent-only startup and scrolling history with long-lived conversations. It is opt-in and does not run in this task-detail rig.