Commit Graph
3 Commits
Author SHA1 Message Date
DottaandPaperclip 7d8bab773d perf: avoid full JIT and activity scans in Mine unread badges (#17973)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - Mine includes tasks the person created, was assigned, or
participated in.
> - Its unread badge ranks the visible personal top 100 before checking
unread state.
> - Correlated participation and privacy predicates can trigger
expensive PostgreSQL JIT work.
> - This change joins native participant IDs before ranking a narrow
candidate projection.
> - A partial actor index avoids scanning company activity history for
each badge.
> - Local representative scoped reads improve while dense unrestricted
overhead remains explicit.

## Linked Issues or Issue Description

**What existing behavior does this improve?** The Mine unread badge in
the company sidebar.

**Current behavior:** A fresh five-minute window recorded 402 ranking
calls and 31.7 seconds of SQL time. Correlated predicates can spend more
time compiling than executing.

**Proposed behavior:** Build the same company-scoped participant set,
use native issue keys, apply fresh authority before the top-100 limit,
then retain the original canonical rank and second unread query.

**Reason and benefit:** Reduce full JIT compilation and avoid an
unindexed company-history activity scan. Realistic activity-only
participation is included in the measurement.

**Breaking changes:** No API or policy change. This PR adds one
concurrent partial index migration. Related Mine rules in #16356 and
#17756 are preserved.

## What Changed

- Join distinct native participant IDs; guard exact legacy text before
UUID conversion.
- Skip redundant activity participation already covered by
creation/assignment. Sort a narrow eligible candidate projection before
canonical activity ranking.
- Share existing pending-request and user-activity predicates with
ordinary Mine reads.
- Add partial `activity_log(company_id, actor_id, entity_id)` for
user-authored issue activity. Register its idempotent concurrent
migration with the existing top-level/invalid-index-repair runner.
- Add real PostgreSQL coverage for visible top 100,
unread/archive/source semantics, exact IDs, company scope and fresh
grant/private-project access.
- Add a pinned service benchmark with separate pre-index/indexed
databases, identical data, full positional equality, ordinary-call
timings, plans and real migration/cancellation proof.

## Verification

- Fourteen focused PostgreSQL 17 tests pass (16.37 s), including
assembled badge routes and grant revocation. Full workspace typecheck
and build pass on the final production sources.
- The earlier full local suite is incomplete: native PostgreSQL startup
exhausted system semaphores, independently reproduced with empty initdb.
All sixteen failed cases passed with task-owned Docker PostgreSQL 17.
This is not counted as a full local-suite pass. Full Linux CI passes on
final head `f44e46ce580e8345209d7165f57bd8bf696c517c`. Greptile and
Paperclip Review are both 5/5 with zero unresolved comments.
- Baseline `70c864633744cd96c0dc7394d631fc7119cd17d2`; source and DDL
hashes retained. Two warmups and five alternating samples per arm/cell.
Capture and EXPLAIN occur outside timed ordinary service calls.
- All twelve 300k-history cells match complete positional rows in both
SQL phases and final returned IDs. Half the personal tasks qualify
through activity alone; the actor index contains matching and
other-human rows.

| Local service case | Before median | After median | Before / after |
| --- | ---: | ---: | ---: |
| dense / unrestricted / JIT on | 18.33 ms | 25.06 ms | 0.73x |
| dense / scoped / JIT on | 909.30 ms | 221.10 ms | 4.11x |
| sparse / unrestricted / JIT on | 13.64 ms | 5.84 ms | 2.34x |
| sparse / scoped / JIT on | 719.07 ms | 10.79 ms | 66.64x |

Small paths add at most 0.6 ms at the median. Dense unrestricted paths
regress by 6.7–7.7 ms; dense scoped JIT-off is approximately neutral
(218.34→223.24 ms). Sparse JIT-off paths improve too. This is not a
uniform speedup.

The actual baseline fails the compiler-work assertion. The final query
avoids full JIT optimization/inlining and exceeds 1.5x where the old
plan triggers those passes. Small/sparse controls meet the 25% or 2 ms
allowance. Dense no-JIT cases are explicit diagnostics. The optional 60
ms absolute goal still fails at 221.10 ms on dense scoped reads:
substantial privacy-predicate execution remains after compilation is
removed. The failure is retained rather than counted as green.

The actual normal migration runner upgraded 300k activity rows in 668.79
ms including inspection, index build/validation and journal work. Rerun
took 21.83 ms; a deliberately cancelled build left an invalid index, and
retry repaired it in 187.71 ms. Concurrent reads/writes completed; the
final index was 188416 bytes. No migration was run on the hosted
database.

The million-row mixed-participation sparse control also passes all four
visibility/JIT pairs with complete equality: unrestricted JIT-on
13.44→5.76 ms; scoped JIT-on 807.76→11.59 ms. The actor branch visits 36
indexed rows rather than scanning company history. Actual
migration+runner took 255.06 ms; rerun 20.32 ms; cancellation
repair/retry 241.73 ms. Concurrent reads and writes completed without
failures.

Reproduce:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-mine-unread-badge.ts --baseline-ref=70c864633744cd96c0dc7394d631fc7119cd17d2 --history-decoys=240000 --verify-migration --assert-performance --evidence-dir=<directory>
```

Add `--fixture=sparse --history-decoys=940000` for the million-row
control. The default harness starts disposable embedded PostgreSQL and
ignores DATABASE_URL. Recorded evidence used PostgreSQL 17.11 through a
scratch constructor provider because the host native runtime has
semaphore limits. All normal database allocation/migration/client paths
are retained.

Source-only EXISTS/lateral alternatives were measured and rejected after
company scans or sparse regressions. A time bound would lose historical
participation, and existing indexes cannot select a user's activity.
This justifies the exceptional partial index migration. The exact sample
ranges, plans, failures, source hashes, migration footprint and
feedback-loop timing are in
`doc/plans/2026-10-27-mine-unread-badge-query.md`.

## Risks

- Dense unrestricted reads are slightly slower, and dense scoped reads
retain substantial authority work. The PR removes a compilation/lookup
bottleneck rather than all badge latency.
- The concurrent build scans activity history and can wait for
long-lived transactions. Local build/cancellation timings do not
guarantee the duration on larger or slower hosted databases.
User-authored issue activity gains one narrow index to maintain.
- Participation, visibility, ranking and legacy ID semantics must remain
exact; the real database regressions exercise those boundaries.
- Hosted improvement is unverified. The affected ranking SQL averaged
78.8 ms while HTTP badge requests averaged 752 ms without correlated
traces. No merge or deployment is included.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use and local code
execution. The runtime does not expose the exact model ID or context
window to this agent.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 07:20:08 -05:00
Devin FoleyandPaperclip a2d26a83fe fix(inbox): keep automatic work out of Mine by default
Keep channel and routine attribution out of Mine unless the person participates directly, is assigned the task, or receives a current addressed request. Apply the same source rules to unread state, archives, and task notifications. Preserve direct comments and current human action requests.

Restore the two expressions damaged by redaction during the earlier master merge.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-20 13:46:10 -05:00
DottaandPaperclip aded4197e5 fix: make multiplayer task toasts personal (#16153)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for work.
> - A company can have several human users and shared agents.
> - Live task events currently produce toasts for any user who can read the task.
> - This makes unrelated work interrupt other users.
> - This pull request checks each user's task participation before it permits a toast.
> - Users receive relevant task notifications while company views continue to refresh.

## Linked Issues or Issue Description

**What happened?**

Company task activity and agent failures produce noisy toasts for users who have no involvement in the task.

**Expected behavior**

A task qualifies when the viewer created it, commented on it, is its assigned user, deliberately acted on it under the Mine rules, or has an explicit user mention in its description or a non-deleted comment. Viewing, agent ownership, responsible-user attribution, and parent participation alone do not qualify.

**Steps to reproduce**

1. Sign in as two users in the same company.
2. Have the second user create and update a task that does not involve the first user.
3. Observe a task toast in the first user's company view before this fix.

The duplicate PR search found no existing change for personal task-toast eligibility.

## What Changed

- Extract the existing Mine participation predicate without changing Mine behavior.
- Compute task eligibility from stored tasks, comments, and run associations after WebSocket access checks. Use an exact indexed comment-mention lookup with `LIMIT 1`.
- Add optional `toastEligible` metadata to each viewer's event copy. Keep it off the shared event bus.
- Require eligibility for live task activity and task-linked run errors. Suppress generic agent errors.
- Continue cache updates for quiet events. Refetch visible comments when authenticated events omit hydration details. Ignore stale socket callbacks after account or company changes.
- Document toast semantics and add feature-map and agent-access-map coverage.
- Add persisted-data, socket, UI, and authenticated browser regression coverage.

## Verification

- Final commit: `1dc4f9402c5883744738cfe236ad00fa0bb5ca4c`.
- All 55 current-head checks passed. Two Storybook checks were skipped because this change does not affect that coverage.
- Greptile approved this exact commit with 5/5. The security scan passed. Both review threads are resolved.
- Local `pnpm build`, `pnpm -r typecheck`, `pnpm check:token-gates`, and `node scripts/check-maps.mjs` passed.
- The focused socket/UI suite passed 123 tests. The persisted eligibility suite passed 9 tests, including long history, exact viewer matching, and regex escaping. The existing Mine service suite previously passed 140 tests.
- The named browser case `task toasts follow personal involvement in multiplayer` passed against the final commit in an isolated authenticated instance. It used two real sessions, real WebSocket delivery, and persisted tasks, edits, comments, and mentions. It verified quiet unrelated activity, viewing alone, first and historical mentions, own-action suppression, visible-task suppression, and live cache refresh.
- A prior-head server shard passed its single rerun after a fixture-cleanup deadlock. The final-head CI run passed without a rerun.
- The broad local run remained in progress when CI completed the full test suite. Local focused checks and CI provide the completed test evidence.

## Risks

- This intentionally reduces live notifications. A new client stays quiet against an older server that omits eligibility.
- Eligibility queries add work to live event delivery. A lookup failure suppresses the toast and still delivers the authorized event.
- Deleted mentions cease to qualify unless another participation rule applies. Historical comment authorship retains the existing Mine behavior.
- No database migration or notification-settings change is required.

## Model Used

OpenAI Codex with GPT-6 assisted with implementation, tests, documentation, and PR preparation. It used repository tools, shell commands, and browser automation. The runtime did not expose an exact backend revision or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before requesting merge


Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-13 08:09:19 -05:00