mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 12:59:13 +01:00
perf: avoid company scans in task privacy ancestry (#18024)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task reads use one shared privacy predicate for detail, list, badge, and derived access. > - Its recursive parent join can scan all tasks for each frontier, including tasks with no parent. > - A 1,200-task local reproduction inspects 1,440,000 recursive task rows without finding any parent. > - Each task has at most one same-company parent, so the query can probe that parent directly. > - This pull request exposes that single-parent contract to the planner and keeps the existing access rules. > - The result removes repeated scans and avoids JIT in the measured large callers. ## Linked Issues or Issue Description Refs #14718 and #17973. I searched public issues and open/closed PRs for privacy ancestry and privacy performance work. No competing open fix was found. **What happened?** Task privacy checks repeat company task scans and private-project checks during list and unread badge reads. The recursive join may do this work even when every parent is NULL. **Expected behavior** Read each actual same-company parent through the existing unique task index. Preserve current grants, project membership, user intersections, cycle termination, and visibility before pagination. **Steps to reproduce** Run `authorization-privacy-query.test.ts` against the pre-fix query. The semantic ID check passes. Its plan-work assertion fails at 1,440,000 rows against a 3,600-row bound. The same test passes after this change. **Paperclip version or commit** Measured baseline `8279c990080b15abdcd9981f5793290fab37f9ad`. The branch was replayed onto `71b448b9cb841e6fa18fe883585430c5081245da`. Both have identical pre-fix authorization, issue-service and participation-helper sources. Treatment source SHA-256 is `a99faf974070a35f2786c92407e023c668c1f7b99b6604f6281c89bce089b1a8`. **Deployment mode** Source checkout, isolated local PostgreSQL 17.11 on ARM64 Linux. No hosted write, migration, merge, or deployment was performed. ## What Changed - Skip NULL privacy parents. Use a lateral lookup for the exact same-company parent through existing indexes. - Keep `UNION` cycle detection and every existing principal, grant, private-project and membership expression. Both `LIMIT 1` clauses are redundant: one anchor and one unique parent give at most one row per recursive generation. The query still follows a complete deep chain. - Cover open ancestors inside private projects, including current project visibility and ancestor grant revocation. Place the benchmark under `server/scripts/` with ordinary package imports. - Add real PostgreSQL regressions and a pinned-baseline benchmark for complete badge/list calls. Keep all 36 measured cells, ranges, neutral paths, failed experiments, and the single-chain proof in [the query proof](https://github.com/paperclipai/paperclip/blob/codex/perf-task-privacy-ancestry/doc/plans/2026-10-28-task-privacy-query.md). ## Verification Local warm full-service medians, with two warmups and five alternating measured calls per arm: | Scoped caller | Baseline | Treatment | Ratio | | --- | --- | --- | --- | | Dense badge, default JIT | 645.33 ms | 77.11 ms | 8.37x | | Dense list, default JIT | 214.77 ms | 29.87 ms | 7.19x | | Sparse list, default JIT | 117.92 ms | 13.22 ms | 8.92x | | Dense list, JIT off | 55.69 ms | 30.62 ms | 1.82x | | Sparse list, JIT off | 16.61 ms | 15.12 ms | 1.10x | All 36 cases match every positional SQL field and row order across all captured queries. Complete final badge IDs and list objects also match. Both badge phases and ordinary list enrichment remain in the measured caller. Dense/sparse fixtures have 1,200 tasks, 300,000 activity rows and 24,000 comments, with wide random content. Small, unrestricted, private fan-out, 64-level ancestry, cycle, and JIT on/off controls are included. The dense ranking plan reduces recursive task and project work from 720,000 visits each to zero. Its cost estimate falls from 103,000.61 to 20,458.73 and JIT disappears. Separate EXPLAIN execution falls from 524.04 to 34.34 ms. The dense JIT-off main list statement falls from 29.52 to 1.89 ms; remaining service work limits the full-call gain. **Retained failed goals:** `--assert-performance --max-dense-ms=60` exited 1. The original 60 ms dense badge goal remains unmet at 77.11 ms. Four full-service cases also miss the exploratory blanket 2x goal. These criteria remain in the harness and are not counted as green. The delivered optimization is proved by the deterministic query-work red/green test and complete result equality. Small and unrestricted controls meet their original 25% or 2 ms allowance. All 12 unrestricted cells use byte-identical SQL and parameters in both arms; their overlapping timing variation is neutral/inconclusive. Modest slower scoped controls are retained, including small JIT-off list 16.18→18.78 ms and cycle JIT-off list 14.78→16.29 ms. There is no claim that all paths improve. - Final head `328078f494d6ec585936fb75a9410ad39eefd099`: full CI passes (55 success, 1 neutral, 2 skipped); fresh Greptile is 5/5 with zero unresolved review threads. The optional Paperclip Review ended neutral because its worker stopped; its earlier two specific recommendations are both addressed below. - Final-runtime focused Docker PostgreSQL run: **44/44 tests passed** across ancestry work, production privacy, unread badge, and creation visibility suites. Includes fresh revocation, private projects, grant direction, three-principal agent intersections, cycles, deep inheritance, foreign-company parents, and access before LIMIT. - `pnpm -r typecheck` and `pnpm build`: passed before replay. The real PostgreSQL ancestry suite passed after replay (5/5 tests, 17.44 seconds). The added private-project ancestor case then passed in the six-test suite (43.04 seconds). Database and server typechecks and builds passed after replay; review follow-ups change tests and benchmark location only. A small relocated-command smoke preserves byte-identical generated SQL and complete result comparisons. - `pnpm test:run`: attempted, then stopped with exit 143 after 16 native database bootstrap failures in 940 executor cases. An independent empty native `initdb` fails with `semget` semaphore exhaustion before application SQL. This local full suite is incomplete, not passed. All 16 affected cases then passed on real PostgreSQL 17 Docker in 155.04 seconds (924 other cases were intentionally filtered out). - An earlier focused command passed 35 tests but skipped eight after a 20-second database setup timeout. The later 44/44 pass resolves that coverage gap; the failed command is retained. - The first CI e2e shard attempt completed functional split-handoff assertions, then cleanup run cancellation hit a one-second row-lock timeout. The unchanged local-trusted cleanup path does not use the changed privacy predicate. A single same-head job retry passed. The original failure is retained. - The first lateral-only draft removed scans but raised estimated ancestry rows from 11 to 101 and retained JIT. The per-generation fence fixes that estimate. No failed attempt was discarded. Reproduce the measured caller pair: ```sh pnpm --filter @paperclipai/server exec tsx scripts/benchmark-task-privacy.ts \ --baseline-ref=71b448b9cb841e6fa18fe883585430c5081245da \ --evidence-dir=/tmp/task-privacy-proof ``` The full matrix takes about 107 seconds locally, including 8 seconds of setup. Capture and EXPLAIN run outside ordinary timing samples. Only the old authorization module changes between arms; caller source and all relevant query helpers remain constant. Hosted request latency remains unverified. ## Risks - The per-generation bound depends on the single anchor and unique scalar parent contract. A future multi-parent model must revise this query and its deep/cycle tests. - Tiny tables can still use a cheap sequential lookup for a real parent. The guard prevents broad scans for absent parents; it does not force index use everywhere. - The 60 ms badge goal and blanket full-service 2x goal remain failed. Unchanged enrichment, application work and local variation remain visible in the measurements. - This changes shared read authorization SQL. No permissions are cached, no authorization moves after LIMIT, and no schema or startup change is included. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, local code execution and real PostgreSQL profiling. The exact model ID and context-window size are not exposed by this session. No more specific ID or window size is claimed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
52b86c2032
commit
a30194e008
5 files changed
+549
-3
No files matched your search
@@ -0,0 +1,198 @@
|
||||
# Task privacy ancestry query proof
|
||||
|
||||
The shared task privacy predicate belongs to `authorization.ts`. Detail reads,
|
||||
list/search/count filters, unread badge phases, and derived authorization all use
|
||||
this predicate. This change preserves that owner and the current access rules.
|
||||
|
||||
## Cause and change
|
||||
|
||||
Measured baseline commit: `8279c990080b15abdcd9981f5793290fab37f9ad`.
|
||||
The branch was later replayed onto `71b448b9cb841e6fa18fe883585430c5081245da`.
|
||||
Both commits have byte-identical pre-fix authorization, issue-service and
|
||||
participation-helper sources. The reproduction command uses the newer equivalent
|
||||
pre-fix baseline so unrelated upstream voice-schema changes stay constant in
|
||||
both arms. The measured source hashes below are unchanged by that replay.
|
||||
The previous recursive join can scan every company task and check its project
|
||||
for each privacy frontier. In the 1,200-open-task regression this reads 1,440,000
|
||||
task rows even though every parent is NULL. The semantic ID assertion passes;
|
||||
the deterministic work assertion fails at the unchanged baseline.
|
||||
|
||||
The replacement skips NULL parent frontiers and reads the exact same-company
|
||||
parent through a correlated lateral lookup. The existing unique task ID and
|
||||
company/ID index support the lookup. No schema or migration change is needed.
|
||||
|
||||
Both `LIMIT 1` clauses are redundant under the current schema and anchor:
|
||||
|
||||
1. The anchor selects at most one row by the outer task's ID and company.
|
||||
2. Each task has one scalar privacy parent ID. That ID selects at most one
|
||||
same-company parent. The lateral limit prevents the planner from replacing
|
||||
the parameterized lookup with a company-wide hash scan.
|
||||
3. Inductively, each recursive generation has at most one row. The recursive-arm
|
||||
limit exposes this fact to the planner. It does not limit chain depth.
|
||||
4. `UNION` still removes visited rows and terminates cycles. An open parent
|
||||
outside a private project still stops traversal at the same place.
|
||||
|
||||
A first lateral-only draft removed repeated scans but raised estimated ancestry
|
||||
rows from 11 to 101. This retained JIT compilation in the representative callers.
|
||||
The per-generation cardinality fence addresses that estimate. Tiny tables can
|
||||
still use a cheap sequential lookup for a real parent. The work guard prevents
|
||||
company scans for absent parents; it does not force every small-table index plan.
|
||||
|
||||
Implicit principals, live task grants, private project membership, responsible
|
||||
user and external authorizer intersections, and company membership checks use
|
||||
the same expressions. There is no cached permission result or visibility check
|
||||
moved after pagination.
|
||||
|
||||
## Local proof and reproduction
|
||||
|
||||
Run the real service benchmark with a pinned source baseline:
|
||||
|
||||
```sh
|
||||
pnpm --filter @paperclipai/server exec tsx scripts/benchmark-task-privacy.ts \
|
||||
--baseline-ref=71b448b9cb841e6fa18fe883585430c5081245da \
|
||||
--evidence-dir=/tmp/task-privacy-proof
|
||||
```
|
||||
|
||||
It uses disposable embedded PostgreSQL by default. An explicit
|
||||
`PRIVACY_BENCHMARK_ADMIN_URL` selects an isolated loopback PostgreSQL server.
|
||||
The harness creates and removes only its own `privacy_ancestry_*` database.
|
||||
|
||||
Adding `--assert-performance --max-dense-ms=60` reproduces the retained failed
|
||||
stretch-goal command. It must not be used as evidence of a green performance gate.
|
||||
|
||||
The harness loads old `authorization.ts` from Git, rewrites only its relative
|
||||
imports, and uses the unchanged real `issueService` for both arms. It rejects
|
||||
unisolated runtime dependency changes. Both arms use one identical database,
|
||||
index set, user, task IDs, clock inputs and session settings. It records source
|
||||
hashes, baseline revision, PostgreSQL and dependency versions, settings, and
|
||||
fixture sizes.
|
||||
|
||||
Ordinary service timings exclude SQL capture, assertions, and EXPLAIN. Each case
|
||||
has two warmups and five measured calls per arm in alternating order. Capture
|
||||
runs afterwards. Every positional SQL field and row order must match across all
|
||||
captured queries. The entire final badge output and list objects must also match.
|
||||
The badge runs both ranking and unread phases. The list includes its ordinary
|
||||
post-query enrichment. These are local service measurements, not HTTP latency.
|
||||
|
||||
The matrix includes small, dense and sparse participation with JIT on/off, plus
|
||||
64-level private ancestry, 1,200-task private fan-out, and a 32-task cycle. Dense
|
||||
and sparse fixtures contain 1,200 tasks, 60,000 task activity rows, 240,000 unrelated
|
||||
run activity rows, and 24,000 comments. Their activity/comment payloads use 4 KiB
|
||||
Base64 text from 3 KiB of random bytes per value. Half of personal participation comes from the creator and
|
||||
half from user activity. The dense fixture has 1,200 personal tasks; sparse has
|
||||
12. Controls also omit the privacy predicate entirely.
|
||||
|
||||
Acceptance was set before the final run: no recursive task reads for absent
|
||||
parents, at least 2x gain on representative scoped dense/sparse callers, complete
|
||||
output equality, and small/open controls within 25% or 2 ms. The original 60 ms
|
||||
dense scoped badge target is a separate explicit check. It is not treated as
|
||||
passed without the final measured result.
|
||||
|
||||
## Correctness and failures retained
|
||||
|
||||
The dedicated real PostgreSQL tests cover NULL parents, child-to-descendant grant
|
||||
flow, sibling/ancestor denial, revocation, a 64-level root-only grant, cycles,
|
||||
public traversal barriers, foreign-company parent pointers, all three agent/user
|
||||
principals, inactive user membership, private project membership, and access
|
||||
checks before list pagination. Existing production privacy, creation visibility,
|
||||
and unread badge tests exercise the shared consumers and HTTP routes.
|
||||
|
||||
The first baseline work test failed as intended: 1,440,000 rows against the 3,600
|
||||
bound, after semantic equality passed. A wrong relative-path edit then reran the
|
||||
old query; that failed command is retained and is not a treatment pass.
|
||||
|
||||
The first four-suite lateral-only run passed 35 tests, but eight creation tests
|
||||
did not run because their real database setup exceeded its 20-second hook limit.
|
||||
That command failed. Final-runtime checks and the normal CI result are recorded
|
||||
separately below. Exploratory service times collected during other builds are
|
||||
retained as diagnostics and are not the published final benchmark.
|
||||
|
||||
## Feedback loop
|
||||
|
||||
The first deterministic red command took 24.31 seconds including PostgreSQL
|
||||
setup and the plan assertion. The baseline dense caller capture took 89.81
|
||||
seconds including 11.96 seconds of database setup. The first lateral-only dense
|
||||
comparison took 93.37 seconds including 12.22 seconds of setup. The small
|
||||
cardinality smoke kept the next query experiment narrow. A reserved quiet slot
|
||||
is required for the final published measurements; other agents pause builds,
|
||||
tests and seeds during that slot.
|
||||
|
||||
## Final measured result
|
||||
|
||||
The full matrix completed in 107.20 seconds, including 8.22 seconds of database setup. Other agents paused builds, tests and seeds. PostgreSQL was 17.11 on ARM64 Linux with 128MB shared buffers, 4MB work memory, default 100,000 JIT threshold and 500,000 full-optimization thresholds. Node was v26.4.0, postgres.js 3.4.9, and Drizzle 0.45.2.
|
||||
|
||||
All 36 cells passed complete positional SQL and final service output/order comparison. All absent-parent recursive arms read zero task rows. The deterministic work regression is green.
|
||||
|
||||
**The combined strict benchmark command exited 1.** The 60ms dense scoped badge goal remains unmet at 77.11ms. Four full-service cells also miss the predeclared2x goal: dense JIT-off list 1.82x, sparse JIT-on/off badges 1.60x/1.71x, and sparse JIT-off list 1.10x. These criteria remain in the harness and are not reported as passed. The deliverable is the independently proven query-work optimization with preserved results.
|
||||
|
||||
Warm ordinary full-service timings below are milliseconds, median(min–max), with five measured calls per arm after two warmups. The list page limit is 100.
|
||||
|
||||
| Fixture | Scope | JIT | Caller | Baseline | Treatment | Ratio |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| small | scoped | on | badge | 6.54 (5.27–7.60) | 5.74 (5.53–7.01) | 1.14x |
|
||||
| small | scoped | on | list | 13.24 (11.67–15.46) | 12.06 (10.57–14.84) | 1.10x |
|
||||
| small | scoped | off | badge | 10.02 (8.09–11.62) | 9.26 (6.91–11.89) | 1.08x |
|
||||
| small | scoped | off | list | 16.18 (14.36–17.22) | 18.78 (13.23–22.03) | 0.86x |
|
||||
| small | unrestricted | on | badge | 6.53 (5.51–7.56) | 6.93 (5.96–7.90) | 0.94x |
|
||||
| small | unrestricted | on | list | 16.59 (15.28–17.60) | 19.02 (14.90–20.94) | 0.87x |
|
||||
| small | unrestricted | off | badge | 5.01 (4.99–6.70) | 6.49 (5.20–7.38) | 0.77x |
|
||||
| small | unrestricted | off | list | 12.43 (10.19–17.69) | 11.54 (10.66–12.70) | 1.08x |
|
||||
| dense | scoped | on | badge | 645.33 (607.50–1184.54) | 77.11 (54.52–121.13) | 8.37x |
|
||||
| dense | scoped | on | list | 214.77 (186.22–264.03) | 29.87 (24.21–33.64) | 7.19x |
|
||||
| dense | scoped | off | badge | 427.80 (323.86–452.47) | 59.91 (55.82–75.45) | 7.14x |
|
||||
| dense | scoped | off | list | 55.69 (54.20–66.11) | 30.62 (29.64–37.22) | 1.82x |
|
||||
| dense | unrestricted | on | badge | 41.05 (34.52–42.89) | 40.30 (36.24–42.94) | 1.02x |
|
||||
| dense | unrestricted | on | list | 34.99 (28.71–92.45) | 43.25 (29.16–83.33) | 0.81x |
|
||||
| dense | unrestricted | off | badge | 39.27 (37.17–50.13) | 42.41 (35.10–43.88) | 0.93x |
|
||||
| dense | unrestricted | off | list | 30.33 (28.02–55.84) | 28.33 (27.35–44.69) | 1.07x |
|
||||
| sparse | scoped | on | badge | 10.98 (10.11–11.67) | 6.84 (6.08–8.13) | 1.60x |
|
||||
| sparse | scoped | on | list | 117.92 (114.43–127.75) | 13.22 (12.97–18.58) | 8.92x |
|
||||
| sparse | scoped | off | badge | 10.77 (10.59–10.91) | 6.30 (5.76–6.51) | 1.71x |
|
||||
| sparse | scoped | off | list | 16.61 (15.68–16.97) | 15.12 (12.08–16.08) | 1.10x |
|
||||
| sparse | unrestricted | on | badge | 5.57 (5.12–7.80) | 6.81 (5.40–8.29) | 0.82x |
|
||||
| sparse | unrestricted | on | list | 12.70 (11.74–15.09) | 13.19 (11.27–14.80) | 0.96x |
|
||||
| sparse | unrestricted | off | badge | 6.11 (5.15–7.44) | 6.74 (6.58–6.96) | 0.91x |
|
||||
| sparse | unrestricted | off | list | 15.79 (13.76–17.04) | 14.55 (12.96–16.51) | 1.09x |
|
||||
| deep | scoped | on | badge | 7.78 (6.82–8.55) | 7.24 (6.72–8.35) | 1.07x |
|
||||
| deep | scoped | on | list | 14.64 (13.10–15.62) | 14.50 (13.24–16.47) | 1.01x |
|
||||
| deep | scoped | off | badge | 7.66 (6.76–8.50) | 7.12 (6.30–7.87) | 1.08x |
|
||||
| deep | scoped | off | list | 16.06 (14.11–18.64) | 16.22 (14.04–17.86) | 0.99x |
|
||||
| fanout | scoped | on | badge | 134.75 (120.71–163.75) | 19.76 (17.04–22.47) | 6.82x |
|
||||
| fanout | scoped | on | list | 140.22 (133.39–149.43) | 18.03 (16.14–32.12) | 7.78x |
|
||||
| fanout | scoped | off | badge | 16.68 (16.33–18.35) | 17.81 (16.99–18.59) | 0.94x |
|
||||
| fanout | scoped | off | list | 19.95 (17.12–22.53) | 21.19 (16.64–57.80) | 0.94x |
|
||||
| cycle | scoped | on | badge | 9.26 (7.75–10.56) | 8.46 (7.82–8.79) | 1.09x |
|
||||
| cycle | scoped | on | list | 15.01 (13.85–16.24) | 15.72 (13.81–16.45) | 0.95x |
|
||||
| cycle | scoped | off | badge | 7.17 (6.87–8.04) | 7.09 (6.61–8.29) | 1.01x |
|
||||
| cycle | scoped | off | list | 14.78 (12.63–18.77) | 16.29 (13.92–17.36) | 0.91x |
|
||||
|
||||
Every small and unrestricted control stays within the original 25% or 2ms allowance. Their overlapping spreads and identical unrestricted SQL make these controls neutral. The dense unrestricted JIT-on list median rose 8.26ms, with wide overlapping 28.71–92.45ms and 29.16–83.33ms ranges; this is not a measured query change. The small scoped JIT-off list median rises 16.18→18.78 ms. Fan-out JIT-off service medians rise about 1.1–1.2 ms; the cycle JIT-off list rises 14.78→16.29 ms. These modest, overlapping controls are neutral/inconclusive. There is no claim that all paths improve.
|
||||
|
||||
In the dense JIT-on ranking plan, recursive task inspections fall 720,000→0 and recursive project probes fall 720,000→0. Estimated statement cost falls 103,000.61→20,458.73. The captured EXPLAIN execution falls 524.04→34.34ms and JIT 110.12ms→none. The dense JIT-off list statement EXPLAIN falls 29.52→1.89ms, while the complete service falls 55.69→30.62ms because the remaining enrichment and application work stays intact. EXPLAIN figures are separate diagnostic executions, not median SQL samples.
|
||||
|
||||
The first lateral-only draft estimated 101 ancestry rows. The final arm estimates 11, matching the one-node-per-generation contract. Both badge phases and all list enrichment statements are retained. This proves no hosted latency, CPU-capacity, or rollout claim. The local serial service savings are 568.22ms for the dense default-JIT badge and 184.89ms for the list. Request latency can recover only the affected serial work on its actual critical path; there is no correlated hosted request proof here.
|
||||
|
||||
Measured production source SHA-256: `a99faf974070a35f2786c92407e023c668c1f7b99b6604f6281c89bce089b1a8`. Baseline and treatment share the same index catalog, database, `issues.ts` hash `702674a98b82579078e9b4516e4a61c2043199fd89c1c2eb24c20d48cb3f8dde`, and participation-helper hash `9260d27bb3653aaaff6052cfc57a5877ed55069ee9cc7906458939899cd26173`.
|
||||
|
||||
## Verification status
|
||||
|
||||
The final-runtime focused PostgreSQL 17 run passed 44/44 tests across the dedicated work/ancestry suite, production privacy review, unread badge, and creation visibility suites in 70.44 seconds. This resolves the earlier skipped creation suite without discarding its 20-second setup failure. The scratch provider replaces only the native PostgreSQL process; normal repository database creation, migrations, clients, SQL and cleanup remain real. It is not part of this change.
|
||||
|
||||
Workspace typecheck and full build passed before replay. The full local
|
||||
`pnpm test:run` attempt completed 940 native executor cases with 16 database
|
||||
bootstrap failures, then was stopped with exit 143. An independent empty native
|
||||
`initdb` failed with `semget` semaphore exhaustion before Paperclip schema or SQL
|
||||
ran. This is a failed/incomplete local full-suite result, not a pass. A focused
|
||||
Docker PostgreSQL retry and exact-head CI results are recorded in the PR.
|
||||
|
||||
## Review follow-up
|
||||
|
||||
The review requested a direct test of project-based ancestry. The added case
|
||||
uses a private child, an open parent in a private project, and a grant on the
|
||||
private grandparent. It checks access, changes the parent project to open and
|
||||
back, and revokes the ancestor grant. The six-test ancestry suite passes.
|
||||
|
||||
The benchmark now lives in `server/scripts/` and imports `sql` from the package
|
||||
name. A small real-PostgreSQL smoke verifies the relocated command. Runtime and
|
||||
query-owner source hashes stay unchanged, so the original 36-case measurements
|
||||
and the retained failed stretch goals still apply.
|
||||
@@ -0,0 +1,162 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { applyPendingMigrations, createDb, startEmbeddedPostgresTestDatabase, type Db } from "../../packages/db/src/index.js";
|
||||
import { sql } from "drizzle-orm";
|
||||
import { issueService } from "../src/services/issues.js";
|
||||
import { issueReadSqlCondition } from "../src/services/authorization.js";
|
||||
import { recursiveIssueWork, seedPrivacyFixture } from "../src/__tests__/helpers/privacy-query-fixture.js";
|
||||
|
||||
const started = performance.now();
|
||||
const root = fileURLToPath(new URL("../../", import.meta.url));
|
||||
const git = (...args: string[]) => execFileSync("git", args, { cwd: root, encoding: "utf8" }).trim();
|
||||
const arg = (name: string) => process.argv.find(value => value.startsWith(`--${name}=`))?.slice(name.length + 3);
|
||||
const maxDenseMs = Number(arg("max-dense-ms") ?? Infinity);
|
||||
const ref = arg("baseline-ref");
|
||||
assert.ok(ref, "Pass a pinned --baseline-ref=<pre-fix commit>");
|
||||
const baselineRevision = git("rev-parse", `${ref}^{commit}`);
|
||||
const modulePath = "server/src/services/authorization.ts";
|
||||
const changedRuntime = git("diff", "--name-only", baselineRevision, "--", "server/src/services", "packages/db/src", "packages/shared/src", "pnpm-lock.yaml").split("\n").filter(Boolean);
|
||||
assert.deepEqual(changedRuntime.filter(file => file !== modulePath), [], "Baseline helpers must be isolated when other runtime dependencies change");
|
||||
const proofFiles = [modulePath, "server/src/services/issues.ts", "server/src/services/issue-participation.ts"];
|
||||
const sourceHashes = Object.fromEntries(await Promise.all(proofFiles.map(async file => [file, createHash("sha256").update(await fs.readFile(path.join(root, file))).digest("hex")])));
|
||||
const baselineDir = await fs.mkdtemp(path.join(root, "server/scripts/.privacy-baseline-"));
|
||||
const oldSource = git("show", `${baselineRevision}:${modulePath}`);
|
||||
const rewritten = oldSource.replace(/from "(\.{1,2}\/[^\"]+)"/g, (_match, specifier: string) => `from ${JSON.stringify(fileURLToPath(new URL(specifier, pathToFileURL(path.join(root, modulePath)))))}`);
|
||||
const oldModule = path.join(baselineDir, "authorization.ts");
|
||||
await fs.writeFile(oldModule, rewritten);
|
||||
const baseline = await import(pathToFileURL(oldModule).href) as typeof import("../src/services/authorization.js");
|
||||
const treatment = process.argv.includes("--treatment-is-baseline") ? baseline.issueReadSqlCondition : issueReadSqlCondition;
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let admin: Db | undefined, db: Db | undefined;
|
||||
const dbName = `privacy_ancestry_${randomUUID().replaceAll("-", "")}`;
|
||||
const keepAlive = setInterval(() => {}, 1000);
|
||||
type Query = { sql: string; params: unknown[] };
|
||||
function captureDb(target: Db, queries: Query[]): Db {
|
||||
function builder(value: any) {
|
||||
const from = value.from.bind(value);
|
||||
value.from = (...args: any[]) => {
|
||||
const query = from(...args), execute = query.execute.bind(query);
|
||||
query.execute = async () => { queries.push(query.toSQL()); return execute(); };
|
||||
return query;
|
||||
};
|
||||
return value;
|
||||
}
|
||||
return new Proxy(target, { get(target, key) {
|
||||
if (key === "select") return (...fields: any[]) => builder((target.select as any)(...fields));
|
||||
if (key === "with") return (...ctes: any[]) => {
|
||||
const value = (target.with as any)(...ctes), select = value.select.bind(value);
|
||||
value.select = (...fields: any[]) => builder(select(...fields));
|
||||
return value;
|
||||
};
|
||||
return Reflect.get(target, key);
|
||||
} });
|
||||
}
|
||||
const stats = (values: number[]) => ({ samplesMs: values, medianMs: [...values].sort((a,b) => a-b)[Math.floor(values.length/2)]!, minMs: Math.min(...values), maxMs: Math.max(...values) });
|
||||
try {
|
||||
if (process.env.PRIVACY_BENCHMARK_ADMIN_URL) {
|
||||
const url = new URL(process.env.PRIVACY_BENCHMARK_ADMIN_URL);
|
||||
assert.equal(url.hostname, "127.0.0.1", "External benchmark PostgreSQL must be disposable loopback PostgreSQL");
|
||||
admin = createDb(url.href, { maxConnections: 1 });
|
||||
await admin.$client.unsafe(`CREATE DATABASE "${dbName}"`);
|
||||
url.pathname = `/${dbName}`;
|
||||
await applyPendingMigrations(url.href);
|
||||
db = createDb(url.href, { maxConnections: 1 });
|
||||
} else {
|
||||
database = await startEmbeddedPostgresTestDatabase("privacy-benchmark-");
|
||||
db = createDb(database.connectionString, { maxConnections: 1 });
|
||||
}
|
||||
await db.$client.unsafe("set client_min_messages=warning");
|
||||
const settings = await db.$client.unsafe("select version(),current_setting('shared_buffers') shared_buffers,current_setting('work_mem') work_mem,current_setting('jit_above_cost') jit_above_cost,current_setting('jit_inline_above_cost') jit_inline_above_cost,current_setting('jit_optimize_above_cost') jit_optimize_above_cost");
|
||||
const setupMs = performance.now()-started;
|
||||
const shapes = [
|
||||
{ name: "small", count: 20, participated: 20, history: 2, comments: 2, payloadBytes: 512 },
|
||||
{ name: "dense", count: 1200, participated: 1200, history: 50, comments: 20, decoyHistory: 240000 },
|
||||
{ name: "sparse", count: 1200, participated: 12, history: 50, comments: 20, decoyHistory: 240000 },
|
||||
{ name: "deep", count: 64, participated: 64, history: 2, comments: 2, graph: "deep" as const },
|
||||
{ name: "fanout", count: 1200, participated: 1200, history: 2, comments: 2, graph: "fanout" as const },
|
||||
{ name: "cycle", count: 32, participated: 32, history: 2, comments: 2, graph: "cycle" as const },
|
||||
].filter(shape => !arg("fixture") || shape.name === arg("fixture"));
|
||||
assert.ok(shapes.length, "Unknown fixture");
|
||||
const results = [];
|
||||
for (const shape of shapes) {
|
||||
const seedStart = performance.now();
|
||||
const fixture = await seedPrivacyFixture(db, shape);
|
||||
const fixtureMs = performance.now()-seedStart;
|
||||
const actor = { type: "board" as const, source: "session", userId: fixture.userId, authorizingUserId: fixture.userId };
|
||||
const cases = [];
|
||||
for (const readMode of ["scoped", "unrestricted"]) {
|
||||
if (shape.graph && readMode === "unrestricted") continue;
|
||||
const conditions = {
|
||||
baseline: readMode === "scoped" ? await baseline.issueReadSqlCondition(db, actor) : sql<boolean>`true`,
|
||||
treatment: readMode === "scoped" ? await treatment(db, actor) : sql<boolean>`true`,
|
||||
};
|
||||
for (const jit of ["on", "off"]) {
|
||||
await db.$client.unsafe(`set jit=${jit}`);
|
||||
for (const caller of ["badge", "list"] as const) {
|
||||
const invoke = (target: Db, arm: "baseline" | "treatment") => caller === "badge"
|
||||
? issueService(target).unreadInboxIssueIds(fixture.companyId, fixture.userId, conditions[arm])
|
||||
: issueService(target).list(fixture.companyId, { readCondition: conditions[arm], touchedByUserId: fixture.userId, limit: 100, sortField: "id", sortDir: "asc" });
|
||||
const samples = { baseline: [] as number[], treatment: [] as number[] };
|
||||
for (let round=0; round<7; round++) {
|
||||
let before: unknown, after: unknown;
|
||||
for (const arm of round%2 ? ["treatment", "baseline"] as const : ["baseline", "treatment"] as const) {
|
||||
const start = performance.now();
|
||||
const result = await invoke(db, arm);
|
||||
if (round >= 2) samples[arm].push(performance.now()-start);
|
||||
if (arm === "baseline") before=result; else after=result;
|
||||
}
|
||||
assert.deepEqual(after, before, `${shape.name}/${readMode}/${jit}/${caller}: full service output/order`);
|
||||
}
|
||||
const plans = [];
|
||||
let expectedRows: unknown[][] | undefined;
|
||||
for (const arm of ["baseline", "treatment"] as const) {
|
||||
const queries: Query[] = [];
|
||||
await invoke(captureDb(db, queries), arm);
|
||||
assert.ok(queries.length >= (caller === "badge" ? 2 : 1), "Capture the complete real caller");
|
||||
const rows: unknown[][] = [], queryPlans = [];
|
||||
for (const query of queries) {
|
||||
rows.push(await db.$client.unsafe(query.sql, query.params as never[]).values());
|
||||
const [explain] = await db.$client.unsafe(`explain (analyze,buffers,format json) ${query.sql}`, query.params as never[]);
|
||||
queryPlans.push({ ...query, rows: rows.at(-1)!.length, work: recursiveIssueWork(explain["QUERY PLAN"][0].Plan), plan: explain["QUERY PLAN"] });
|
||||
}
|
||||
if (arm === "baseline") expectedRows = rows;
|
||||
else assert.deepEqual(rows, expectedRows, `${shape.name}/${readMode}/${jit}/${caller}: every positional SQL field and order`);
|
||||
plans.push({ arm, queries: queryPlans });
|
||||
}
|
||||
const before = stats(samples.baseline), after = stats(samples.treatment);
|
||||
cases.push({ readMode, jit, caller, baseline: before, treatment: after, speedup: before.medianMs/after.medianMs, positionalEquality: true, serviceEquality: true, plans });
|
||||
console.error(JSON.stringify({ fixture: shape.name, readMode, jit, caller, baseline: before.medianMs, treatment: after.medianMs }));
|
||||
}
|
||||
}
|
||||
}
|
||||
results.push({ fixture, fixtureMs, cases });
|
||||
await db.$client.unsafe("truncate companies cascade");
|
||||
}
|
||||
const dependencyVersions = Object.fromEntries(await Promise.all(["postgres", "drizzle-orm"].map(async name => [name, JSON.parse(await fs.readFile(path.join(root,`packages/db/node_modules/${name}/package.json`),"utf8")).version])));
|
||||
const report = { baselineRevision, dependencyVersions, maxDenseMs: Number.isFinite(maxDenseMs) ? maxDenseMs : null, treatmentRevision: git("rev-parse", "HEAD"), sourceHashes, baselineSourceSha256: createHash("sha256").update(oldSource).digest("hex"), changedRuntime, checkedInLockSha256: createHash("sha256").update(await fs.readFile(path.join(root,"pnpm-lock.yaml"))).digest("hex"), nodeVersion: process.version, settings, setupMs, totalMs: performance.now()-started, warmupsPerArm: 2, measuredSamplesPerArm: 5, results };
|
||||
const evidence = arg("evidence-dir");
|
||||
if (evidence) { await fs.mkdir(evidence,{ recursive: true }); await fs.writeFile(path.join(evidence,"benchmark.json"),JSON.stringify(report,null,2)); }
|
||||
console.log(JSON.stringify({ ...report, results: results.map(result => ({ ...result, cases: result.cases.map(({ plans, ...rest }) => rest) })) },null,2));
|
||||
if (process.argv.includes("--assert-performance")) {
|
||||
for (const result of results) for (const c of result.cases) {
|
||||
if (c.readMode === "scoped") {
|
||||
for (const query of c.plans.find(p => p.arm === "treatment")!.queries) {
|
||||
if (!result.fixture.graph) assert.equal(query.work.inspectedRows, 0, "Absent parent frontiers must not inspect company tasks");
|
||||
}
|
||||
if (["dense","sparse"].includes(result.fixture.name)) assert.ok(c.speedup >= 2, `Scoped representative ${result.fixture.name}/${c.jit}/${c.caller} requires 2x improvement`);
|
||||
}
|
||||
if (result.fixture.name === "dense" && c.caller === "badge" && c.readMode === "scoped" && c.jit === "on") assert.ok(c.treatment.medianMs <= maxDenseMs, `Original dense scoped badge target ${maxDenseMs}ms remains unmet: ${c.treatment.medianMs.toFixed(2)}ms`);
|
||||
if (result.fixture.name === "small" || c.readMode === "unrestricted") assert.ok(c.treatment.medianMs <= Math.max(c.baseline.medianMs*1.25,c.baseline.medianMs+2), "Small/open controls must stay within 25% or 2ms");
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
await db?.$client.end({timeout:0});
|
||||
await database?.cleanup();
|
||||
if (admin) { await admin.$client.unsafe(`DROP DATABASE IF EXISTS "${dbName}" WITH (FORCE)`); await admin.$client.end({timeout:0}); }
|
||||
await fs.rm(baselineDir,{recursive:true,force:true});
|
||||
clearInterval(keepAlive);
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq, inArray, sql } from "drizzle-orm";
|
||||
import { agents, companies, companyMemberships, createDb, issueAccessGrants, issues, projectAccessMembers, projects, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { canActorReadIssuePrivacy, issueReadSqlCondition, type AuthorizationActor } from "../services/authorization.js";
|
||||
import { issueService } from "../services/issues.js";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { recursiveIssueWork, seedPrivacyFixture } from "./helpers/privacy-query-fixture.js";
|
||||
|
||||
describe("privacy ancestry query work", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>, db: ReturnType<typeof createDb>;
|
||||
beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("privacy-query-"); db = createDb(database.connectionString); }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
||||
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
|
||||
|
||||
it("does not rescan company tasks when resolving absent privacy parents", async () => {
|
||||
const f = await seedPrivacyFixture(db, { name: "open roots", count: 1200, participated: 0, history: 0, comments: 0 });
|
||||
const condition = await issueReadSqlCondition(db, { type: "board", userId: f.userId, authorizingUserId: f.userId });
|
||||
const query = db.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, f.companyId), condition)).orderBy(issues.id);
|
||||
expect((await query).map(row => row.id)).toEqual([...f.ids].sort());
|
||||
const statement = query.toSQL();
|
||||
await db.execute(sql`set jit=off`);
|
||||
const [explain] = await db.$client.unsafe(`explain (analyze,buffers,format json) ${statement.sql}`, statement.params as never[]);
|
||||
const work = recursiveIssueWork(explain["QUERY PLAN"][0].Plan);
|
||||
expect(work.recursiveNodes).toBeGreaterThan(0);
|
||||
expect(work.inspectedRows).toBeLessThanOrEqual(f.ids.length * 3);
|
||||
expect(work.projectProbes).toBeLessThanOrEqual(f.ids.length * 3);
|
||||
});
|
||||
it("retains grant direction, private-only traversal, cycles and tenant boundaries", async () => {
|
||||
const companyId = randomUUID(), otherCompany = randomUUID(), userId = randomUUID();
|
||||
await db.insert(companies).values([companyId, otherCompany].map(id => ({ id, name: "Privacy graph", issuePrefix: `P${id.slice(0,7)}` })));
|
||||
const ids = Object.fromEntries(["root", "child", "grandchild", "sibling", "barrier", "behindBarrier", "cycleA", "cycleB", "foreign", "foreignChild"].map(name => [name, randomUUID()]));
|
||||
await db.insert(issues).values(Object.entries(ids).map(([name,id]) => ({ id, companyId: name === "foreign" ? otherCompany : companyId, title: name, visibility: name === "barrier" ? "open" : "private", createdByUserId: "other-user" })));
|
||||
const parents = { child: "root", grandchild: "child", sibling: "root", barrier: "root", behindBarrier: "barrier", cycleA: "cycleB", cycleB: "cycleA", foreignChild: "foreign" };
|
||||
for (const [child,parent] of Object.entries(parents)) await db.update(issues).set({ privacyParentIssueId: ids[parent]! }).where(eq(issues.id,ids[child]!));
|
||||
const privacyRows = new Map((await db.select().from(issues).where(eq(issues.companyId, companyId))).map(row => [row.id, row]));
|
||||
const actor: AuthorizationActor = { type: "board", userId, authorizingUserId: userId };
|
||||
const [grant] = await db.insert(issueAccessGrants).values({ issueId: ids.child!, subjectType: "user", subjectId: userId, source: "explicit" }).returning();
|
||||
const condition = await issueReadSqlCondition(db, actor);
|
||||
const read = async () => (await db.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId,companyId),condition))).map(row=>row.id).sort();
|
||||
expect(await read()).toEqual([ids.child!,ids.grandchild!].sort());
|
||||
await db.update(issueAccessGrants).set({ revokedAt:new Date() }).where(eq(issueAccessGrants.id,grant!.id));
|
||||
expect(await read()).toEqual([]);
|
||||
await db.insert(issueAccessGrants).values([ids.root!,ids.cycleB!,ids.foreign!].map(issueId=>({ issueId, subjectType:"user", subjectId:userId, source:"explicit" })));
|
||||
expect(await read()).toEqual([ids.root!,ids.child!,ids.grandchild!,ids.sibling!,ids.barrier!,ids.cycleA!,ids.cycleB!].sort());
|
||||
for (const [name,expected] of [["cycleA",true],["cycleB",true],["behindBarrier",false],["foreignChild",false]] as const) {
|
||||
expect(await canActorReadIssuePrivacy(db,actor,privacyRows.get(ids[name]!)!)).toBe(expected);
|
||||
}
|
||||
await db.update(issueAccessGrants).set({ revokedAt:new Date() }).where(eq(issueAccessGrants.issueId,ids.cycleB!));
|
||||
expect(await canActorReadIssuePrivacy(db,actor,privacyRows.get(ids.cycleA!)!)).toBe(false);
|
||||
});
|
||||
|
||||
it("intersects current agent, responsible-user and authorizing-user grants through ancestors", async () => {
|
||||
const companyId=randomUUID(), owner=randomUUID(), authorizer=randomUUID();
|
||||
await db.insert(companies).values({id:companyId,name:"Agent privacy",issuePrefix:`P${companyId.slice(0,7)}`});
|
||||
await db.insert(companyMemberships).values([owner,authorizer].map(principalId=>({companyId,principalType:"user",principalId,status:"active",membershipRole:"operator"})));
|
||||
const [agent]=await db.insert(agents).values({companyId,name:"Shared agent",adapterType:"process"}).returning();
|
||||
const [parent]=await db.insert(issues).values({companyId,title:"Ancestor",visibility:"private",createdByUserId:"someone-else"}).returning();
|
||||
const [child]=await db.insert(issues).values({companyId,title:"Descendant",visibility:"private",privacyParentIssueId:parent!.id,createdByUserId:"someone-else"}).returning();
|
||||
const actor:AuthorizationActor={type:"agent",agentId:agent!.id,onBehalfOfUserId:owner,authorizingUserId:authorizer};
|
||||
const readable=()=>canActorReadIssuePrivacy(db,actor,child!);
|
||||
await db.insert(issueAccessGrants).values({issueId:parent!.id,subjectType:"agent",subjectId:agent!.id,source:"explicit"});
|
||||
expect(await readable()).toBe(false);
|
||||
await db.insert(issueAccessGrants).values({issueId:parent!.id,subjectType:"user",subjectId:owner,source:"explicit"});
|
||||
expect(await readable()).toBe(false);
|
||||
const [grant]=await db.insert(issueAccessGrants).values({issueId:parent!.id,subjectType:"user",subjectId:authorizer,source:"explicit"}).returning();
|
||||
expect(await readable()).toBe(true);
|
||||
const condition=await issueReadSqlCondition(db,actor);
|
||||
expect((await db.select({id:issues.id}).from(issues).where(and(eq(issues.id,child!.id),condition))).length).toBe(1);
|
||||
await db.update(issueAccessGrants).set({revokedAt:new Date()}).where(eq(issueAccessGrants.id,grant!.id));
|
||||
expect(await readable()).toBe(false);
|
||||
expect(await db.select({id:issues.id}).from(issues).where(and(eq(issues.id,child!.id),condition))).toEqual([]);
|
||||
await db.update(issueAccessGrants).set({revokedAt:null}).where(eq(issueAccessGrants.id,grant!.id));
|
||||
await db.update(companyMemberships).set({status:"archived"}).where(and(eq(companyMemberships.companyId,companyId),eq(companyMemberships.principalId,owner)));
|
||||
expect(await readable()).toBe(false);
|
||||
});
|
||||
|
||||
it("applies fresh inherited project membership before list pagination", async () => {
|
||||
const f=await seedPrivacyFixture(db,{name:"paged",count:130,participated:130,history:1,comments:1});
|
||||
await db.update(issues).set({createdByUserId:"other-human",visibility:"private"}).where(eq(issues.companyId,f.companyId));
|
||||
const [project]=await db.insert(projects).values({companyId:f.companyId,name:"Private project",visibility:"private"}).returning();
|
||||
const ordered=[...f.ids].sort(), allowed=ordered.slice(100);
|
||||
await db.update(issues).set({projectId:project!.id}).where(inArray(issues.id,allowed));
|
||||
const actor:AuthorizationActor={type:"board",userId:f.userId,authorizingUserId:f.userId};
|
||||
const condition=await issueReadSqlCondition(db,actor), svc=issueService(db);
|
||||
const read=()=>svc.list(f.companyId,{readCondition:condition,limit:20,sortField:"id",sortDir:"asc",touchedByUserId:f.userId});
|
||||
expect(await read()).toEqual([]);
|
||||
const [member]=await db.insert(projectAccessMembers).values({companyId:f.companyId,projectId:project!.id,subjectType:"user",subjectId:f.userId}).returning();
|
||||
expect((await read()).map(row=>row.id)).toEqual(allowed.slice(0,20));
|
||||
expect((await svc.unreadInboxIssueIds(f.companyId,f.userId,condition)).sort()).toEqual(allowed);
|
||||
await db.delete(projectAccessMembers).where(eq(projectAccessMembers.id,member!.id));
|
||||
expect(await read()).toEqual([]);
|
||||
expect(await svc.unreadInboxIssueIds(f.companyId,f.userId,condition)).toEqual([]);
|
||||
});
|
||||
|
||||
it("traverses a complete deep private chain with a root-only grant and revokes it fresh", async () => {
|
||||
const f=await seedPrivacyFixture(db,{name:"deep",count:64,participated:0,history:0,comments:0,graph:"deep"});
|
||||
const actor:AuthorizationActor={type:"board",userId:f.userId,authorizingUserId:f.userId};
|
||||
const condition=await issueReadSqlCondition(db,actor);
|
||||
const query=()=>db.select({id:issues.id}).from(issues).where(and(eq(issues.companyId,f.companyId),condition)).orderBy(issues.id);
|
||||
expect((await query()).map(row=>row.id)).toEqual([...f.ids].sort());
|
||||
await db.update(issueAccessGrants).set({revokedAt:new Date()}).where(eq(issueAccessGrants.issueId,f.ids[0]!));
|
||||
expect(await query()).toEqual([]);
|
||||
});
|
||||
|
||||
it("follows an open ancestor in a private project and rechecks its restriction and inherited grant", async () => {
|
||||
const companyId = randomUUID(), userId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Project ancestor", issuePrefix: `P${companyId.slice(0, 7)}` });
|
||||
const [project] = await db.insert(projects).values({ companyId, name: "Private ancestor project", visibility: "private" }).returning();
|
||||
const [grandparent] = await db.insert(issues).values({ companyId, title: "Granted grandparent", visibility: "private", createdByUserId: "other-user" }).returning();
|
||||
const [parent] = await db.insert(issues).values({ companyId, title: "Open parent in private project", visibility: "open", projectId: project!.id, privacyParentIssueId: grandparent!.id, createdByUserId: "other-user" }).returning();
|
||||
const [child] = await db.insert(issues).values({ companyId, title: "Private child", visibility: "private", privacyParentIssueId: parent!.id, createdByUserId: "other-user" }).returning();
|
||||
const actor: AuthorizationActor = { type: "board", userId, authorizingUserId: userId };
|
||||
const condition = await issueReadSqlCondition(db, actor);
|
||||
const read = async () => (await db.select({ id: issues.id }).from(issues).where(and(eq(issues.id, child!.id), condition))).map(row => row.id);
|
||||
expect(await canActorReadIssuePrivacy(db, actor, child!)).toBe(false);
|
||||
const [grant] = await db.insert(issueAccessGrants).values({ issueId: grandparent!.id, subjectType: "user", subjectId: userId, source: "explicit" }).returning();
|
||||
expect(await read()).toEqual([child!.id]);
|
||||
expect(await canActorReadIssuePrivacy(db, actor, child!)).toBe(true);
|
||||
await db.update(projects).set({ visibility: "open" }).where(eq(projects.id, project!.id));
|
||||
expect(await read()).toEqual([]);
|
||||
await db.update(projects).set({ visibility: "private" }).where(eq(projects.id, project!.id));
|
||||
expect(await read()).toEqual([child!.id]);
|
||||
await db.update(issueAccessGrants).set({ revokedAt: new Date() }).where(eq(issueAccessGrants.id, grant!.id));
|
||||
expect(await read()).toEqual([]);
|
||||
expect(await canActorReadIssuePrivacy(db, actor, child!)).toBe(false);
|
||||
});
|
||||
|
||||
});
|
||||
@@ -0,0 +1,50 @@
|
||||
import { randomBytes, randomUUID } from "node:crypto";
|
||||
import { companies, type Db } from "@paperclipai/db";
|
||||
|
||||
export type PrivacyPlan = Record<string, any>;
|
||||
export function recursiveIssueWork(plan: PrivacyPlan) {
|
||||
let recursiveNodes = 0, inspectedRows = 0, projectProbes = 0;
|
||||
function walk(node: PrivacyPlan, recursive = false) {
|
||||
if (node["Node Type"] === "Recursive Union") {
|
||||
recursiveNodes++;
|
||||
for (const child of (node.Plans ?? []).slice(1)) walk(child, true);
|
||||
return;
|
||||
}
|
||||
const visits = (Number(node["Actual Rows"] ?? 0) + Number(node["Rows Removed by Filter"] ?? 0)) * Number(node["Actual Loops"] ?? 0);
|
||||
if (recursive && node["Relation Name"] === "issues") inspectedRows += visits;
|
||||
if (recursive && node["Relation Name"] === "projects") projectProbes += Number(node["Actual Loops"] ?? 0);
|
||||
for (const child of node.Plans ?? []) walk(child, recursive);
|
||||
}
|
||||
walk(plan);
|
||||
return { recursiveNodes, inspectedRows, projectProbes };
|
||||
}
|
||||
|
||||
export async function seedPrivacyFixture(db: Db, input: { name: string; count: number; participated: number; history: number; comments: number; payloadBytes?: number; decoyHistory?: number; graph?: "deep" | "fanout" | "cycle" }) {
|
||||
const companyId = randomUUID(), userId = `privacy-${randomUUID()}`;
|
||||
const ids = Array.from({ length: input.count }, () => randomUUID());
|
||||
const payload = randomBytes(Math.ceil((input.payloadBytes ?? 4096) * 3 / 4)).toString("base64");
|
||||
await db.insert(companies).values({ id: companyId, name: `Synthetic ${input.name}`, issuePrefix: `P${companyId.slice(0, 7)}` });
|
||||
await db.$client.unsafe(`insert into issues(id,company_id,title,status,created_by_user_id,created_at,updated_at,visibility)
|
||||
select id,$2::uuid,'Synthetic '||n,'todo',case when n <= $4 and n%2=0 then $3 else 'other-human' end,
|
||||
'2026-01-01'::timestamptz,'2026-01-01'::timestamptz+n*interval '1 second',$5
|
||||
from unnest($1::uuid[]) with ordinality as f(id,n)`, [ids, companyId, userId, input.participated, input.graph ? "private" : "open"]);
|
||||
if (input.graph) {
|
||||
const parents = ids.map((_, index) => input.graph === "deep" ? (index ? ids[index - 1]! : null) : input.graph === "cycle" ? ids[(index + 1) % ids.length]! : index ? ids[0]! : null);
|
||||
await db.$client.unsafe(`update issues i set privacy_parent_issue_id=f.parent from unnest($1::uuid[],$2::uuid[]) f(id,parent) where i.id=f.id`, [ids, parents]);
|
||||
await db.$client.unsafe(`insert into issue_access_grants(issue_id,subject_type,subject_id,source) values($1,'user',$2,'explicit')`, [ids[0]!, userId]);
|
||||
}
|
||||
if (input.history) await db.$client.unsafe(`insert into activity_log(company_id,entity_type,entity_id,actor_type,actor_id,action,details,created_at)
|
||||
select $1::uuid,'issue',i.id::text,case when n=1 or n%10=0 then 'user' else 'agent' end,
|
||||
case when (n=1 or n%20=0) and i.id=ANY($5::uuid[]) then $4 when n=1 or n%10=0 then 'other-human' else 'synthetic-worker' end,
|
||||
'issue.updated',jsonb_build_object('padding',$2::text),'2026-01-01'::timestamptz+n*interval '1 second'
|
||||
from issues i cross join generate_series(1,$3::int)n where company_id=$1::uuid order by n,i.id`, [companyId, payload, input.history, userId, ids.slice(0, input.participated)]);
|
||||
if (input.comments) await db.$client.unsafe(`insert into issue_comments(company_id,issue_id,body,created_at)
|
||||
select $1::uuid,id,$2,'2026-01-02'::timestamptz+n*interval '1 second'
|
||||
from issues cross join generate_series(1,$3::int)n where company_id=$1::uuid order by n,id`, [companyId, payload, input.comments]);
|
||||
if (input.decoyHistory) await db.$client.unsafe(`insert into activity_log(company_id,entity_type,entity_id,actor_type,actor_id,action,details,created_at)
|
||||
select $1::uuid,'heartbeat_run',gen_random_uuid()::text,'agent','synthetic-worker','heartbeat.run.created',
|
||||
jsonb_build_object('padding',left($2,256)),'2026-01-01'::timestamptz+n*interval '1 second'
|
||||
from generate_series(1,$3::int)n`, [companyId, payload, input.decoyHistory]);
|
||||
await db.$client.unsafe("ANALYZE");
|
||||
return { ...input, companyId, userId, ids, payloadBytes: Buffer.byteLength(payload), activityRows: input.count * input.history, commentRows: input.count * input.comments };
|
||||
}
|
||||
@@ -629,10 +629,18 @@ function issuePrivacyPrincipalCondition(actor: AuthorizationActor): SQL<boolean>
|
||||
p.responsible_user_id, p.created_by_user_id, p.assignee_user_id, p.assignee_agent_id
|
||||
from issues p where p.id = ${issues.id} and p.company_id = ${issues.companyId}
|
||||
union
|
||||
select p.id, p.company_id, p.privacy_parent_issue_id, p.visibility, p.project_id,
|
||||
(select p.id, p.company_id, p.privacy_parent_issue_id, p.visibility, p.project_id,
|
||||
p.responsible_user_id, p.created_by_user_id, p.assignee_user_id, p.assignee_agent_id
|
||||
from issues p join privacy_ancestors c on p.id = c.privacy_parent_issue_id and p.company_id = c.company_id
|
||||
where p.visibility = 'private' or exists (select 1 from projects pp where pp.id = p.project_id and pp.company_id = p.company_id and pp.visibility = 'private')
|
||||
from privacy_ancestors c
|
||||
cross join lateral (
|
||||
select p.id, p.company_id, p.privacy_parent_issue_id, p.visibility, p.project_id,
|
||||
p.responsible_user_id, p.created_by_user_id, p.assignee_user_id, p.assignee_agent_id
|
||||
from issues p where p.id = c.privacy_parent_issue_id and p.company_id = c.company_id
|
||||
limit 1
|
||||
) p
|
||||
where c.privacy_parent_issue_id is not null
|
||||
and (p.visibility = 'private' or exists (select 1 from projects pp where pp.id = p.project_id and pp.company_id = p.company_id and pp.visibility = 'private'))
|
||||
limit 1)
|
||||
)
|
||||
select 1 from privacy_ancestors n
|
||||
where ${implicit} or ${grant} or ${projectMember}
|
||||
|
||||
Reference in new issue
Block a user