diff --git a/doc/plans/2026-10-28-task-privacy-query.md b/doc/plans/2026-10-28-task-privacy-query.md new file mode 100644 index 00000000000..4072c028e08 --- /dev/null +++ b/doc/plans/2026-10-28-task-privacy-query.md @@ -0,0 +1,198 @@ +# Task privacy ancestry query proof + +The shared task privacy predicate belongs to `authorization.ts`. Detail reads, +list/search/count filters, unread badge phases, and derived authorization all use +this predicate. This change preserves that owner and the current access rules. + +## Cause and change + +Measured baseline commit: `8279c990080b15abdcd9981f5793290fab37f9ad`. +The branch was later replayed onto `71b448b9cb841e6fa18fe883585430c5081245da`. +Both commits have byte-identical pre-fix authorization, issue-service and +participation-helper sources. The reproduction command uses the newer equivalent +pre-fix baseline so unrelated upstream voice-schema changes stay constant in +both arms. The measured source hashes below are unchanged by that replay. +The previous recursive join can scan every company task and check its project +for each privacy frontier. In the 1,200-open-task regression this reads 1,440,000 +task rows even though every parent is NULL. The semantic ID assertion passes; +the deterministic work assertion fails at the unchanged baseline. + +The replacement skips NULL parent frontiers and reads the exact same-company +parent through a correlated lateral lookup. The existing unique task ID and +company/ID index support the lookup. No schema or migration change is needed. + +Both `LIMIT 1` clauses are redundant under the current schema and anchor: + +1. The anchor selects at most one row by the outer task's ID and company. +2. Each task has one scalar privacy parent ID. That ID selects at most one + same-company parent. The lateral limit prevents the planner from replacing + the parameterized lookup with a company-wide hash scan. +3. Inductively, each recursive generation has at most one row. The recursive-arm + limit exposes this fact to the planner. It does not limit chain depth. +4. `UNION` still removes visited rows and terminates cycles. An open parent + outside a private project still stops traversal at the same place. + +A first lateral-only draft removed repeated scans but raised estimated ancestry +rows from 11 to 101. This retained JIT compilation in the representative callers. +The per-generation cardinality fence addresses that estimate. Tiny tables can +still use a cheap sequential lookup for a real parent. The work guard prevents +company scans for absent parents; it does not force every small-table index plan. + +Implicit principals, live task grants, private project membership, responsible +user and external authorizer intersections, and company membership checks use +the same expressions. There is no cached permission result or visibility check +moved after pagination. + +## Local proof and reproduction + +Run the real service benchmark with a pinned source baseline: + +```sh +pnpm --filter @paperclipai/server exec tsx scripts/benchmark-task-privacy.ts \ + --baseline-ref=71b448b9cb841e6fa18fe883585430c5081245da \ + --evidence-dir=/tmp/task-privacy-proof +``` + +It uses disposable embedded PostgreSQL by default. An explicit +`PRIVACY_BENCHMARK_ADMIN_URL` selects an isolated loopback PostgreSQL server. +The harness creates and removes only its own `privacy_ancestry_*` database. + +Adding `--assert-performance --max-dense-ms=60` reproduces the retained failed +stretch-goal command. It must not be used as evidence of a green performance gate. + +The harness loads old `authorization.ts` from Git, rewrites only its relative +imports, and uses the unchanged real `issueService` for both arms. It rejects +unisolated runtime dependency changes. Both arms use one identical database, +index set, user, task IDs, clock inputs and session settings. It records source +hashes, baseline revision, PostgreSQL and dependency versions, settings, and +fixture sizes. + +Ordinary service timings exclude SQL capture, assertions, and EXPLAIN. Each case +has two warmups and five measured calls per arm in alternating order. Capture +runs afterwards. Every positional SQL field and row order must match across all +captured queries. The entire final badge output and list objects must also match. +The badge runs both ranking and unread phases. The list includes its ordinary +post-query enrichment. These are local service measurements, not HTTP latency. + +The matrix includes small, dense and sparse participation with JIT on/off, plus +64-level private ancestry, 1,200-task private fan-out, and a 32-task cycle. Dense +and sparse fixtures contain 1,200 tasks, 60,000 task activity rows, 240,000 unrelated +run activity rows, and 24,000 comments. Their activity/comment payloads use 4 KiB +Base64 text from 3 KiB of random bytes per value. Half of personal participation comes from the creator and +half from user activity. The dense fixture has 1,200 personal tasks; sparse has +12. Controls also omit the privacy predicate entirely. + +Acceptance was set before the final run: no recursive task reads for absent +parents, at least 2x gain on representative scoped dense/sparse callers, complete +output equality, and small/open controls within 25% or 2 ms. The original 60 ms +dense scoped badge target is a separate explicit check. It is not treated as +passed without the final measured result. + +## Correctness and failures retained + +The dedicated real PostgreSQL tests cover NULL parents, child-to-descendant grant +flow, sibling/ancestor denial, revocation, a 64-level root-only grant, cycles, +public traversal barriers, foreign-company parent pointers, all three agent/user +principals, inactive user membership, private project membership, and access +checks before list pagination. Existing production privacy, creation visibility, +and unread badge tests exercise the shared consumers and HTTP routes. + +The first baseline work test failed as intended: 1,440,000 rows against the 3,600 +bound, after semantic equality passed. A wrong relative-path edit then reran the +old query; that failed command is retained and is not a treatment pass. + +The first four-suite lateral-only run passed 35 tests, but eight creation tests +did not run because their real database setup exceeded its 20-second hook limit. +That command failed. Final-runtime checks and the normal CI result are recorded +separately below. Exploratory service times collected during other builds are +retained as diagnostics and are not the published final benchmark. + +## Feedback loop + +The first deterministic red command took 24.31 seconds including PostgreSQL +setup and the plan assertion. The baseline dense caller capture took 89.81 +seconds including 11.96 seconds of database setup. The first lateral-only dense +comparison took 93.37 seconds including 12.22 seconds of setup. The small +cardinality smoke kept the next query experiment narrow. A reserved quiet slot +is required for the final published measurements; other agents pause builds, +tests and seeds during that slot. + +## Final measured result + +The full matrix completed in 107.20 seconds, including 8.22 seconds of database setup. Other agents paused builds, tests and seeds. PostgreSQL was 17.11 on ARM64 Linux with 128MB shared buffers, 4MB work memory, default 100,000 JIT threshold and 500,000 full-optimization thresholds. Node was v26.4.0, postgres.js 3.4.9, and Drizzle 0.45.2. + +All 36 cells passed complete positional SQL and final service output/order comparison. All absent-parent recursive arms read zero task rows. The deterministic work regression is green. + +**The combined strict benchmark command exited 1.** The 60ms dense scoped badge goal remains unmet at 77.11ms. Four full-service cells also miss the predeclared2x goal: dense JIT-off list 1.82x, sparse JIT-on/off badges 1.60x/1.71x, and sparse JIT-off list 1.10x. These criteria remain in the harness and are not reported as passed. The deliverable is the independently proven query-work optimization with preserved results. + +Warm ordinary full-service timings below are milliseconds, median(min–max), with five measured calls per arm after two warmups. The list page limit is 100. + +| Fixture | Scope | JIT | Caller | Baseline | Treatment | Ratio | +| --- | --- | --- | --- | --- | --- | --- | +| small | scoped | on | badge | 6.54 (5.27–7.60) | 5.74 (5.53–7.01) | 1.14x | +| small | scoped | on | list | 13.24 (11.67–15.46) | 12.06 (10.57–14.84) | 1.10x | +| small | scoped | off | badge | 10.02 (8.09–11.62) | 9.26 (6.91–11.89) | 1.08x | +| small | scoped | off | list | 16.18 (14.36–17.22) | 18.78 (13.23–22.03) | 0.86x | +| small | unrestricted | on | badge | 6.53 (5.51–7.56) | 6.93 (5.96–7.90) | 0.94x | +| small | unrestricted | on | list | 16.59 (15.28–17.60) | 19.02 (14.90–20.94) | 0.87x | +| small | unrestricted | off | badge | 5.01 (4.99–6.70) | 6.49 (5.20–7.38) | 0.77x | +| small | unrestricted | off | list | 12.43 (10.19–17.69) | 11.54 (10.66–12.70) | 1.08x | +| dense | scoped | on | badge | 645.33 (607.50–1184.54) | 77.11 (54.52–121.13) | 8.37x | +| dense | scoped | on | list | 214.77 (186.22–264.03) | 29.87 (24.21–33.64) | 7.19x | +| dense | scoped | off | badge | 427.80 (323.86–452.47) | 59.91 (55.82–75.45) | 7.14x | +| dense | scoped | off | list | 55.69 (54.20–66.11) | 30.62 (29.64–37.22) | 1.82x | +| dense | unrestricted | on | badge | 41.05 (34.52–42.89) | 40.30 (36.24–42.94) | 1.02x | +| dense | unrestricted | on | list | 34.99 (28.71–92.45) | 43.25 (29.16–83.33) | 0.81x | +| dense | unrestricted | off | badge | 39.27 (37.17–50.13) | 42.41 (35.10–43.88) | 0.93x | +| dense | unrestricted | off | list | 30.33 (28.02–55.84) | 28.33 (27.35–44.69) | 1.07x | +| sparse | scoped | on | badge | 10.98 (10.11–11.67) | 6.84 (6.08–8.13) | 1.60x | +| sparse | scoped | on | list | 117.92 (114.43–127.75) | 13.22 (12.97–18.58) | 8.92x | +| sparse | scoped | off | badge | 10.77 (10.59–10.91) | 6.30 (5.76–6.51) | 1.71x | +| sparse | scoped | off | list | 16.61 (15.68–16.97) | 15.12 (12.08–16.08) | 1.10x | +| sparse | unrestricted | on | badge | 5.57 (5.12–7.80) | 6.81 (5.40–8.29) | 0.82x | +| sparse | unrestricted | on | list | 12.70 (11.74–15.09) | 13.19 (11.27–14.80) | 0.96x | +| sparse | unrestricted | off | badge | 6.11 (5.15–7.44) | 6.74 (6.58–6.96) | 0.91x | +| sparse | unrestricted | off | list | 15.79 (13.76–17.04) | 14.55 (12.96–16.51) | 1.09x | +| deep | scoped | on | badge | 7.78 (6.82–8.55) | 7.24 (6.72–8.35) | 1.07x | +| deep | scoped | on | list | 14.64 (13.10–15.62) | 14.50 (13.24–16.47) | 1.01x | +| deep | scoped | off | badge | 7.66 (6.76–8.50) | 7.12 (6.30–7.87) | 1.08x | +| deep | scoped | off | list | 16.06 (14.11–18.64) | 16.22 (14.04–17.86) | 0.99x | +| fanout | scoped | on | badge | 134.75 (120.71–163.75) | 19.76 (17.04–22.47) | 6.82x | +| fanout | scoped | on | list | 140.22 (133.39–149.43) | 18.03 (16.14–32.12) | 7.78x | +| fanout | scoped | off | badge | 16.68 (16.33–18.35) | 17.81 (16.99–18.59) | 0.94x | +| fanout | scoped | off | list | 19.95 (17.12–22.53) | 21.19 (16.64–57.80) | 0.94x | +| cycle | scoped | on | badge | 9.26 (7.75–10.56) | 8.46 (7.82–8.79) | 1.09x | +| cycle | scoped | on | list | 15.01 (13.85–16.24) | 15.72 (13.81–16.45) | 0.95x | +| cycle | scoped | off | badge | 7.17 (6.87–8.04) | 7.09 (6.61–8.29) | 1.01x | +| cycle | scoped | off | list | 14.78 (12.63–18.77) | 16.29 (13.92–17.36) | 0.91x | + +Every small and unrestricted control stays within the original 25% or 2ms allowance. Their overlapping spreads and identical unrestricted SQL make these controls neutral. The dense unrestricted JIT-on list median rose 8.26ms, with wide overlapping 28.71–92.45ms and 29.16–83.33ms ranges; this is not a measured query change. The small scoped JIT-off list median rises 16.18→18.78 ms. Fan-out JIT-off service medians rise about 1.1–1.2 ms; the cycle JIT-off list rises 14.78→16.29 ms. These modest, overlapping controls are neutral/inconclusive. There is no claim that all paths improve. + +In the dense JIT-on ranking plan, recursive task inspections fall 720,000→0 and recursive project probes fall 720,000→0. Estimated statement cost falls 103,000.61→20,458.73. The captured EXPLAIN execution falls 524.04→34.34ms and JIT 110.12ms→none. The dense JIT-off list statement EXPLAIN falls 29.52→1.89ms, while the complete service falls 55.69→30.62ms because the remaining enrichment and application work stays intact. EXPLAIN figures are separate diagnostic executions, not median SQL samples. + +The first lateral-only draft estimated 101 ancestry rows. The final arm estimates 11, matching the one-node-per-generation contract. Both badge phases and all list enrichment statements are retained. This proves no hosted latency, CPU-capacity, or rollout claim. The local serial service savings are 568.22ms for the dense default-JIT badge and 184.89ms for the list. Request latency can recover only the affected serial work on its actual critical path; there is no correlated hosted request proof here. + +Measured production source SHA-256: `a99faf974070a35f2786c92407e023c668c1f7b99b6604f6281c89bce089b1a8`. Baseline and treatment share the same index catalog, database, `issues.ts` hash `702674a98b82579078e9b4516e4a61c2043199fd89c1c2eb24c20d48cb3f8dde`, and participation-helper hash `9260d27bb3653aaaff6052cfc57a5877ed55069ee9cc7906458939899cd26173`. + +## Verification status + +The final-runtime focused PostgreSQL 17 run passed 44/44 tests across the dedicated work/ancestry suite, production privacy review, unread badge, and creation visibility suites in 70.44 seconds. This resolves the earlier skipped creation suite without discarding its 20-second setup failure. The scratch provider replaces only the native PostgreSQL process; normal repository database creation, migrations, clients, SQL and cleanup remain real. It is not part of this change. + +Workspace typecheck and full build passed before replay. The full local +`pnpm test:run` attempt completed 940 native executor cases with 16 database +bootstrap failures, then was stopped with exit 143. An independent empty native +`initdb` failed with `semget` semaphore exhaustion before Paperclip schema or SQL +ran. This is a failed/incomplete local full-suite result, not a pass. A focused +Docker PostgreSQL retry and exact-head CI results are recorded in the PR. + +## Review follow-up + +The review requested a direct test of project-based ancestry. The added case +uses a private child, an open parent in a private project, and a grant on the +private grandparent. It checks access, changes the parent project to open and +back, and revokes the ancestor grant. The six-test ancestry suite passes. + +The benchmark now lives in `server/scripts/` and imports `sql` from the package +name. A small real-PostgreSQL smoke verifies the relocated command. Runtime and +query-owner source hashes stay unchanged, so the original 36-case measurements +and the retained failed stretch goals still apply. diff --git a/server/scripts/benchmark-task-privacy.ts b/server/scripts/benchmark-task-privacy.ts new file mode 100644 index 00000000000..5bf6712d335 --- /dev/null +++ b/server/scripts/benchmark-task-privacy.ts @@ -0,0 +1,162 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { createHash, randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { applyPendingMigrations, createDb, startEmbeddedPostgresTestDatabase, type Db } from "../../packages/db/src/index.js"; +import { sql } from "drizzle-orm"; +import { issueService } from "../src/services/issues.js"; +import { issueReadSqlCondition } from "../src/services/authorization.js"; +import { recursiveIssueWork, seedPrivacyFixture } from "../src/__tests__/helpers/privacy-query-fixture.js"; + +const started = performance.now(); +const root = fileURLToPath(new URL("../../", import.meta.url)); +const git = (...args: string[]) => execFileSync("git", args, { cwd: root, encoding: "utf8" }).trim(); +const arg = (name: string) => process.argv.find(value => value.startsWith(`--${name}=`))?.slice(name.length + 3); +const maxDenseMs = Number(arg("max-dense-ms") ?? Infinity); +const ref = arg("baseline-ref"); +assert.ok(ref, "Pass a pinned --baseline-ref="); +const baselineRevision = git("rev-parse", `${ref}^{commit}`); +const modulePath = "server/src/services/authorization.ts"; +const changedRuntime = git("diff", "--name-only", baselineRevision, "--", "server/src/services", "packages/db/src", "packages/shared/src", "pnpm-lock.yaml").split("\n").filter(Boolean); +assert.deepEqual(changedRuntime.filter(file => file !== modulePath), [], "Baseline helpers must be isolated when other runtime dependencies change"); +const proofFiles = [modulePath, "server/src/services/issues.ts", "server/src/services/issue-participation.ts"]; +const sourceHashes = Object.fromEntries(await Promise.all(proofFiles.map(async file => [file, createHash("sha256").update(await fs.readFile(path.join(root, file))).digest("hex")]))); +const baselineDir = await fs.mkdtemp(path.join(root, "server/scripts/.privacy-baseline-")); +const oldSource = git("show", `${baselineRevision}:${modulePath}`); +const rewritten = oldSource.replace(/from "(\.{1,2}\/[^\"]+)"/g, (_match, specifier: string) => `from ${JSON.stringify(fileURLToPath(new URL(specifier, pathToFileURL(path.join(root, modulePath)))))}`); +const oldModule = path.join(baselineDir, "authorization.ts"); +await fs.writeFile(oldModule, rewritten); +const baseline = await import(pathToFileURL(oldModule).href) as typeof import("../src/services/authorization.js"); +const treatment = process.argv.includes("--treatment-is-baseline") ? baseline.issueReadSqlCondition : issueReadSqlCondition; +let database: Awaited> | undefined; +let admin: Db | undefined, db: Db | undefined; +const dbName = `privacy_ancestry_${randomUUID().replaceAll("-", "")}`; +const keepAlive = setInterval(() => {}, 1000); +type Query = { sql: string; params: unknown[] }; +function captureDb(target: Db, queries: Query[]): Db { + function builder(value: any) { + const from = value.from.bind(value); + value.from = (...args: any[]) => { + const query = from(...args), execute = query.execute.bind(query); + query.execute = async () => { queries.push(query.toSQL()); return execute(); }; + return query; + }; + return value; + } + return new Proxy(target, { get(target, key) { + if (key === "select") return (...fields: any[]) => builder((target.select as any)(...fields)); + if (key === "with") return (...ctes: any[]) => { + const value = (target.with as any)(...ctes), select = value.select.bind(value); + value.select = (...fields: any[]) => builder(select(...fields)); + return value; + }; + return Reflect.get(target, key); + } }); +} +const stats = (values: number[]) => ({ samplesMs: values, medianMs: [...values].sort((a,b) => a-b)[Math.floor(values.length/2)]!, minMs: Math.min(...values), maxMs: Math.max(...values) }); +try { + if (process.env.PRIVACY_BENCHMARK_ADMIN_URL) { + const url = new URL(process.env.PRIVACY_BENCHMARK_ADMIN_URL); + assert.equal(url.hostname, "127.0.0.1", "External benchmark PostgreSQL must be disposable loopback PostgreSQL"); + admin = createDb(url.href, { maxConnections: 1 }); + await admin.$client.unsafe(`CREATE DATABASE "${dbName}"`); + url.pathname = `/${dbName}`; + await applyPendingMigrations(url.href); + db = createDb(url.href, { maxConnections: 1 }); + } else { + database = await startEmbeddedPostgresTestDatabase("privacy-benchmark-"); + db = createDb(database.connectionString, { maxConnections: 1 }); + } + await db.$client.unsafe("set client_min_messages=warning"); + const settings = await db.$client.unsafe("select version(),current_setting('shared_buffers') shared_buffers,current_setting('work_mem') work_mem,current_setting('jit_above_cost') jit_above_cost,current_setting('jit_inline_above_cost') jit_inline_above_cost,current_setting('jit_optimize_above_cost') jit_optimize_above_cost"); + const setupMs = performance.now()-started; + const shapes = [ + { name: "small", count: 20, participated: 20, history: 2, comments: 2, payloadBytes: 512 }, + { name: "dense", count: 1200, participated: 1200, history: 50, comments: 20, decoyHistory: 240000 }, + { name: "sparse", count: 1200, participated: 12, history: 50, comments: 20, decoyHistory: 240000 }, + { name: "deep", count: 64, participated: 64, history: 2, comments: 2, graph: "deep" as const }, + { name: "fanout", count: 1200, participated: 1200, history: 2, comments: 2, graph: "fanout" as const }, + { name: "cycle", count: 32, participated: 32, history: 2, comments: 2, graph: "cycle" as const }, + ].filter(shape => !arg("fixture") || shape.name === arg("fixture")); + assert.ok(shapes.length, "Unknown fixture"); + const results = []; + for (const shape of shapes) { + const seedStart = performance.now(); + const fixture = await seedPrivacyFixture(db, shape); + const fixtureMs = performance.now()-seedStart; + const actor = { type: "board" as const, source: "session", userId: fixture.userId, authorizingUserId: fixture.userId }; + const cases = []; + for (const readMode of ["scoped", "unrestricted"]) { + if (shape.graph && readMode === "unrestricted") continue; + const conditions = { + baseline: readMode === "scoped" ? await baseline.issueReadSqlCondition(db, actor) : sql`true`, + treatment: readMode === "scoped" ? await treatment(db, actor) : sql`true`, + }; + for (const jit of ["on", "off"]) { + await db.$client.unsafe(`set jit=${jit}`); + for (const caller of ["badge", "list"] as const) { + const invoke = (target: Db, arm: "baseline" | "treatment") => caller === "badge" + ? issueService(target).unreadInboxIssueIds(fixture.companyId, fixture.userId, conditions[arm]) + : issueService(target).list(fixture.companyId, { readCondition: conditions[arm], touchedByUserId: fixture.userId, limit: 100, sortField: "id", sortDir: "asc" }); + const samples = { baseline: [] as number[], treatment: [] as number[] }; + for (let round=0; round<7; round++) { + let before: unknown, after: unknown; + for (const arm of round%2 ? ["treatment", "baseline"] as const : ["baseline", "treatment"] as const) { + const start = performance.now(); + const result = await invoke(db, arm); + if (round >= 2) samples[arm].push(performance.now()-start); + if (arm === "baseline") before=result; else after=result; + } + assert.deepEqual(after, before, `${shape.name}/${readMode}/${jit}/${caller}: full service output/order`); + } + const plans = []; + let expectedRows: unknown[][] | undefined; + for (const arm of ["baseline", "treatment"] as const) { + const queries: Query[] = []; + await invoke(captureDb(db, queries), arm); + assert.ok(queries.length >= (caller === "badge" ? 2 : 1), "Capture the complete real caller"); + const rows: unknown[][] = [], queryPlans = []; + for (const query of queries) { + rows.push(await db.$client.unsafe(query.sql, query.params as never[]).values()); + const [explain] = await db.$client.unsafe(`explain (analyze,buffers,format json) ${query.sql}`, query.params as never[]); + queryPlans.push({ ...query, rows: rows.at(-1)!.length, work: recursiveIssueWork(explain["QUERY PLAN"][0].Plan), plan: explain["QUERY PLAN"] }); + } + if (arm === "baseline") expectedRows = rows; + else assert.deepEqual(rows, expectedRows, `${shape.name}/${readMode}/${jit}/${caller}: every positional SQL field and order`); + plans.push({ arm, queries: queryPlans }); + } + const before = stats(samples.baseline), after = stats(samples.treatment); + cases.push({ readMode, jit, caller, baseline: before, treatment: after, speedup: before.medianMs/after.medianMs, positionalEquality: true, serviceEquality: true, plans }); + console.error(JSON.stringify({ fixture: shape.name, readMode, jit, caller, baseline: before.medianMs, treatment: after.medianMs })); + } + } + } + results.push({ fixture, fixtureMs, cases }); + await db.$client.unsafe("truncate companies cascade"); + } + const dependencyVersions = Object.fromEntries(await Promise.all(["postgres", "drizzle-orm"].map(async name => [name, JSON.parse(await fs.readFile(path.join(root,`packages/db/node_modules/${name}/package.json`),"utf8")).version]))); + const report = { baselineRevision, dependencyVersions, maxDenseMs: Number.isFinite(maxDenseMs) ? maxDenseMs : null, treatmentRevision: git("rev-parse", "HEAD"), sourceHashes, baselineSourceSha256: createHash("sha256").update(oldSource).digest("hex"), changedRuntime, checkedInLockSha256: createHash("sha256").update(await fs.readFile(path.join(root,"pnpm-lock.yaml"))).digest("hex"), nodeVersion: process.version, settings, setupMs, totalMs: performance.now()-started, warmupsPerArm: 2, measuredSamplesPerArm: 5, results }; + const evidence = arg("evidence-dir"); + if (evidence) { await fs.mkdir(evidence,{ recursive: true }); await fs.writeFile(path.join(evidence,"benchmark.json"),JSON.stringify(report,null,2)); } + console.log(JSON.stringify({ ...report, results: results.map(result => ({ ...result, cases: result.cases.map(({ plans, ...rest }) => rest) })) },null,2)); + if (process.argv.includes("--assert-performance")) { + for (const result of results) for (const c of result.cases) { + if (c.readMode === "scoped") { + for (const query of c.plans.find(p => p.arm === "treatment")!.queries) { + if (!result.fixture.graph) assert.equal(query.work.inspectedRows, 0, "Absent parent frontiers must not inspect company tasks"); + } + if (["dense","sparse"].includes(result.fixture.name)) assert.ok(c.speedup >= 2, `Scoped representative ${result.fixture.name}/${c.jit}/${c.caller} requires 2x improvement`); + } + if (result.fixture.name === "dense" && c.caller === "badge" && c.readMode === "scoped" && c.jit === "on") assert.ok(c.treatment.medianMs <= maxDenseMs, `Original dense scoped badge target ${maxDenseMs}ms remains unmet: ${c.treatment.medianMs.toFixed(2)}ms`); + if (result.fixture.name === "small" || c.readMode === "unrestricted") assert.ok(c.treatment.medianMs <= Math.max(c.baseline.medianMs*1.25,c.baseline.medianMs+2), "Small/open controls must stay within 25% or 2ms"); + } + } +} finally { + await db?.$client.end({timeout:0}); + await database?.cleanup(); + if (admin) { await admin.$client.unsafe(`DROP DATABASE IF EXISTS "${dbName}" WITH (FORCE)`); await admin.$client.end({timeout:0}); } + await fs.rm(baselineDir,{recursive:true,force:true}); + clearInterval(keepAlive); +} diff --git a/server/src/__tests__/authorization-privacy-query.test.ts b/server/src/__tests__/authorization-privacy-query.test.ts new file mode 100644 index 00000000000..3f3c3de068f --- /dev/null +++ b/server/src/__tests__/authorization-privacy-query.test.ts @@ -0,0 +1,128 @@ +import { randomUUID } from "node:crypto"; +import { and, eq, inArray, sql } from "drizzle-orm"; +import { agents, companies, companyMemberships, createDb, issueAccessGrants, issues, projectAccessMembers, projects, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { canActorReadIssuePrivacy, issueReadSqlCondition, type AuthorizationActor } from "../services/authorization.js"; +import { issueService } from "../services/issues.js"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { recursiveIssueWork, seedPrivacyFixture } from "./helpers/privacy-query-fixture.js"; + +describe("privacy ancestry query work", () => { + let database: Awaited>, db: ReturnType; + beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("privacy-query-"); db = createDb(database.connectionString); }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + + it("does not rescan company tasks when resolving absent privacy parents", async () => { + const f = await seedPrivacyFixture(db, { name: "open roots", count: 1200, participated: 0, history: 0, comments: 0 }); + const condition = await issueReadSqlCondition(db, { type: "board", userId: f.userId, authorizingUserId: f.userId }); + const query = db.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, f.companyId), condition)).orderBy(issues.id); + expect((await query).map(row => row.id)).toEqual([...f.ids].sort()); + const statement = query.toSQL(); + await db.execute(sql`set jit=off`); + const [explain] = await db.$client.unsafe(`explain (analyze,buffers,format json) ${statement.sql}`, statement.params as never[]); + const work = recursiveIssueWork(explain["QUERY PLAN"][0].Plan); + expect(work.recursiveNodes).toBeGreaterThan(0); + expect(work.inspectedRows).toBeLessThanOrEqual(f.ids.length * 3); + expect(work.projectProbes).toBeLessThanOrEqual(f.ids.length * 3); + }); + it("retains grant direction, private-only traversal, cycles and tenant boundaries", async () => { + const companyId = randomUUID(), otherCompany = randomUUID(), userId = randomUUID(); + await db.insert(companies).values([companyId, otherCompany].map(id => ({ id, name: "Privacy graph", issuePrefix: `P${id.slice(0,7)}` }))); + const ids = Object.fromEntries(["root", "child", "grandchild", "sibling", "barrier", "behindBarrier", "cycleA", "cycleB", "foreign", "foreignChild"].map(name => [name, randomUUID()])); + await db.insert(issues).values(Object.entries(ids).map(([name,id]) => ({ id, companyId: name === "foreign" ? otherCompany : companyId, title: name, visibility: name === "barrier" ? "open" : "private", createdByUserId: "other-user" }))); + const parents = { child: "root", grandchild: "child", sibling: "root", barrier: "root", behindBarrier: "barrier", cycleA: "cycleB", cycleB: "cycleA", foreignChild: "foreign" }; + for (const [child,parent] of Object.entries(parents)) await db.update(issues).set({ privacyParentIssueId: ids[parent]! }).where(eq(issues.id,ids[child]!)); + const privacyRows = new Map((await db.select().from(issues).where(eq(issues.companyId, companyId))).map(row => [row.id, row])); + const actor: AuthorizationActor = { type: "board", userId, authorizingUserId: userId }; + const [grant] = await db.insert(issueAccessGrants).values({ issueId: ids.child!, subjectType: "user", subjectId: userId, source: "explicit" }).returning(); + const condition = await issueReadSqlCondition(db, actor); + const read = async () => (await db.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId,companyId),condition))).map(row=>row.id).sort(); + expect(await read()).toEqual([ids.child!,ids.grandchild!].sort()); + await db.update(issueAccessGrants).set({ revokedAt:new Date() }).where(eq(issueAccessGrants.id,grant!.id)); + expect(await read()).toEqual([]); + await db.insert(issueAccessGrants).values([ids.root!,ids.cycleB!,ids.foreign!].map(issueId=>({ issueId, subjectType:"user", subjectId:userId, source:"explicit" }))); + expect(await read()).toEqual([ids.root!,ids.child!,ids.grandchild!,ids.sibling!,ids.barrier!,ids.cycleA!,ids.cycleB!].sort()); + for (const [name,expected] of [["cycleA",true],["cycleB",true],["behindBarrier",false],["foreignChild",false]] as const) { + expect(await canActorReadIssuePrivacy(db,actor,privacyRows.get(ids[name]!)!)).toBe(expected); + } + await db.update(issueAccessGrants).set({ revokedAt:new Date() }).where(eq(issueAccessGrants.issueId,ids.cycleB!)); + expect(await canActorReadIssuePrivacy(db,actor,privacyRows.get(ids.cycleA!)!)).toBe(false); + }); + + it("intersects current agent, responsible-user and authorizing-user grants through ancestors", async () => { + const companyId=randomUUID(), owner=randomUUID(), authorizer=randomUUID(); + await db.insert(companies).values({id:companyId,name:"Agent privacy",issuePrefix:`P${companyId.slice(0,7)}`}); + await db.insert(companyMemberships).values([owner,authorizer].map(principalId=>({companyId,principalType:"user",principalId,status:"active",membershipRole:"operator"}))); + const [agent]=await db.insert(agents).values({companyId,name:"Shared agent",adapterType:"process"}).returning(); + const [parent]=await db.insert(issues).values({companyId,title:"Ancestor",visibility:"private",createdByUserId:"someone-else"}).returning(); + const [child]=await db.insert(issues).values({companyId,title:"Descendant",visibility:"private",privacyParentIssueId:parent!.id,createdByUserId:"someone-else"}).returning(); + const actor:AuthorizationActor={type:"agent",agentId:agent!.id,onBehalfOfUserId:owner,authorizingUserId:authorizer}; + const readable=()=>canActorReadIssuePrivacy(db,actor,child!); + await db.insert(issueAccessGrants).values({issueId:parent!.id,subjectType:"agent",subjectId:agent!.id,source:"explicit"}); + expect(await readable()).toBe(false); + await db.insert(issueAccessGrants).values({issueId:parent!.id,subjectType:"user",subjectId:owner,source:"explicit"}); + expect(await readable()).toBe(false); + const [grant]=await db.insert(issueAccessGrants).values({issueId:parent!.id,subjectType:"user",subjectId:authorizer,source:"explicit"}).returning(); + expect(await readable()).toBe(true); + const condition=await issueReadSqlCondition(db,actor); + expect((await db.select({id:issues.id}).from(issues).where(and(eq(issues.id,child!.id),condition))).length).toBe(1); + await db.update(issueAccessGrants).set({revokedAt:new Date()}).where(eq(issueAccessGrants.id,grant!.id)); + expect(await readable()).toBe(false); + expect(await db.select({id:issues.id}).from(issues).where(and(eq(issues.id,child!.id),condition))).toEqual([]); + await db.update(issueAccessGrants).set({revokedAt:null}).where(eq(issueAccessGrants.id,grant!.id)); + await db.update(companyMemberships).set({status:"archived"}).where(and(eq(companyMemberships.companyId,companyId),eq(companyMemberships.principalId,owner))); + expect(await readable()).toBe(false); + }); + + it("applies fresh inherited project membership before list pagination", async () => { + const f=await seedPrivacyFixture(db,{name:"paged",count:130,participated:130,history:1,comments:1}); + await db.update(issues).set({createdByUserId:"other-human",visibility:"private"}).where(eq(issues.companyId,f.companyId)); + const [project]=await db.insert(projects).values({companyId:f.companyId,name:"Private project",visibility:"private"}).returning(); + const ordered=[...f.ids].sort(), allowed=ordered.slice(100); + await db.update(issues).set({projectId:project!.id}).where(inArray(issues.id,allowed)); + const actor:AuthorizationActor={type:"board",userId:f.userId,authorizingUserId:f.userId}; + const condition=await issueReadSqlCondition(db,actor), svc=issueService(db); + const read=()=>svc.list(f.companyId,{readCondition:condition,limit:20,sortField:"id",sortDir:"asc",touchedByUserId:f.userId}); + expect(await read()).toEqual([]); + const [member]=await db.insert(projectAccessMembers).values({companyId:f.companyId,projectId:project!.id,subjectType:"user",subjectId:f.userId}).returning(); + expect((await read()).map(row=>row.id)).toEqual(allowed.slice(0,20)); + expect((await svc.unreadInboxIssueIds(f.companyId,f.userId,condition)).sort()).toEqual(allowed); + await db.delete(projectAccessMembers).where(eq(projectAccessMembers.id,member!.id)); + expect(await read()).toEqual([]); + expect(await svc.unreadInboxIssueIds(f.companyId,f.userId,condition)).toEqual([]); + }); + + it("traverses a complete deep private chain with a root-only grant and revokes it fresh", async () => { + const f=await seedPrivacyFixture(db,{name:"deep",count:64,participated:0,history:0,comments:0,graph:"deep"}); + const actor:AuthorizationActor={type:"board",userId:f.userId,authorizingUserId:f.userId}; + const condition=await issueReadSqlCondition(db,actor); + const query=()=>db.select({id:issues.id}).from(issues).where(and(eq(issues.companyId,f.companyId),condition)).orderBy(issues.id); + expect((await query()).map(row=>row.id)).toEqual([...f.ids].sort()); + await db.update(issueAccessGrants).set({revokedAt:new Date()}).where(eq(issueAccessGrants.issueId,f.ids[0]!)); + expect(await query()).toEqual([]); + }); + + it("follows an open ancestor in a private project and rechecks its restriction and inherited grant", async () => { + const companyId = randomUUID(), userId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Project ancestor", issuePrefix: `P${companyId.slice(0, 7)}` }); + const [project] = await db.insert(projects).values({ companyId, name: "Private ancestor project", visibility: "private" }).returning(); + const [grandparent] = await db.insert(issues).values({ companyId, title: "Granted grandparent", visibility: "private", createdByUserId: "other-user" }).returning(); + const [parent] = await db.insert(issues).values({ companyId, title: "Open parent in private project", visibility: "open", projectId: project!.id, privacyParentIssueId: grandparent!.id, createdByUserId: "other-user" }).returning(); + const [child] = await db.insert(issues).values({ companyId, title: "Private child", visibility: "private", privacyParentIssueId: parent!.id, createdByUserId: "other-user" }).returning(); + const actor: AuthorizationActor = { type: "board", userId, authorizingUserId: userId }; + const condition = await issueReadSqlCondition(db, actor); + const read = async () => (await db.select({ id: issues.id }).from(issues).where(and(eq(issues.id, child!.id), condition))).map(row => row.id); + expect(await canActorReadIssuePrivacy(db, actor, child!)).toBe(false); + const [grant] = await db.insert(issueAccessGrants).values({ issueId: grandparent!.id, subjectType: "user", subjectId: userId, source: "explicit" }).returning(); + expect(await read()).toEqual([child!.id]); + expect(await canActorReadIssuePrivacy(db, actor, child!)).toBe(true); + await db.update(projects).set({ visibility: "open" }).where(eq(projects.id, project!.id)); + expect(await read()).toEqual([]); + await db.update(projects).set({ visibility: "private" }).where(eq(projects.id, project!.id)); + expect(await read()).toEqual([child!.id]); + await db.update(issueAccessGrants).set({ revokedAt: new Date() }).where(eq(issueAccessGrants.id, grant!.id)); + expect(await read()).toEqual([]); + expect(await canActorReadIssuePrivacy(db, actor, child!)).toBe(false); + }); + +}); diff --git a/server/src/__tests__/helpers/privacy-query-fixture.ts b/server/src/__tests__/helpers/privacy-query-fixture.ts new file mode 100644 index 00000000000..61354c7ce3e --- /dev/null +++ b/server/src/__tests__/helpers/privacy-query-fixture.ts @@ -0,0 +1,50 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { companies, type Db } from "@paperclipai/db"; + +export type PrivacyPlan = Record; +export function recursiveIssueWork(plan: PrivacyPlan) { + let recursiveNodes = 0, inspectedRows = 0, projectProbes = 0; + function walk(node: PrivacyPlan, recursive = false) { + if (node["Node Type"] === "Recursive Union") { + recursiveNodes++; + for (const child of (node.Plans ?? []).slice(1)) walk(child, true); + return; + } + const visits = (Number(node["Actual Rows"] ?? 0) + Number(node["Rows Removed by Filter"] ?? 0)) * Number(node["Actual Loops"] ?? 0); + if (recursive && node["Relation Name"] === "issues") inspectedRows += visits; + if (recursive && node["Relation Name"] === "projects") projectProbes += Number(node["Actual Loops"] ?? 0); + for (const child of node.Plans ?? []) walk(child, recursive); + } + walk(plan); + return { recursiveNodes, inspectedRows, projectProbes }; +} + +export async function seedPrivacyFixture(db: Db, input: { name: string; count: number; participated: number; history: number; comments: number; payloadBytes?: number; decoyHistory?: number; graph?: "deep" | "fanout" | "cycle" }) { + const companyId = randomUUID(), userId = `privacy-${randomUUID()}`; + const ids = Array.from({ length: input.count }, () => randomUUID()); + const payload = randomBytes(Math.ceil((input.payloadBytes ?? 4096) * 3 / 4)).toString("base64"); + await db.insert(companies).values({ id: companyId, name: `Synthetic ${input.name}`, issuePrefix: `P${companyId.slice(0, 7)}` }); + await db.$client.unsafe(`insert into issues(id,company_id,title,status,created_by_user_id,created_at,updated_at,visibility) + select id,$2::uuid,'Synthetic '||n,'todo',case when n <= $4 and n%2=0 then $3 else 'other-human' end, + '2026-01-01'::timestamptz,'2026-01-01'::timestamptz+n*interval '1 second',$5 + from unnest($1::uuid[]) with ordinality as f(id,n)`, [ids, companyId, userId, input.participated, input.graph ? "private" : "open"]); + if (input.graph) { + const parents = ids.map((_, index) => input.graph === "deep" ? (index ? ids[index - 1]! : null) : input.graph === "cycle" ? ids[(index + 1) % ids.length]! : index ? ids[0]! : null); + await db.$client.unsafe(`update issues i set privacy_parent_issue_id=f.parent from unnest($1::uuid[],$2::uuid[]) f(id,parent) where i.id=f.id`, [ids, parents]); + await db.$client.unsafe(`insert into issue_access_grants(issue_id,subject_type,subject_id,source) values($1,'user',$2,'explicit')`, [ids[0]!, userId]); + } + if (input.history) await db.$client.unsafe(`insert into activity_log(company_id,entity_type,entity_id,actor_type,actor_id,action,details,created_at) + select $1::uuid,'issue',i.id::text,case when n=1 or n%10=0 then 'user' else 'agent' end, + case when (n=1 or n%20=0) and i.id=ANY($5::uuid[]) then $4 when n=1 or n%10=0 then 'other-human' else 'synthetic-worker' end, + 'issue.updated',jsonb_build_object('padding',$2::text),'2026-01-01'::timestamptz+n*interval '1 second' + from issues i cross join generate_series(1,$3::int)n where company_id=$1::uuid order by n,i.id`, [companyId, payload, input.history, userId, ids.slice(0, input.participated)]); + if (input.comments) await db.$client.unsafe(`insert into issue_comments(company_id,issue_id,body,created_at) + select $1::uuid,id,$2,'2026-01-02'::timestamptz+n*interval '1 second' + from issues cross join generate_series(1,$3::int)n where company_id=$1::uuid order by n,id`, [companyId, payload, input.comments]); + if (input.decoyHistory) await db.$client.unsafe(`insert into activity_log(company_id,entity_type,entity_id,actor_type,actor_id,action,details,created_at) + select $1::uuid,'heartbeat_run',gen_random_uuid()::text,'agent','synthetic-worker','heartbeat.run.created', + jsonb_build_object('padding',left($2,256)),'2026-01-01'::timestamptz+n*interval '1 second' + from generate_series(1,$3::int)n`, [companyId, payload, input.decoyHistory]); + await db.$client.unsafe("ANALYZE"); + return { ...input, companyId, userId, ids, payloadBytes: Buffer.byteLength(payload), activityRows: input.count * input.history, commentRows: input.count * input.comments }; +} diff --git a/server/src/services/authorization.ts b/server/src/services/authorization.ts index 0deca602bc7..a9c65b694ec 100644 --- a/server/src/services/authorization.ts +++ b/server/src/services/authorization.ts @@ -629,10 +629,18 @@ function issuePrivacyPrincipalCondition(actor: AuthorizationActor): SQL p.responsible_user_id, p.created_by_user_id, p.assignee_user_id, p.assignee_agent_id from issues p where p.id = ${issues.id} and p.company_id = ${issues.companyId} union - select p.id, p.company_id, p.privacy_parent_issue_id, p.visibility, p.project_id, + (select p.id, p.company_id, p.privacy_parent_issue_id, p.visibility, p.project_id, p.responsible_user_id, p.created_by_user_id, p.assignee_user_id, p.assignee_agent_id - from issues p join privacy_ancestors c on p.id = c.privacy_parent_issue_id and p.company_id = c.company_id - where p.visibility = 'private' or exists (select 1 from projects pp where pp.id = p.project_id and pp.company_id = p.company_id and pp.visibility = 'private') + from privacy_ancestors c + cross join lateral ( + select p.id, p.company_id, p.privacy_parent_issue_id, p.visibility, p.project_id, + p.responsible_user_id, p.created_by_user_id, p.assignee_user_id, p.assignee_agent_id + from issues p where p.id = c.privacy_parent_issue_id and p.company_id = c.company_id + limit 1 + ) p + where c.privacy_parent_issue_id is not null + and (p.visibility = 'private' or exists (select 1 from projects pp where pp.id = p.project_id and pp.company_id = p.company_id and pp.visibility = 'private')) + limit 1) ) select 1 from privacy_ancestors n where ${implicit} or ${grant} or ${projectMember}