Files
PaperClipAI/server/scripts/benchmark-sidebar-subtasks.ts
T
DottaandPaperclip 5b750fdf25 feat(ui): add foldable sidebar subtasks with bounded refreshes (#21649)
## Thinking Path

> - Paperclip helps people manage AI agents for work.
> - The sidebar helps people return to recent tasks.
> - Recent parents need foldable child tasks.
> - Busy companies must not refresh every child group for unrelated
work.
> - Exact unread counts must preserve personal reads and task access.
> - This pull request adds child groups and bounds their refresh work.
> - People can scan subtasks without repeated background requests.

## Linked Issues or Issue Description

**Subsystem affected**

Task navigation in the standard and Classic sidebars.

**Problem or motivation**

Recent parent tasks did not show their subtasks. The first
implementation refreshed every previously viewed group. One unrelated
update caused 52 child-page requests and one summary request after
viewing 50 groups. Exact child counts also repeated privacy and unread
checks per child.

**Proposed solution**

Show direct subtasks with independent folding and ten-row pages. Refresh
affected groups and summary batches. Stop offscreen reads. Bound
child-request concurrency. Aggregate viewer facts once and preserve the
existing inbox unread rule.

**Roadmap alignment**

This extends existing task navigation. The existing feature PR is reused
for its performance repair. No new PR duplicates it.

## What Changed

- Add shared parent groups to both sidebar layouts. Preserve Classic row
actions and active-child visibility.
- Keep unread state personal to each task. Folding and paging do not
mark tasks read. Counts include readable children beyond ten.
- Target live refreshes with authorized current and previous parent IDs.
Reuse parent metadata from the narrow run-binding read. Resolve fallback
lookups once per delivery batch. Keep the previous summary fallback for
older servers.
- Limit child requests to three at a time. Cancel active and queued
reads when groups leave the viewport.
- Discard extra cached pages on Show less and collapse.
- Compute exact navigation counts in one SQL statement. Materialize
readable candidate IDs once. Use indexed participation sources and
conservative unread candidates before the shared inbox rule.
- Skip recursive privacy walks for standalone open tasks without a
project. Keep the recursive rule for all restricted or inherited shapes
and all delegated principals.
- Add request-count, cancellation, reparenting, read-receipt,
inbox-parity and privacy regressions. Include a disposable PostgreSQL
benchmark.
- Update product and API documentation. Retain the production Storybook
states.

## Verification

- Published head: `4148861437c1b6b635479ad92509dbc71155a232`. Its tree
matches local `04be9d66d481d9bcd1c4d6158acf063a1e65ac7e`:
`a23533e93d3d155918f0254d296a7fd2f0c7c12f`.
- Rebased on master `ca23330c69b1f2b25bdadfcbda68317db97deaef`. Remote
publication preserves prior feature commits.
- The request regression failed before the fix and passes after it. An
unrelated event now sends zero child requests. Current servers also
avoid the unrelated summary request. Old-server events keep one summary
fallback.
- A relevant update after Show less reads one page. Reparenting
refreshes two affected groups. Offscreen reads stay at zero and resume
once on return. Active and queued cancellation tests pass.
- The same synthetic authenticated HTTP workload with 50,000 read
children improved from 767 ms to 92.84 ms median. Five warm samples per
case. The all-unread case improved from 463.85 to 351.65 ms. Untouched
children improved from 208.01 to 131.09 ms. The old-receipt case
improved from 799.70 to 245.80 ms. These local measurements are not a
production capacity guarantee.
- Focused PostgreSQL routes, inbox-parity, privacy and WebSocket tests
passed: 38 tests.
- All 325 affected UI tests passed. All 38 focused PostgreSQL, privacy
and WebSocket tests passed. UI typecheck and low-memory server
declaration checks passed.
- All 41 Chromium Storybook checks passed. They cover 20 states in both
themes and keyboard, folding, paging and unread behavior. They use
production components with HTTP fixtures.
- Both new review findings failed as regressions before their fixes and
pass afterward. Explicit child reads include waiting external
conversations. Deleting a child publishes a separate parent-authorized
update without deleted-child content. Both review threads are resolved.
- `pnpm verify --explain` returned exit 2 because product documentation
requires broader CI. [The prior CI
run](https://github.com/paperclipai/paperclip/actions/runs/43711489306)
found type errors in the new test fixtures. The same errors failed
locally. The fixture fix passes `node
server/scripts/typecheck-layers.mjs --layer test-tree-0` and all 21
issue summary route tests. Production code is unchanged. That CI run
then found a live-delivery read regression. The existing test failed
locally with eight statements against its six-statement budget. A new
PostgreSQL regression also found three parent lookups for two same-batch
calls. Run delivery now reuses parent metadata from its narrow
run-binding query. Fallback lookups resolve to one promise. Both
regressions pass without increasing the budgets. All 74 targeted route,
WebSocket and privacy tests pass. The affected below, above and
test-tree-0 typecheck layers pass. [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/43715541480/attempts/2)
passes on the published head. All 62 check runs are terminal and clean:
57 succeeded, two were neutral and three were skipped. Typecheck, build,
policy, all general and serialized server suites, E2E and the [final
verify
gate](https://github.com/paperclipai/paperclip/actions/runs/43715541480/job/132829783231)
pass. The separate security status passes. Attempt 1's server shard 7/12
reached its 20-minute job limit without a test verdict. One retry of
that job passed without a source change. [Greptile completed
successfully on this
head](https://github.com/paperclipai/paperclip/runs/132816163906) with a
[5/5
score](https://github.com/paperclipai/paperclip/pull/21649#issuecomment-6883916606).
All five review threads are resolved. The final CI gate is complete.
- A full local Storybook build exceeded available memory while server
typechecking ran. Browser verification uses the current-source
development server. Rebuilding stale Runner and SDK declarations
resolved the initial server check errors. The first browser attempt
lacked Chromium; all checks passed after installation.
- Reproduce the database measurements with `pnpm --filter
@paperclipai/server exec tsx scripts/benchmark-sidebar-subtasks.ts`.
- Extra local shard diagnosis ran 164 suites. Under this shell's `0077`
creation mask, 157 suites passed and three had fixture permission or
timing failures. Rerunning those three with `0022` resolved the
permission failures. The remaining ten cases passed with one worker: two
workspace cases and eight archive cases. The eight archive cases also
passed on the pre-performance-fix revision, whose archive source is
identical. The broad local shard is not claimed as one passing
execution. The same-head CI retry supplies complete final CI coverage.
- Nothing is merged or deployed.

## Risks

- Exact counts still grow with the readable hierarchy and comment
history. The change reduces repeated work; it does not cap or
approximate counts.
- A relevant summary batch can contain several recent parents. Older
servers retain a broader summary fallback until upgraded.
- Live delivery adds bounded, permission-checked parent lookups. Short
delivery-window reuse limits repeated reads.
- The authorization fast path applies only to open tasks with no privacy
parent and no project. Restricted cases retain the shared predicate.
- No migration is required.

## Model Used

OpenAI Codex, based on GPT-6. Used code execution, native GitHub tools,
PostgreSQL and Chromium verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked a public issue or described the issue in this
PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal issue
id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open findings
- [x] I will address all review comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-15 08:51:04 -06:00

114 lines
9.4 KiB
TypeScript

import assert from 'node:assert/strict';
import { randomUUID } from 'node:crypto';
import { mkdir, writeFile } from 'node:fs/promises';
import { resolve } from 'node:path';
import { pathToFileURL } from 'node:url';
import { performance } from 'node:perf_hooks';
import express from 'express';
import { drizzle } from 'drizzle-orm/postgres-js';
import { createDb, startEmbeddedPostgresTestDatabase } from '../../packages/db/src/index.ts';
import { issueRoutes } from '../../server/src/routes/issues.ts';
import { errorHandler } from '../../server/src/middleware/index.ts';
import { seedIssueReadVisibilityFixture } from '../../server/src/__tests__/helpers/issue-read-visibility-fixture.ts';
import { listIssueNavigationState } from '../../server/src/services/issue-summaries.ts';
import { issueReadSqlCondition } from '../../server/src/services/authorization.ts';
// Run from the server package: pnpm --filter @paperclipai/server exec tsx scripts/benchmark-sidebar-subtasks.ts
// All data is synthetic. The temporary PostgreSQL cluster is removed in finally.
const directory = resolve(process.env.PAPERCLIP_BENCHMARK_OUTPUT ?? '../test-results/sidebar-subtasks-benchmark');
await mkdir(directory, { recursive: true });
const out = pathToFileURL(directory + '/');
const report: any = { synthetic: true, node: process.version, pool: 10, samples: 5, cases: [] };
const keeper = setInterval(() => {}, 1000);
const fixture = await startEmbeddedPostgresTestDatabase('sidebar-subtasks-perf-');
const db = createDb(fixture.connectionString, { maxConnections: 10 });
const client = db.$client;
const statements: Array<{query: string; params: unknown[]}> = [];
let record = false;
const counted = drizzle(client, { schema: db._.fullSchema, logger: { logQuery(query, params) { if (record) statements.push({query, params}); } } }) as typeof db;
let server: ReturnType<typeof express>['listen'] extends (...a:any[]) => infer R ? R : never;
const stats = (xs: number[]) => { const s = [...xs].sort((a,b)=>a-b); return {medianMs:+s[Math.floor(s.length/2)]!.toFixed(2), maxMs:+s.at(-1)!.toFixed(2), samplesMs:xs.map(x=>+x.toFixed(2))}; };
try {
process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = 'enforce';
const f = await seedIssueReadVisibilityFixture(db, {publicCount: 50});
const actor = f.actors['cloud tenant owner'];
const userId = (actor as any).userId!;
const parents = f.publicIds;
// Make each measured parent independent and participated-in by this viewer.
await client.unsafe('update issues set parent_id=null, created_by_user_id=$2 where company_id=$1 and id=any($3::uuid[])',[f.companyId,userId,parents]);
await client.unsafe(`create table perf_parents as select id from issues where id=any($1::uuid[])`,[parents]);
await client.unsafe(`insert into issues (company_id,title,status,created_by_user_id) select $1,'Unrelated history '||n,'done',$2 from generate_series(1,20000)n`,[f.companyId,f.strangerUserId]);
await client.unsafe(`insert into activity_log(company_id,actor_type,actor_id,action,entity_type,entity_id,created_at) select $1,'system','synthetic-agent','issue.updated','issue',$2,now() from generate_series(1,100000)n`,[f.companyId,parents[0]]);
const connections=await Promise.all(Array.from({length:10},()=>client.reserve()));
for(const connection of connections){await connection.unsafe("set statement_timeout='20s'");connection.release();}
const app=express();
app.use((req,_res,next)=>{req.actor=actor as any;next();});
app.use('/api',issueRoutes(counted,{} as never)); app.use(errorHandler);
server=app.listen(0,'127.0.0.1'); await new Promise<void>(r=>server.once('listening',r));
const origin=`http://127.0.0.1:${(server.address() as any).port}`;
const summaryPath=`/api/companies/${f.companyId}/issues/summaries?ids=${parents.join(',')}`;
async function read(path:string) {const r=await fetch(origin+path);const body=await r.json();assert.equal(r.status,200,JSON.stringify(body));return body;}
async function measure(path:string) {await read(path);const times=[];for(let n=0;n<5;n++){const t=performance.now();await read(path);times.push(performance.now()-t);}record=true;statements.length=0;const body=await read(path);record=false;return {time:stats(times),statements:statements.length,sqlBytes:statements.reduce((n,x)=>n+Buffer.byteLength(x.query),0),responseBytes:Buffer.byteLength(JSON.stringify(body)),body};}
report.postgres=(await client.unsafe('select version()'))[0]!.version;
report.fixture={parents:50,noiseIssues:20000,noiseActivity:100000,privacy:'enforce',actor:'cloud tenant owner',commentBytes:128};
let previous=0;
for(const perParent of [0,10,100,1000]) {
if(perParent>previous) {
await client.unsafe(`insert into issues(company_id,parent_id,title,status,created_by_user_id,created_at,updated_at)
select $1,p.id,'Synthetic child '||n,'todo',$2,'2026-01-01','2026-01-01' from perf_parents p cross join generate_series($3::int,$4::int)n`,[f.companyId,userId,previous+1,perParent]);
// One external comment per child, deliberately all read: the unread test
// must check its pending/withdrawal branches rather than short-circuit.
await client.unsafe(`insert into issue_comments(company_id,issue_id,author_user_id,body,created_at)
select $1,i.id,$2,repeat('x',128),'2026-01-02' from issues i where i.company_id=$1 and i.parent_id=any($3::uuid[]) and not exists(select 1 from issue_comments c where c.issue_id=i.id)`,[f.companyId,f.strangerUserId,parents]);
await client.unsafe(`insert into issue_read_states(company_id,issue_id,user_id,last_read_at)
select $1,i.id,$2,'2026-01-03' from issues i where i.company_id=$1 and i.parent_id=any($3::uuid[]) on conflict(company_id,issue_id,user_id) do nothing`,[f.companyId,userId,parents]);
}
await client.unsafe('vacuum analyze');
const base=await measure(summaryPath);
const nav=await measure(summaryPath+'&navigation=true');
for(const row of nav.body) {assert.equal(row.subtaskCount,perParent);assert.equal(row.unreadSubtaskCount,0);}
const plans: any[]=[];
for(const [index,s] of statements.entries()) {
const plan=await client.unsafe('explain (analyze,buffers,format json) '+s.query,s.params as any[]);
plans.push({index,sql:s.query,plan:plan[0]['QUERY PLAN']});
}
await writeFile(new URL(`plans-${perParent}.json`,out),JSON.stringify(plans,null,2));
const childPath=`/api/companies/${f.companyId}/issues?parentId=${parents[0]}&view=rows&limit=10&sortField=id&sortDir=asc&includeRoutineExecutions=true&includeReadState=true`;
const page=perParent?await measure(childPath):null;
const item={childrenPerParent:perParent,totalCountedChildren:perParent*50,baseline:{...base,body:undefined},navigation:{...nav,body:undefined},firstChildPage:page?{...page,body:undefined,rows:page.body.length}:null};
report.cases.push(item);console.log(JSON.stringify(item));
await writeFile(new URL('database-results.json',out),JSON.stringify(report,null,2));
previous=perParent;
}
// Compare the same large hierarchy when children are unread and when they
// have never been touched by this person.
for(const state of ['unread','untouched']) {
if(state==='unread') await client.unsafe(`update issue_read_states set last_read_at='2026-01-01' where company_id=$1`,[f.companyId]);
else await client.unsafe(`update issues set created_by_user_id=$2 where company_id=$1 and parent_id=any($3::uuid[])`,[f.companyId,f.strangerUserId,parents]);
await client.unsafe('vacuum analyze');
const m=await measure(summaryPath+'&navigation=true');
assert.ok(m.body.every((x:any)=>x.unreadSubtaskCount===(state==='unread'?1000:0)));
const plans=[];
for(const [index,s] of statements.entries()) {
const plan=await client.unsafe('explain (analyze,buffers,format json) '+s.query,s.params as any[]);
plans.push({index,sql:s.query,plan:plan[0]['QUERY PLAN']});
}
await writeFile(new URL(`plans-${state}.json`,out),JSON.stringify(plans,null,2));
const item={state,childrenPerParent:1000,...m,body:undefined};report.cases.push(item);console.log(JSON.stringify(item));
}
// A mature viewer's old withdrawal receipts exercise the indexed JSON lookup.
await client.unsafe(`insert into activity_log(company_id,actor_type,actor_id,action,entity_type,entity_id,details,created_at)
select $1,'user',$2,'issue.thread_interaction_withdrawn','issue',$3,jsonb_build_object('handledForUserId',$2::text,'receipt',jsonb_build_object('outcome','withdrawn','issueId',$3::text)),'2025-12-01' from generate_series(1,5000)n`,[f.companyId,userId,parents[0]]);
await client.unsafe(`update issues set created_by_user_id=$2 where company_id=$1 and parent_id=any($3::uuid[])`,[f.companyId,userId,parents]);
await client.unsafe(`update issue_read_states set last_read_at='2026-01-03' where company_id=$1`,[f.companyId]);
await client.unsafe('vacuum analyze');
const receipts=await measure(summaryPath+'&navigation=true');
report.receiptStress={oldViewerReceipts:5000,...receipts,body:undefined};console.log(JSON.stringify(report.receiptStress));
const readCondition=await issueReadSqlCondition(db,actor);
const q=()=>listIssueNavigationState(db,f.companyId,parents,readCondition,userId);
const start=performance.now();await Promise.all(Array.from({length:3},q));report.concurrentThreeServiceRequestsMs=+(performance.now()-start).toFixed(2);
await writeFile(new URL('database-results.json',out),JSON.stringify(report,null,2));
} finally {
server?.close(); await client.end({timeout:0}); await fixture.cleanup(); clearInterval(keeper);
}