mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip helps people manage AI agents for work. > - The sidebar helps people return to recent tasks. > - Recent parents need foldable child tasks. > - Busy companies must not refresh every child group for unrelated work. > - Exact unread counts must preserve personal reads and task access. > - This pull request adds child groups and bounds their refresh work. > - People can scan subtasks without repeated background requests. ## Linked Issues or Issue Description **Subsystem affected** Task navigation in the standard and Classic sidebars. **Problem or motivation** Recent parent tasks did not show their subtasks. The first implementation refreshed every previously viewed group. One unrelated update caused 52 child-page requests and one summary request after viewing 50 groups. Exact child counts also repeated privacy and unread checks per child. **Proposed solution** Show direct subtasks with independent folding and ten-row pages. Refresh affected groups and summary batches. Stop offscreen reads. Bound child-request concurrency. Aggregate viewer facts once and preserve the existing inbox unread rule. **Roadmap alignment** This extends existing task navigation. The existing feature PR is reused for its performance repair. No new PR duplicates it. ## What Changed - Add shared parent groups to both sidebar layouts. Preserve Classic row actions and active-child visibility. - Keep unread state personal to each task. Folding and paging do not mark tasks read. Counts include readable children beyond ten. - Target live refreshes with authorized current and previous parent IDs. Reuse parent metadata from the narrow run-binding read. Resolve fallback lookups once per delivery batch. Keep the previous summary fallback for older servers. - Limit child requests to three at a time. Cancel active and queued reads when groups leave the viewport. - Discard extra cached pages on Show less and collapse. - Compute exact navigation counts in one SQL statement. Materialize readable candidate IDs once. Use indexed participation sources and conservative unread candidates before the shared inbox rule. - Skip recursive privacy walks for standalone open tasks without a project. Keep the recursive rule for all restricted or inherited shapes and all delegated principals. - Add request-count, cancellation, reparenting, read-receipt, inbox-parity and privacy regressions. Include a disposable PostgreSQL benchmark. - Update product and API documentation. Retain the production Storybook states. ## Verification - Published head: `4148861437c1b6b635479ad92509dbc71155a232`. Its tree matches local `04be9d66d481d9bcd1c4d6158acf063a1e65ac7e`: `a23533e93d3d155918f0254d296a7fd2f0c7c12f`. - Rebased on master `ca23330c69b1f2b25bdadfcbda68317db97deaef`. Remote publication preserves prior feature commits. - The request regression failed before the fix and passes after it. An unrelated event now sends zero child requests. Current servers also avoid the unrelated summary request. Old-server events keep one summary fallback. - A relevant update after Show less reads one page. Reparenting refreshes two affected groups. Offscreen reads stay at zero and resume once on return. Active and queued cancellation tests pass. - The same synthetic authenticated HTTP workload with 50,000 read children improved from 767 ms to 92.84 ms median. Five warm samples per case. The all-unread case improved from 463.85 to 351.65 ms. Untouched children improved from 208.01 to 131.09 ms. The old-receipt case improved from 799.70 to 245.80 ms. These local measurements are not a production capacity guarantee. - Focused PostgreSQL routes, inbox-parity, privacy and WebSocket tests passed: 38 tests. - All 325 affected UI tests passed. All 38 focused PostgreSQL, privacy and WebSocket tests passed. UI typecheck and low-memory server declaration checks passed. - All 41 Chromium Storybook checks passed. They cover 20 states in both themes and keyboard, folding, paging and unread behavior. They use production components with HTTP fixtures. - Both new review findings failed as regressions before their fixes and pass afterward. Explicit child reads include waiting external conversations. Deleting a child publishes a separate parent-authorized update without deleted-child content. Both review threads are resolved. - `pnpm verify --explain` returned exit 2 because product documentation requires broader CI. [The prior CI run](https://github.com/paperclipai/paperclip/actions/runs/43711489306) found type errors in the new test fixtures. The same errors failed locally. The fixture fix passes `node server/scripts/typecheck-layers.mjs --layer test-tree-0` and all 21 issue summary route tests. Production code is unchanged. That CI run then found a live-delivery read regression. The existing test failed locally with eight statements against its six-statement budget. A new PostgreSQL regression also found three parent lookups for two same-batch calls. Run delivery now reuses parent metadata from its narrow run-binding query. Fallback lookups resolve to one promise. Both regressions pass without increasing the budgets. All 74 targeted route, WebSocket and privacy tests pass. The affected below, above and test-tree-0 typecheck layers pass. [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/43715541480/attempts/2) passes on the published head. All 62 check runs are terminal and clean: 57 succeeded, two were neutral and three were skipped. Typecheck, build, policy, all general and serialized server suites, E2E and the [final verify gate](https://github.com/paperclipai/paperclip/actions/runs/43715541480/job/132829783231) pass. The separate security status passes. Attempt 1's server shard 7/12 reached its 20-minute job limit without a test verdict. One retry of that job passed without a source change. [Greptile completed successfully on this head](https://github.com/paperclipai/paperclip/runs/132816163906) with a [5/5 score](https://github.com/paperclipai/paperclip/pull/21649#issuecomment-6883916606). All five review threads are resolved. The final CI gate is complete. - A full local Storybook build exceeded available memory while server typechecking ran. Browser verification uses the current-source development server. Rebuilding stale Runner and SDK declarations resolved the initial server check errors. The first browser attempt lacked Chromium; all checks passed after installation. - Reproduce the database measurements with `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-sidebar-subtasks.ts`. - Extra local shard diagnosis ran 164 suites. Under this shell's `0077` creation mask, 157 suites passed and three had fixture permission or timing failures. Rerunning those three with `0022` resolved the permission failures. The remaining ten cases passed with one worker: two workspace cases and eight archive cases. The eight archive cases also passed on the pre-performance-fix revision, whose archive source is identical. The broad local shard is not claimed as one passing execution. The same-head CI retry supplies complete final CI coverage. - Nothing is merged or deployed. ## Risks - Exact counts still grow with the readable hierarchy and comment history. The change reduces repeated work; it does not cap or approximate counts. - A relevant summary batch can contain several recent parents. Older servers retain a broader summary fallback until upgraded. - Live delivery adds bounded, permission-checked parent lookups. Short delivery-window reuse limits repeated reads. - The authorization fast path applies only to open tasks with no privacy parent and no project. Restricted cases retain the shared predicate. - No migration is required. ## Model Used OpenAI Codex, based on GPT-6. Used code execution, native GitHub tools, PostgreSQL and Chromium verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked a public issue or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
114 lines
9.4 KiB
TypeScript
114 lines
9.4 KiB
TypeScript
import assert from 'node:assert/strict';
|
|
import { randomUUID } from 'node:crypto';
|
|
import { mkdir, writeFile } from 'node:fs/promises';
|
|
import { resolve } from 'node:path';
|
|
import { pathToFileURL } from 'node:url';
|
|
import { performance } from 'node:perf_hooks';
|
|
import express from 'express';
|
|
import { drizzle } from 'drizzle-orm/postgres-js';
|
|
import { createDb, startEmbeddedPostgresTestDatabase } from '../../packages/db/src/index.ts';
|
|
import { issueRoutes } from '../../server/src/routes/issues.ts';
|
|
import { errorHandler } from '../../server/src/middleware/index.ts';
|
|
import { seedIssueReadVisibilityFixture } from '../../server/src/__tests__/helpers/issue-read-visibility-fixture.ts';
|
|
import { listIssueNavigationState } from '../../server/src/services/issue-summaries.ts';
|
|
import { issueReadSqlCondition } from '../../server/src/services/authorization.ts';
|
|
|
|
// Run from the server package: pnpm --filter @paperclipai/server exec tsx scripts/benchmark-sidebar-subtasks.ts
|
|
// All data is synthetic. The temporary PostgreSQL cluster is removed in finally.
|
|
const directory = resolve(process.env.PAPERCLIP_BENCHMARK_OUTPUT ?? '../test-results/sidebar-subtasks-benchmark');
|
|
await mkdir(directory, { recursive: true });
|
|
const out = pathToFileURL(directory + '/');
|
|
const report: any = { synthetic: true, node: process.version, pool: 10, samples: 5, cases: [] };
|
|
const keeper = setInterval(() => {}, 1000);
|
|
const fixture = await startEmbeddedPostgresTestDatabase('sidebar-subtasks-perf-');
|
|
const db = createDb(fixture.connectionString, { maxConnections: 10 });
|
|
const client = db.$client;
|
|
const statements: Array<{query: string; params: unknown[]}> = [];
|
|
let record = false;
|
|
const counted = drizzle(client, { schema: db._.fullSchema, logger: { logQuery(query, params) { if (record) statements.push({query, params}); } } }) as typeof db;
|
|
let server: ReturnType<typeof express>['listen'] extends (...a:any[]) => infer R ? R : never;
|
|
const stats = (xs: number[]) => { const s = [...xs].sort((a,b)=>a-b); return {medianMs:+s[Math.floor(s.length/2)]!.toFixed(2), maxMs:+s.at(-1)!.toFixed(2), samplesMs:xs.map(x=>+x.toFixed(2))}; };
|
|
try {
|
|
process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = 'enforce';
|
|
const f = await seedIssueReadVisibilityFixture(db, {publicCount: 50});
|
|
const actor = f.actors['cloud tenant owner'];
|
|
const userId = (actor as any).userId!;
|
|
const parents = f.publicIds;
|
|
// Make each measured parent independent and participated-in by this viewer.
|
|
await client.unsafe('update issues set parent_id=null, created_by_user_id=$2 where company_id=$1 and id=any($3::uuid[])',[f.companyId,userId,parents]);
|
|
await client.unsafe(`create table perf_parents as select id from issues where id=any($1::uuid[])`,[parents]);
|
|
await client.unsafe(`insert into issues (company_id,title,status,created_by_user_id) select $1,'Unrelated history '||n,'done',$2 from generate_series(1,20000)n`,[f.companyId,f.strangerUserId]);
|
|
await client.unsafe(`insert into activity_log(company_id,actor_type,actor_id,action,entity_type,entity_id,created_at) select $1,'system','synthetic-agent','issue.updated','issue',$2,now() from generate_series(1,100000)n`,[f.companyId,parents[0]]);
|
|
const connections=await Promise.all(Array.from({length:10},()=>client.reserve()));
|
|
for(const connection of connections){await connection.unsafe("set statement_timeout='20s'");connection.release();}
|
|
const app=express();
|
|
app.use((req,_res,next)=>{req.actor=actor as any;next();});
|
|
app.use('/api',issueRoutes(counted,{} as never)); app.use(errorHandler);
|
|
server=app.listen(0,'127.0.0.1'); await new Promise<void>(r=>server.once('listening',r));
|
|
const origin=`http://127.0.0.1:${(server.address() as any).port}`;
|
|
const summaryPath=`/api/companies/${f.companyId}/issues/summaries?ids=${parents.join(',')}`;
|
|
async function read(path:string) {const r=await fetch(origin+path);const body=await r.json();assert.equal(r.status,200,JSON.stringify(body));return body;}
|
|
async function measure(path:string) {await read(path);const times=[];for(let n=0;n<5;n++){const t=performance.now();await read(path);times.push(performance.now()-t);}record=true;statements.length=0;const body=await read(path);record=false;return {time:stats(times),statements:statements.length,sqlBytes:statements.reduce((n,x)=>n+Buffer.byteLength(x.query),0),responseBytes:Buffer.byteLength(JSON.stringify(body)),body};}
|
|
report.postgres=(await client.unsafe('select version()'))[0]!.version;
|
|
report.fixture={parents:50,noiseIssues:20000,noiseActivity:100000,privacy:'enforce',actor:'cloud tenant owner',commentBytes:128};
|
|
let previous=0;
|
|
for(const perParent of [0,10,100,1000]) {
|
|
if(perParent>previous) {
|
|
await client.unsafe(`insert into issues(company_id,parent_id,title,status,created_by_user_id,created_at,updated_at)
|
|
select $1,p.id,'Synthetic child '||n,'todo',$2,'2026-01-01','2026-01-01' from perf_parents p cross join generate_series($3::int,$4::int)n`,[f.companyId,userId,previous+1,perParent]);
|
|
// One external comment per child, deliberately all read: the unread test
|
|
// must check its pending/withdrawal branches rather than short-circuit.
|
|
await client.unsafe(`insert into issue_comments(company_id,issue_id,author_user_id,body,created_at)
|
|
select $1,i.id,$2,repeat('x',128),'2026-01-02' from issues i where i.company_id=$1 and i.parent_id=any($3::uuid[]) and not exists(select 1 from issue_comments c where c.issue_id=i.id)`,[f.companyId,f.strangerUserId,parents]);
|
|
await client.unsafe(`insert into issue_read_states(company_id,issue_id,user_id,last_read_at)
|
|
select $1,i.id,$2,'2026-01-03' from issues i where i.company_id=$1 and i.parent_id=any($3::uuid[]) on conflict(company_id,issue_id,user_id) do nothing`,[f.companyId,userId,parents]);
|
|
}
|
|
await client.unsafe('vacuum analyze');
|
|
const base=await measure(summaryPath);
|
|
const nav=await measure(summaryPath+'&navigation=true');
|
|
for(const row of nav.body) {assert.equal(row.subtaskCount,perParent);assert.equal(row.unreadSubtaskCount,0);}
|
|
const plans: any[]=[];
|
|
for(const [index,s] of statements.entries()) {
|
|
const plan=await client.unsafe('explain (analyze,buffers,format json) '+s.query,s.params as any[]);
|
|
plans.push({index,sql:s.query,plan:plan[0]['QUERY PLAN']});
|
|
}
|
|
await writeFile(new URL(`plans-${perParent}.json`,out),JSON.stringify(plans,null,2));
|
|
const childPath=`/api/companies/${f.companyId}/issues?parentId=${parents[0]}&view=rows&limit=10&sortField=id&sortDir=asc&includeRoutineExecutions=true&includeReadState=true`;
|
|
const page=perParent?await measure(childPath):null;
|
|
const item={childrenPerParent:perParent,totalCountedChildren:perParent*50,baseline:{...base,body:undefined},navigation:{...nav,body:undefined},firstChildPage:page?{...page,body:undefined,rows:page.body.length}:null};
|
|
report.cases.push(item);console.log(JSON.stringify(item));
|
|
await writeFile(new URL('database-results.json',out),JSON.stringify(report,null,2));
|
|
previous=perParent;
|
|
}
|
|
// Compare the same large hierarchy when children are unread and when they
|
|
// have never been touched by this person.
|
|
for(const state of ['unread','untouched']) {
|
|
if(state==='unread') await client.unsafe(`update issue_read_states set last_read_at='2026-01-01' where company_id=$1`,[f.companyId]);
|
|
else await client.unsafe(`update issues set created_by_user_id=$2 where company_id=$1 and parent_id=any($3::uuid[])`,[f.companyId,f.strangerUserId,parents]);
|
|
await client.unsafe('vacuum analyze');
|
|
const m=await measure(summaryPath+'&navigation=true');
|
|
assert.ok(m.body.every((x:any)=>x.unreadSubtaskCount===(state==='unread'?1000:0)));
|
|
const plans=[];
|
|
for(const [index,s] of statements.entries()) {
|
|
const plan=await client.unsafe('explain (analyze,buffers,format json) '+s.query,s.params as any[]);
|
|
plans.push({index,sql:s.query,plan:plan[0]['QUERY PLAN']});
|
|
}
|
|
await writeFile(new URL(`plans-${state}.json`,out),JSON.stringify(plans,null,2));
|
|
const item={state,childrenPerParent:1000,...m,body:undefined};report.cases.push(item);console.log(JSON.stringify(item));
|
|
}
|
|
// A mature viewer's old withdrawal receipts exercise the indexed JSON lookup.
|
|
await client.unsafe(`insert into activity_log(company_id,actor_type,actor_id,action,entity_type,entity_id,details,created_at)
|
|
select $1,'user',$2,'issue.thread_interaction_withdrawn','issue',$3,jsonb_build_object('handledForUserId',$2::text,'receipt',jsonb_build_object('outcome','withdrawn','issueId',$3::text)),'2025-12-01' from generate_series(1,5000)n`,[f.companyId,userId,parents[0]]);
|
|
await client.unsafe(`update issues set created_by_user_id=$2 where company_id=$1 and parent_id=any($3::uuid[])`,[f.companyId,userId,parents]);
|
|
await client.unsafe(`update issue_read_states set last_read_at='2026-01-03' where company_id=$1`,[f.companyId]);
|
|
await client.unsafe('vacuum analyze');
|
|
const receipts=await measure(summaryPath+'&navigation=true');
|
|
report.receiptStress={oldViewerReceipts:5000,...receipts,body:undefined};console.log(JSON.stringify(report.receiptStress));
|
|
const readCondition=await issueReadSqlCondition(db,actor);
|
|
const q=()=>listIssueNavigationState(db,f.companyId,parents,readCondition,userId);
|
|
const start=performance.now();await Promise.all(Array.from({length:3},q));report.concurrentThreeServiceRequestsMs=+(performance.now()-start).toFixed(2);
|
|
await writeFile(new URL('database-results.json',out),JSON.stringify(report,null,2));
|
|
} finally {
|
|
server?.close(); await client.end({timeout:0}); await fixture.cleanup(); clearInterval(keeper);
|
|
}
|