From b993702ae366601a3f970aefafa9e14206e4bdd0 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 11:50:24 -0600 Subject: [PATCH] perf(ui): read the compact task index on the dashboard and refresh it once per window (#21413) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI keeps company lists fresh from live task activity events > - The dashboard, the activity feed and board chat read the full company task list, and every burst of task activity refetches it > - One full list request costs the server 60 statements and 61-90 MB of allocation for 500 tasks, and the staging server is saturated by allocation and garbage collection > - This pull request moves those three views to the compact task index under its own key, and task activity refreshes that key at most once per 30 seconds > - The benefit is about one tenth of the server allocation per open dashboard tab while agents work, with the same rows on screen ## Linked Issues or Issue Description **What existing behavior does this improve?** The dashboard, the activity feed and board chat load the full company task list. Every burst of task activity refetches that list. **Subsystem affected** Board UI: live updates, the dashboard, the activity feed and board chat. **Current behavior** The three views read the full company task list (`GET /api/companies/:id/issues`, no view) with the key `["issues", companyId]`. `LiveUpdatesProvider` invalidates that key on every task activity event. Each open tab refetches the list at most every 5 seconds while agents work. One request costs the server 60 statements and 61-90 MB of allocation for 500 tasks (local measurement). The staging server allocates 57.6 MB per second and is saturated by garbage collection. **Proposed behavior** The views read only the fields they show, from the compact task index. Task activity from other people refreshes the index at most once per window. The viewer's own task changes refresh it at once. **Reason and benefit** Less server allocation and fewer statements per open tab, with the same rows on screen. **Breaking changes** None. No API, schema or server change. ## What Changed - The dashboard, the activity feed and board chat read `view=index` under a new key `issues.listIndex(companyId)`. - Task activity from other people refreshes that key at most once per 30 seconds (`TASK_INDEX_REFRESH_WINDOW_MS`). The viewer's own task activity refreshes it at once. The first event after a quiet window refetches at once. Later events in the window share one trailing refetch. Nothing from the event goes into the cache. An unwatched index is marked stale and refetches on the next mount. The user's own task changes still invalidate the whole `["issues", companyId]` prefix immediately. - The dashboard takes its ten recent tasks from the index, sorted by updatedAt as before. It reads the status glyph fields for those ten tasks from `issues/summaries`, because the index omits displayStatus and blockerAttention. - Toast and route lookups that read the full list cache also read the index cache. - `server/scripts/query-cost/` is the per-caller query cost harness used for the measurements below. It records the caller of each query at construction, the bytes allocated to build it, the rows it returns and the statements sent. ## Verification - `pnpm --filter @paperclipai/ui exec vitest run src/context/LiveUpdatesProvider src/pages/Dashboard.test.ts`: 261 pass. - Red/green: `LiveUpdatesProvider.task-index.test.ts` fails 6 of its first 10 cases against the provider on master and passes with this change. The tests cover a burst of events (4 index fetches in 90 seconds), the first event after a quiet window, a new task, a deleted task, a hidden task, a status/title/priority change, an event for a task outside the cache, an event for a task the viewer cannot read (nothing from the event goes into the cache), the viewer's own activity (refreshes at once), a task change made in this tab (refreshes at once through the company list invalidation), the trailing refresh (the last state shows when the window ends after activity stops), an unwatched index, and the live-event batcher path. - Other readers of `["issues", companyId]` (My Tasks, the task properties picker, the command palette, task detail cache writes, filtered and board-column lists) are not changed. The broad invalidation excludes only the new index key, so they refresh as before. A hidden tab keeps today's behavior: the live-event batcher still turns its refetches into stale marks. - Field parity (local, 1,500 tasks): the index returns the same 500 ids as the full list, with 0 differences in id, identifier, title, status, priority, createdAt, updatedAt and assigneeAgentId. It omits displayStatus and blockerAttention on all 500 rows, so the recent rows take those fields from summaries. - Real browser (headless Chromium, Playwright) against a local server from this branch, started with `test-drive --data-dir `, which sets `PAPERCLIP_HOME` and `PAPERCLIP_CONFIG` to that directory: - Dashboard: the recent tasks and the "Tasks by Status" chart show the new tasks. The page sent 0 full-list requests, 2 index requests and 1 summaries request. - A task rename by this board user showed on the dashboard after 0.8 seconds, with 1 index refetch. - A second tab on the dashboard showed the same tasks. - Board chat loaded with 0 full-list requests and 3 index requests. Its activity feed showed the task names. - Not checked in the browser: activity from another actor. An agent key cannot edit a task without a heartbeat run context, so the window for other people's activity is covered by the unit tests only. - Nothing visual changed. The views keep their render code. Only their data source changed. - Cost per request (local, embedded PostgreSQL, Node 26, uninstrumented pass, 5 requests each), measured: `TSX_DISABLE_CACHE=1 UNITS=board ROUTES="issues list,issues list view=index,issues/summaries (10 ids)" REPEAT=5 NODE_OPTIONS=--expose-gc pnpm --filter @paperclipai/server exec tsx scripts/query-cost/drive.ts` | Request | Statements | Allocated | | --- | --- | --- | | full list | 60 | 61-90 MB | | view=index | 12 | 22-37 MB | | summaries, 10 ids | 23 | 2.8 MB | The ranges show the difference between runs. - Estimate, not measured: per open dashboard tab with continuous task activity, before about 13 MB/s (one full list per 5 seconds), after about 1.3 MB/s (one index per 30 seconds and one 10-id summaries request per 5 seconds). At the day-1 staging route rate (0.116 full lists per second) that is about 6.5 MB/s less. The real rate of task activity events on staging is not measured. ## Risks - Freshness: the dashboard's task charts and recent-task order, the activity feed's task names and board chat's task lookup can lag task activity from other people by up to 30 seconds. The status glyphs of the ten recent tasks stay on the 5-second lane. The viewer's own task changes refresh at once. The window is one constant, `TASK_INDEX_REFRESH_WINDOW_MS` in `LiveUpdatesProvider.tsx`, so a different value is a one-line change. - The index returns the same rows as the full list, so the charts count the same tasks. - The recent-task rows have no task description. The rows do not show it. - My Tasks, the task properties picker and the command palette still read the full list. They are not in this change. ## Model Used - Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no documentation describes this behavior) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- server/scripts/query-cost/drive.ts | 273 ++++++++++++++++++ server/scripts/query-cost/instrument.ts | 216 ++++++++++++++ ui/src/components/ActivityFeed.tsx | 4 +- .../LiveUpdatesProvider.task-index.test.ts | 177 ++++++++++++ ui/src/context/LiveUpdatesProvider.tsx | 49 +++- ui/src/lib/queryKeys.ts | 1 + ui/src/pages/BoardChat.test.tsx | 4 +- ui/src/pages/BoardChat.tsx | 4 +- ui/src/pages/Dashboard.test.ts | 37 ++- ui/src/pages/Dashboard.tsx | 51 +++- 10 files changed, 795 insertions(+), 21 deletions(-) create mode 100644 server/scripts/query-cost/drive.ts create mode 100644 server/scripts/query-cost/instrument.ts create mode 100644 ui/src/context/LiveUpdatesProvider.task-index.test.ts diff --git a/server/scripts/query-cost/drive.ts b/server/scripts/query-cost/drive.ts new file mode 100644 index 00000000000..2ed385a90ac --- /dev/null +++ b/server/scripts/query-cost/drive.ts @@ -0,0 +1,273 @@ +import "./instrument.js"; +import assert from "node:assert/strict"; +import { randomBytes, randomUUID } from "node:crypto"; +import type { AddressInfo } from "node:net"; +import { writeFileSync } from "node:fs"; +import express from "express"; +import { completionContracts, createDb, heartbeatRuns, issues, startEmbeddedPostgresTestDatabase } from "../../../packages/db/src/index.js"; +import { countStatements, diff, format, setEnabled, snapshot } from "./instrument.js"; +import { seedReviewAttentionFixture } from "../../src/__tests__/helpers/review-attention-fixture.js"; +import { seedAttentionSummaryVolume } from "../../src/__tests__/helpers/attention-summary-fixture.js"; + +process.env.PAPERCLIP_SECRETS_MASTER_KEY ??= randomBytes(32).toString("base64"); +const only = (process.env.UNITS ?? "board,event").split(","); +const TASKS = Number(process.env.TASKS ?? 1500), AGENTS = Number(process.env.AGENTS ?? 40), REPEAT = Number(process.env.REPEAT ?? 3); +const keepAlive = setInterval(() => {}, 1_000); +const temporary = await startEmbeddedPostgresTestDatabase("query-cost-"); +const db = createDb(temporary.connectionString); +countStatements(db as never); +const report: string[] = []; +const out = process.env.OUT ?? "query-cost.md"; + +async function measure(label: string, units: number, action: () => Promise) { + await action(); + (globalThis as { gc?: () => void }).gc?.(); + const plain = snapshot(); + for (let i = 0; i < units; i++) await action(); + const uninstrumented = diff(plain); + setEnabled(true); + const before = snapshot(); + for (let i = 0; i < units; i++) await action(); + const result = diff(before); + setEnabled(false); + const text = `uninstrumented: statements ${(uninstrumented.statements / units).toFixed(1)}, allocated ${(uninstrumented.allocated / 1024 / units).toFixed(1)} KB, wall ${(uninstrumented.wallMs / units).toFixed(1)} ms per unit\n` + format(label, units, result); + report.push(text); + console.log(text + "\n"); + writeFileSync(out, report.join("\n\n") + "\n"); +} + +try { + const fixture = await seedReviewAttentionFixture(db, { tasks: TASKS, agents: AGENTS, prefix: "SYN" }); + await seedAttentionSummaryVolume(db, fixture); + const { companyId, userId } = fixture; + const agentId = fixture.agentIds[0]!; + + if (only.includes("calibrate")) { + const { sql } = await import("drizzle-orm"); + const v8 = await import("node:v8"); + const alloc = () => (v8.getHeapStatistics() as unknown as { total_allocated_bytes: number }).total_allocated_bytes; + const lines = ["## read calibration (prepared, build excluded): KB allocated per execution", "| rows | fields | bytes/value | KB/exec | wire KB | bytes alloc per wire byte |", "| --- | --- | --- | --- | --- | --- |"]; + for (const [rows, fields, width] of [[1, 1, 8], [1, 20, 8], [1, 80, 8], [50, 20, 8], [50, 20, 200], [500, 10, 40], [500, 3, 2000]] as const) { + const fieldsObj = Object.fromEntries(Array.from({ length: fields }, (_, i) => [`c${i}`, sql`repeat('x', ${sql.raw(String(width))})`.as(`c${i}`)])); + const prepared = db.select(fieldsObj).from(sql`generate_series(1, ${sql.raw(String(rows))})`).prepare(`cal_${rows}_${fields}_${width}`); + for (let i = 0; i < 5; i++) await prepared.execute(); + const start = alloc(); + for (let i = 0; i < 30; i++) await prepared.execute(); + const kb = (alloc() - start) / 30 / 1024, wire = rows * fields * (width + 4) / 1024; + lines.push(`| ${rows} | ${fields} | ${width} | ${kb.toFixed(1)} | ${wire.toFixed(1)} | ${(kb / wire).toFixed(1)} |`); + } + report.push(lines.join("\n")); console.log(lines.join("\n")); + } + + if (only.includes("board")) { + const { attentionRoutes } = await import("../../src/routes/attention.js"); + const { issueRoutes } = await import("../../src/routes/issues.js"); + const { agentRoutes } = await import("../../src/routes/agents.js"); + const { errorHandler } = await import("../../src/middleware/index.js"); + const runIds: string[] = []; + for (const [index, issueId] of fixture.taskIds.slice(0, 12).entries()) { + const id = randomUUID(); + runIds.push(id); + await db.insert(heartbeatRuns).values({ id, companyId, agentId: fixture.agentIds[index % fixture.agentIds.length]!, status: "running", + issueId, contextSnapshot: { issueId }, startedAt: new Date() }); + } + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { + (req as unknown as { actor: unknown }).actor = { type: "board", source: "session", userId, companyIds: [companyId], isInstanceAdmin: false }; + next(); + }); + app.use("/api", attentionRoutes(db)); + app.use("/api", issueRoutes(db, {} as never)); + app.use("/api", agentRoutes(db, {} as never)); + app.use(errorHandler); + const server = app.listen(0, "127.0.0.1"); + await new Promise(resolve => server.once("listening", resolve)); + const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}/api`; + const get = async (route: string) => { + const response = await fetch(`${base}${route}`); + const body = await response.text(); + assert.equal(response.status, 200, `${route}: ${body.slice(0, 300)}`); + return body.length; + }; + const ids = fixture.taskIds.slice(0, 50).join(","); + const routes: Record = { + "attention/summary": `/companies/${companyId}/attention/summary`, + "issues list": `/companies/${companyId}/issues`, + "issues list view=index": `/companies/${companyId}/issues?view=index`, + "issues list view=rows": `/companies/${companyId}/issues?view=rows`, + "issues/summaries (50 ids)": `/companies/${companyId}/issues/summaries?ids=${ids}`, + "issues/summaries (10 ids)": `/companies/${companyId}/issues/summaries?ids=${fixture.taskIds.slice(0, 10).join(",")}`, + "live-runs": `/companies/${companyId}/live-runs`, + "agents list": `/companies/${companyId}/agents`, + "issue detail": `/issues/${fixture.taskIds[3]}`, + "heartbeat run detail": `/heartbeat-runs/${runIds[0]}`, + }; + if (only.includes("fields")) { + const read = async (route: string) => JSON.parse(await (await fetch(`${base}${route}`)).text()) as Array>; + const full = await read(`/companies/${companyId}/issues`); + const index = await read(`/companies/${companyId}/issues?view=index`); + const rows = await read(`/companies/${companyId}/issues?view=rows&sortField=updated&sortDir=desc&limit=10`); + const fields = ["id", "identifier", "title", "status", "priority", "createdAt", "updatedAt", "assigneeAgentId", "displayStatus", "externalConversationState", "blockerAttention"]; + const byId = new Map(full.map(row => [row.id, row])); + const diffs: Record = {}; + for (const [label, set] of [["index", index], ["rows10", rows]] as const) { + for (const row of set) for (const field of fields) { + const a = JSON.stringify(byId.get(row.id)?.[field] ?? null), b = JSON.stringify(row[field] ?? null); + if (a !== b) diffs[`${label}.${field}`] = (diffs[`${label}.${field}`] ?? 0) + 1; + } + } + const recentFull = [...full].sort((a, b) => Date.parse(String(b.updatedAt)) - Date.parse(String(a.updatedAt))).slice(0, 10).map(row => row.id); + console.log(JSON.stringify({ full: full.length, index: index.length, rows: rows.length, + sameIds: JSON.stringify(full.map(r => r.id).sort()) === JSON.stringify(index.map(r => r.id).sort()), + sameRecent: JSON.stringify(recentFull) === JSON.stringify(rows.map(r => r.id)), diffs, + fullHasDisplayStatus: full.filter(r => r.displayStatus !== undefined).length, fullBlocker: full.filter(r => r.blockerAttention).length })); + } + for (const [name, route] of Object.entries(routes).filter(([name]) => !process.env.ROUTES || process.env.ROUTES.split(",").includes(name))) { + try { + const bytes = await get(route); + await measure(`GET ${name} (body ${Math.round(bytes / 1024)} KB)`, REPEAT, async () => { await get(route); }); + } catch (error) { console.log(`skip ${name}: ${(error as Error).message.slice(0, 300)}`); } + } + server.close(); + } + + if (only.includes("recovery")) { + const { assignedWorkSweep } = await import("../../src/services/recovery/assigned-work-sweep.js"); + const sweep = assignedWorkSweep(db, { enqueueWakeup: async () => null, proveOwner: async () => ({ kind: "alive" }) as never }); + const suspects = await sweep.listSuspects({ companyId }, null, 100); + console.log(`suspects ${suspects.length}`); + await measure("assigned-work sweep load (listSuspects page of 100)", 5, async () => { await sweep.listSuspects({ companyId }, null, 100); }); + await measure("assigned-work sweep check of one task (checkIssue)", 25, async () => { + for (const suspect of suspects.slice(0, 1)) await sweep.checkIssue({ companyId, issueId: suspect.id }); + }); + const { loadNativeChatProgressScopes, reconcileNativeChatProgress } = await import("../../src/services/chat-native-progress.js"); + await measure("chat native progress load + check", 5, async () => { + await loadNativeChatProgressScopes(db, { companyId } as never); + await reconcileNativeChatProgress(db, { scope: { companyId, issueId: fixture.taskIds[0]! }, limit: 100 }); + }); + } + + if (only.includes("agent")) { + process.env.PAPERCLIP_AGENT_JWT_SECRET = randomBytes(32).toString("hex"); + const { createLocalAgentJwt } = await import("../../src/agent-auth-jwt.js"); + const { actorMiddleware } = await import("../../src/middleware/auth.js"); + const { issueRoutes } = await import("../../src/routes/issues.js"); + const { errorHandler } = await import("../../src/middleware/index.js"); + const runId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", issueId: fixture.taskIds[3]!, + contextSnapshot: { issueId: fixture.taskIds[3]! }, startedAt: new Date() }); + const token = createLocalAgentJwt(agentId, companyId, "codex_local", runId); + assert.ok(token, "agent token"); + const app = express(); + app.use(express.json()); + app.use(actorMiddleware(db, { deploymentMode: "authenticated", localAgentProcessBoundary: null })); + app.get("/api/whoami", (req, res) => { res.json({ type: (req as unknown as { actor: { type: string } }).actor.type }); }); + app.use("/api", issueRoutes(db, {} as never)); + app.use(errorHandler); + const server = app.listen(0, "127.0.0.1"); + await new Promise(resolve => server.once("listening", resolve)); + const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}/api`; + const get = async (route: string) => { + const response = await fetch(`${base}${route}`, { headers: { authorization: `Bearer ${token}`, "x-paperclip-run-id": runId } }); + const body = await response.text(); + assert.equal(response.status, 200, `${route}: ${body.slice(0, 300)}`); + return body; + }; + assert.match(await get("/whoami"), /agent/); + await measure("agent request authentication only (actorMiddleware, agent JWT)", 20, async () => { await get("/whoami"); }); + await measure("agent GET issues/:id (auth + route)", 5, async () => { await get(`/issues/${fixture.taskIds[3]}`); }); + server.close(); + } + + if (only.includes("legacy")) { + const { createAgentLifecycle, configureAgentLifecycle } = await import("../../src/services/agent-lifecycle.js"); + const { heartbeatService } = await import("../../src/services/heartbeat.js"); + const { createDeliveryWorkCoordinator } = await import("../../src/services/delivery-work-coordinator.js"); + const { enableDatabaseWorkNotifications } = await import("../../../packages/db/src/work-signals.js"); + const { eq } = await import("drizzle-orm"); + enableDatabaseWorkNotifications(db); + const lines = Number(process.env.LINES ?? 20), gapMs = Number(process.env.GAP_MS ?? 50); + const legacyAgentId = randomUUID(), legacyCompanyId = randomUUID(); + const { companies } = await import("../../../packages/db/src/index.js"); + await db.insert(companies).values({ id: legacyCompanyId, name: "Legacy", issuePrefix: `L${legacyCompanyId.slice(0, 6)}`, requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: "legacy-user" }); + await createAgentLifecycle(db).requestHire(legacyCompanyId, { id: legacyAgentId, name: "Legacy process", role: "engineer", status: "idle", + adapterType: "process", adapterConfig: { command: process.execPath, + args: ["-e", `let i=0;const t=setInterval(()=>{console.log("output line "+(++i)+" ".padEnd(80,"x"));if(i>=${lines}){clearInterval(t);}},${gapMs});`], + cwd: process.cwd() }, + runtimeConfig: { heartbeat: { enabled: true, intervalSec: 0, wakeOnDemand: true } }, permissions: {} }, { runnerResolved: true }); + const lifecycleWorker = configureAgentLifecycle(db, { requiredPluginIds: async () => [], runPlugin: async () => "complete", runHost: async () => "complete" }); + for (let i = 0; i < 4; i++) await lifecycleWorker.process(legacyAgentId); + await lifecycleWorker.stop(); + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => console.log(`coordinator error ${String(error).slice(0, 200)}`) }); + const heartbeat = heartbeatService(db, { runtimeEnv: { ...process.env, PAPERCLIP_IN_WORKTREE: undefined } }); + const dispatch = heartbeat.registerRunDispatchWork(coordinator, () => true); + await dispatch.ready; + const runOnce = async () => { + const run = await heartbeat.wakeup(legacyAgentId, { source: "on_demand", reason: "manual", requestedByActorType: "system" }); + if (!run) { + const { agents, agentWakeupRequests } = await import("../../../packages/db/src/index.js"); + console.log(JSON.stringify({ agent: (await db.select({ status: agents.status, holds: agents.lifecycleHolds, state: agents.lifecycleState }).from(agents).where(eq(agents.id, legacyAgentId)))[0], + wakes: await db.select({ status: agentWakeupRequests.status, reason: agentWakeupRequests.reason, error: agentWakeupRequests.error }).from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, legacyAgentId)) })); + } + assert.ok(run, "wake accepted"); + for (let i = 0; i < 600; i++) { + const [row] = await db.select({ status: heartbeatRuns.status }).from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + if (row && !["queued", "running"].includes(row.status)) { + assert.equal(row.status, "succeeded", "a measured legacy run must succeed"); + break; + } + assert.ok(i < 599, "a measured legacy run must finish within 60 seconds"); + await new Promise(resolve => setTimeout(resolve, 100)); + } + await heartbeat.drainActiveRunExecutions(); + }; + await measure(`legacy process run lifecycle (wake, queue, start, ${lines} output lines ${gapMs} ms apart, finish), includes the harness's 100 ms status poll`, 3, runOnce); + await dispatch.stop(); + await coordinator.stop(); + } + + if (only.includes("event")) { + const { PaperclipControlPlanePort } = await import("../../src/services/native-runtime/paperclip-control-plane-port.js"); + const [issue] = await db.insert(issues).values({ companyId, title: "Streaming", status: "in_progress", assigneeAgentId: agentId }).returning(); + const contractSha = "query-cost"; + const [contract] = await db.insert(completionContracts).values({ companyId, issueId: issue!.id, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "benchmark", risk: "standard", + completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: { revision: "benchmark", objective: "Benchmark", criteria: [{ id: "objective", requirement: "Append" }] }, + canonicalSha256: contractSha, createdByActorType: "system", createdByActorId: "benchmark" }).returning(); + const runId = randomUUID(), sessionId = randomUUID(), runnerId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", + nativeIssueId: issue!.id, issueId: issue!.id, nativeSessionId: sessionId, runnerInstanceId: runnerId, + completionContractId: contract!.id, completionContractSha256: contractSha, contextSnapshot: { issueId: issue!.id } }); + const identity = { companyId, issueId: issue!.id, runId, agentId, sessionId }; + const port = new PaperclipControlPlanePort(db, { ...identity, completionContractId: contract!.id, + completionContractSha256: contractSha, sourceInstanceId: runnerId, controlPlaneSourceInstanceId: `control-${runId}` }); + await port.openRun({ identity, backendKind: "mock", sourceInstanceId: runnerId } as never); + const text = randomBytes(72).toString("base64").slice(0, 96); + let ordinal = 0; + const next = async () => { + ordinal += 1; + const step = ordinal % 10; + const base = { schema: "paperclip.prp.event.v1" as const, sourceEventId: `${runnerId}:${ordinal}`, sourceSeq: ordinal, + sourceInstanceId: runnerId, sourceKind: "runner" as const, runId, normalizedSessionId: sessionId, + turnId: "turn-1", schemaVersion: 1 as const, priority: 1, emittedAt: new Date().toISOString() }; + const event = step < 7 ? { ...base, itemId: "message-1", eventType: "item.delta", payload: { kind: "agentMessage", text } } + : step === 9 ? { ...base, itemId: `command-${ordinal}`, eventType: "item.completed", + payload: { kind: "commandExecution", item: { type: "command", id: `command-${ordinal}`, output: text } } } + : { ...base, itemId: `tool-${ordinal}`, eventType: step === 7 ? "tool.execution.started" : "tool.execution.completed", + payload: { schema: "paperclip.tool.execution.v1", executionId: `tool-${ordinal - (step === 7 ? 0 : 1)}`, transport: "builtin", + operation: "execute", name: "shell", target: null, namespace: null, readOnly: true, + status: step === 7 ? "running" : "completed", durationMs: step === 7 ? null : 12, exitCode: step === 7 ? null : 0, + progress: null, output: text, outputBytes: text.length, outputTruncated: false, outputDigest: `sha256:${"a".repeat(64)}` } }; + await port.appendEvent(event as never); + }; + await measure("native runner event via PaperclipControlPlanePort.appendEvent", 200, next); + } +} finally { + clearInterval(keepAlive); + await db.$client.end({ timeout: 1 }).catch(() => {}); + await temporary.cleanup(); +} +console.log(`report: ${out}`); +process.exit(0); diff --git a/server/scripts/query-cost/instrument.ts b/server/scripts/query-cost/instrument.ts new file mode 100644 index 00000000000..6cfce921b13 --- /dev/null +++ b/server/scripts/query-cost/instrument.ts @@ -0,0 +1,216 @@ +import v8 from "node:v8"; +import { performance } from "node:perf_hooks"; +import { PgDatabase, PgDeleteBase, PgInsertBase, PgInsertBuilder, PgSelectBase, PgSelectBuilder, PgUpdateBase, PgUpdateBuilder } from "drizzle-orm/pg-core"; +import { PgRelationalQuery, RelationalQueryBuilder } from "drizzle-orm/pg-core/query-builders/query"; + +type Entry = { + key: string; operation: string; callers: string[]; builds: number; executions: number; + sqlBytes: number; buildBytes: number; buildMs: number; readBytes: number; readMs: number; + rows: number; resultBytes: number; fields: number; calibratedBytes?: number; calibratedMs?: number; sqlPrefix?: string; +}; +const CALIBRATED = 3, LOOP = 40; + +const allocated = () => (v8.getHeapStatistics() as unknown as { total_allocated_bytes: number }).total_allocated_bytes; +if (typeof allocated() !== "number") { + throw new Error(`query-cost needs a Node.js whose v8.getHeapStatistics() reports total_allocated_bytes (Node.js 26 or later); this is ${process.version}`); +} +const site = new WeakMap(); +export const entries = new Map(); +export const totals = { statements: 0, control: 0 }; +let enabled = false; +let depth = 0; +export function setEnabled(value: boolean) { enabled = value; } + +const root = process.cwd().replace(/\/server$/, ""); +function callers(boundary: Function): string[] { + const holder: { stack?: string } = {}; + const limit = Error.stackTraceLimit; + Error.stackTraceLimit = 40; + Error.captureStackTrace(holder, boundary); + Error.stackTraceLimit = limit; + const out: string[] = []; + for (const line of (holder.stack ?? "").split("\n").slice(1)) { + if (line.includes("node_modules") || line.includes("query-cost/") || line.includes("node:")) continue; + const match = line.match(/at (?:async )?(?:(\S+) \()?(?:file:\/\/)?([^()]+):(\d+):\d+\)?$/); + if (!match) continue; + const file = match[2]!.replace(root + "/", ""); + if (file.startsWith("packages/db/src/client") || file.includes("lazy-transaction-queries")) continue; + out.push(`${file}:${match[3]}${match[1] ? ` ${match[1]}` : ""}`); + if (out.length === 3) break; + } + return out; +} + +function tag(builder: object, operation: string, boundary: Function) { + if (!enabled || !builder || typeof builder !== "object") return; + const stack = callers(boundary); + site.set(builder, { key: `${operation} ${stack[0] ?? "unknown"}`, operation, callers: stack }); +} + +function entry(info: { key: string; operation: string; callers: string[] }): Entry { + let value = entries.get(info.key); + if (!value) { + value = { ...info, builds: 0, executions: 0, sqlBytes: 0, buildBytes: 0, buildMs: 0, readBytes: 0, readMs: 0, rows: 0, resultBytes: 0, fields: 0 }; + entries.set(info.key, value); + } + return value; +} + +function resultSize(value: unknown): { rows: number; bytes: number } { + const rows = Array.isArray(value) ? value : (value && typeof value === "object" && Array.isArray((value as { rows?: unknown }).rows)) ? (value as { rows: unknown[] }).rows : []; + let bytes = 0; + try { bytes = Buffer.byteLength(JSON.stringify(rows, (_k, v) => typeof v === "bigint" ? String(v) : v) ?? ""); } catch {} + return { rows: rows.length, bytes }; +} + +function record(e: Entry, value: unknown) { + const size = resultSize(value); + const rows = Array.isArray(value) ? value : []; + const width = e.fields || (rows[0] && typeof rows[0] === "object" ? Object.keys(rows[0] as object).length : 0); + e.rows += size.rows; + e.resultBytes += size.bytes; + e.readBytes += 40 * 1024 + width * 1024 + size.rows * width * 256 + size.bytes * 1.3; +} + +function wrapPrepared(prepared: any, info: { key: string; operation: string; callers: string[] }) { + if (!prepared || typeof prepared.execute !== "function" || prepared.__qcost) return; + prepared.__qcost = true; + const original = prepared.execute; + prepared.execute = function (this: unknown, ...args: unknown[]) { + if (!enabled) return Reflect.apply(original, this, args); + const e = entry(info); + e.executions += 1; + const result = Reflect.apply(original, this, args) as Promise; + return Promise.resolve(result).then(value => { record(e, value); return value; }); + }; +} + +function wrapPrepare(proto: any, operation: string) { + const original = proto._prepare; + proto._prepare = function (this: object, ...args: unknown[]) { + if (!enabled || depth > 0) return Reflect.apply(original, this, args); + const info = site.get(this) ?? { key: `${operation} untagged`, operation, callers: [] }; + depth += 1; + const startBytes = allocated(), startMs = performance.now(); + let prepared: any; + try { prepared = Reflect.apply(original, this, args); } + finally { depth -= 1; } + let buildBytes = allocated() - startBytes, buildMs = performance.now() - startMs; + const e = entry(info); + if (e.builds < CALIBRATED) { + depth += 1; + try { + const loopBytes = allocated(), loopMs = performance.now(); + for (let i = 0; i < LOOP; i++) Reflect.apply(original, this, args); + buildBytes = (allocated() - loopBytes) / LOOP; buildMs = (performance.now() - loopMs) / LOOP; + } catch {} finally { depth -= 1; } + e.calibratedBytes = ((e.calibratedBytes ?? 0) * e.builds + buildBytes) / (e.builds + 1); + e.calibratedMs = ((e.calibratedMs ?? 0) * e.builds + buildMs) / (e.builds + 1); + } else { buildBytes = e.calibratedBytes ?? buildBytes; buildMs = e.calibratedMs ?? buildMs; } + e.builds += 1; + e.buildBytes += buildBytes; + e.buildMs += buildMs; + e.sqlBytes += Buffer.byteLength(prepared?.queryString ?? ""); + e.sqlPrefix ??= String(prepared?.queryString ?? "").slice(0, 200); + e.fields = Array.isArray(prepared?.fields) ? prepared.fields.length : 0; + wrapPrepared(prepared, info); + return prepared; + }; +} + +function wrapFactory(proto: any, method: string, operation: string) { + const original = proto[method]; + proto[method] = function qcostFactory(this: unknown, ...args: unknown[]) { + const result = Reflect.apply(original, this, args); + tag(result, operation, qcostFactory); + return result; + }; +} + +wrapFactory(PgSelectBuilder.prototype, "from", "select"); +wrapFactory(PgInsertBuilder.prototype, "values", "insert"); +wrapFactory(PgInsertBuilder.prototype, "select", "insert"); +wrapFactory(PgUpdateBuilder.prototype, "set", "update"); +wrapFactory(PgDatabase.prototype, "delete", "delete"); +wrapFactory(RelationalQueryBuilder.prototype, "findFirst", "relational"); +wrapFactory(RelationalQueryBuilder.prototype, "findMany", "relational"); +wrapPrepare(PgSelectBase.prototype, "select"); +wrapPrepare(PgInsertBase.prototype, "insert"); +wrapPrepare(PgUpdateBase.prototype, "update"); +wrapPrepare(PgDeleteBase.prototype, "delete"); +wrapPrepare(PgRelationalQuery.prototype, "relational"); + +const originalExecute = (PgDatabase.prototype as any).execute; +(PgDatabase.prototype as any).execute = function qcostExecute(this: any, query: unknown) { + if (!enabled || depth > 0) return Reflect.apply(originalExecute, this, [query]); + const stack = callers(qcostExecute); + const info = { key: `execute ${stack[0] ?? "unknown"}`, operation: "execute", callers: stack }; + depth += 1; + const startBytes = allocated(), startMs = performance.now(); + let raw: any; + try { raw = Reflect.apply(originalExecute, this, [query]); } + finally { depth -= 1; } + const e = entry(info); + let bytes = allocated() - startBytes, ms = performance.now() - startMs; + if (e.builds < CALIBRATED) { + depth += 1; + try { + const sequel = typeof query === "string" ? null : (query as { getSQL(): any }).getSQL(); + const loopBytes = allocated(), loopMs = performance.now(); + for (let i = 0; i < LOOP; i++) if (sequel) this.dialect.sqlToQuery(sequel); + bytes = (allocated() - loopBytes) / LOOP; ms = (performance.now() - loopMs) / LOOP; + } catch {} finally { depth -= 1; } + e.calibratedBytes = ((e.calibratedBytes ?? 0) * e.builds + bytes) / (e.builds + 1); + e.calibratedMs = ((e.calibratedMs ?? 0) * e.builds + ms) / (e.builds + 1); + } else { bytes = e.calibratedBytes ?? bytes; ms = e.calibratedMs ?? ms; } + e.builds += 1; + e.buildBytes += bytes; + e.buildMs += ms; + e.sqlBytes += Buffer.byteLength(raw?.query?.sql ?? ""); + e.sqlPrefix ??= String(raw?.query?.sql ?? "").slice(0, 200); + const execute = raw.execute; + raw.execute = function (this: unknown, ...args: unknown[]) { + if (enabled) e.executions += 1; + return Promise.resolve(Reflect.apply(execute, this, args)).then(value => { if (enabled) record(e, value); return value; }); + }; + return raw; +}; + +export function countStatements(db: { $client: { options: { debug?: unknown } } }) { + db.$client.options.debug = (_connection: number, text: string) => { + totals.statements += 1; + if (/^\s*(begin|commit|rollback|savepoint|release)\b/i.test(text)) totals.control += 1; + }; +} + +export function snapshot() { + return { statements: totals.statements, control: totals.control, allocated: allocated(), at: performance.now(), + entries: new Map([...entries].map(([key, value]) => [key, { ...value }])) }; +} + +export function diff(before: ReturnType) { + const rows: Entry[] = []; + for (const [key, value] of entries) { + const prior = before.entries.get(key); + const d = { ...value } as Entry; + if (prior) for (const field of ["builds", "executions", "sqlBytes", "buildBytes", "buildMs", "readBytes", "readMs", "rows", "resultBytes"] as const) d[field] -= prior[field]; + if (d.builds || d.executions) rows.push(d); + } + rows.sort((a, b) => (b.buildBytes + b.readBytes) - (a.buildBytes + a.readBytes)); + return { statements: totals.statements - before.statements, control: totals.control - before.control, + allocated: allocated() - before.allocated, wallMs: performance.now() - before.at, rows }; +} + +export function format(label: string, units: number, result: ReturnType, top = Number(process.env.TOP ?? 25)) { + const kb = (n: number) => (n / 1024 / units).toFixed(1); + const lines = [`## ${label} (per unit, ${units} units)`, + `statements ${(result.statements / units).toFixed(1)} (control ${(result.control / units).toFixed(1)}), total allocated ${kb(result.allocated)} KB, built ${kb(result.rows.reduce((s, r) => s + r.buildBytes, 0))} KB, read ${kb(result.rows.reduce((s, r) => s + r.readBytes, 0))} KB, wall ${(result.wallMs / units).toFixed(1)} ms`, + "| family | calls | builds | SQL B/build | built KB/call | read KB/call (model) | rows/call | result KB/call | fields | build ms/call |", + "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |"]; + for (const row of result.rows.slice(0, top)) { + const calls = Math.max(row.executions, row.builds); + lines.push(`| ${row.key}${row.callers[1] ? ` <- ${row.callers.slice(1).join(" <- ")}` : ""} | ${(calls / units).toFixed(2)} | ${(row.builds / units).toFixed(2)} | ${row.builds ? Math.round(row.sqlBytes / row.builds) : 0} | ${(row.buildBytes / 1024 / Math.max(1, row.builds)).toFixed(1)} | ${(row.readBytes / 1024 / Math.max(1, row.executions)).toFixed(1)} | ${(row.rows / Math.max(1, row.executions)).toFixed(1)} | ${(row.resultBytes / 1024 / Math.max(1, row.executions)).toFixed(1)} | ${row.fields} | ${(row.buildMs / Math.max(1, row.builds)).toFixed(3)} |`); + } + if (process.env.SQL_PREFIX) for (const row of result.rows.slice(0, top)) lines.push(`- ${row.key}: \`${row.sqlPrefix}\``); + return lines.join("\n"); +} diff --git a/ui/src/components/ActivityFeed.tsx b/ui/src/components/ActivityFeed.tsx index 9c061f64c9d..ce055e201ee 100644 --- a/ui/src/components/ActivityFeed.tsx +++ b/ui/src/components/ActivityFeed.tsx @@ -369,8 +369,8 @@ export function ActivityFeed({ className }: ActivityFeedProps) { // Fetch issues for name resolution const { data: issues } = useQuery({ - queryKey: queryKeys.issues.list(selectedCompanyId ?? ""), - queryFn: () => issuesApi.list(selectedCompanyId!), + queryKey: queryKeys.issues.listIndex(selectedCompanyId ?? ""), + queryFn: ({ signal }) => issuesApi.listCompact(selectedCompanyId!, { projection: "index" }, { signal }), enabled: !!selectedCompanyId, }); diff --git a/ui/src/context/LiveUpdatesProvider.task-index.test.ts b/ui/src/context/LiveUpdatesProvider.task-index.test.ts new file mode 100644 index 00000000000..3123d28323c --- /dev/null +++ b/ui/src/context/LiveUpdatesProvider.task-index.test.ts @@ -0,0 +1,177 @@ +// @vitest-environment node + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { QueryClient, QueryObserver } from "@tanstack/react-query"; +import type { Issue } from "@paperclipai/shared"; +import { __liveUpdatesTestUtils } from "./LiveUpdatesProvider"; +import { createCoalescingQueryClient } from "../lib/query-invalidation-batcher"; +import { queryKeys } from "../lib/queryKeys"; + +const COMPANY = "company-1"; +const ACTOR = { userId: "user-1", agentId: null }; +const INDEX_KEY = queryKeys.issues.listIndex(COMPANY); +const WINDOW_MS = __liveUpdatesTestUtils.TASK_INDEX_REFRESH_WINDOW_MS; + +function task(id: string, patch: Partial = {}): Issue { + return { id, identifier: id.toUpperCase(), title: `Task ${id}`, status: "todo", ...patch } as Issue; +} + +function serverList(initial: Issue[]) { + let rows = initial; + const queryFn = vi.fn(async () => rows); + return { queryFn, set: (next: Issue[]) => { rows = next; } }; +} + +function mount(client: QueryClient, queryKey: readonly unknown[], queryFn: () => Promise) { + const observer = new QueryObserver(client, { queryKey, queryFn, staleTime: 60_000 }); + return observer.subscribe(() => {}); +} + +function emit(client: QueryClient, payload: Record) { + __liveUpdatesTestUtils.invalidateActivityQueries(client, COMPANY, { entityType: "issue", actorType: "agent", actorId: "agent-1", ...payload }, ACTOR); +} + +async function settle(ms = 0) { + await vi.advanceTimersByTimeAsync(ms); +} + +describe("LiveUpdatesProvider company task index", () => { + let client: QueryClient; + beforeEach(() => { + vi.useFakeTimers(); + client = new QueryClient(); + }); + afterEach(() => { + client.clear(); + vi.useRealTimers(); + }); + + it("refetches the index at most once per window during a burst of task activity", async () => { + const index = serverList([task("t1")]); + const other = serverList([]); + const unmountIndex = mount(client, INDEX_KEY, index.queryFn); + const unmountOther = mount(client, [...queryKeys.issues.list(COMPANY), "descendants", "t1"], other.queryFn); + await settle(WINDOW_MS); + expect(index.queryFn).toHaveBeenCalledTimes(1); + + for (let second = 0; second < 60; second += 1) { + emit(client, { entityId: "t1", action: "issue.comment_added" }); + emit(client, { entityId: "t1", action: "issue.updated", details: { status: "in_progress" } }); + await settle(1_000); + } + await settle(WINDOW_MS); + + expect(index.queryFn).toHaveBeenCalledTimes(4); + expect(other.queryFn.mock.calls.length).toBeGreaterThan(60); + unmountIndex(); + unmountOther(); + }); + + it("refetches at once for the first event after a quiet window", async () => { + const index = serverList([task("t1")]); + const unmount = mount(client, INDEX_KEY, index.queryFn); + await settle(WINDOW_MS); + emit(client, { entityId: "t1", action: "issue.updated", details: { title: "Renamed" } }); + await settle(); + expect(index.queryFn).toHaveBeenCalledTimes(2); + unmount(); + }); + + it.each([ + ["a created task", { entityId: "t3", action: "issue.created", details: { identifier: "T3" } }, [task("t1"), task("t2"), task("t3")]], + ["a deleted task", { entityId: "t2", action: "issue.deleted", details: {} }, [task("t1")]], + ["a hidden task", { entityId: "t2", action: "issue.updated", details: { hiddenAt: "2026-11-13T00:00:00.000Z" } }, [task("t1")]], + ["a status, title and priority change", { entityId: "t1", action: "issue.updated", details: { status: "done" } }, + [task("t1", { status: "done", title: "Renamed", priority: "high" }), task("t2")]], + ["an event for a task outside the cache", { entityId: "t9", action: "issue.comment_added" }, [task("t1"), task("t2")]], + ["an event for a task the viewer cannot read", { entityId: "private-1", action: "issue.updated", details: { title: "Secret" } }, [task("t1"), task("t2")]], + ])("shows the server's list after %s, inside one window, with nothing taken from the event", async (_name, event, next) => { + const initial = [task("t1"), task("t2")]; + const index = serverList(initial); + const unmount = mount(client, INDEX_KEY, index.queryFn); + await settle(); + emit(client, { entityId: "t1", action: "issue.comment_added" }); + await settle(1_000); + index.set(next); + emit(client, event); + await settle(); + expect(client.getQueryData(INDEX_KEY)).toEqual(initial); + await settle(WINDOW_MS); + expect(client.getQueryData(INDEX_KEY)).toEqual(next); + expect(JSON.stringify(client.getQueryData(INDEX_KEY))).not.toContain("Secret"); + unmount(); + }); + + it("refreshes at once for the viewer's own task activity, even inside a window", async () => { + const index = serverList([task("t1")]); + const unmount = mount(client, INDEX_KEY, index.queryFn); + await settle(); + emit(client, { entityId: "t1", action: "issue.comment_added" }); + await settle(1_000); + index.set([task("t1"), task("t2")]); + __liveUpdatesTestUtils.invalidateActivityQueries(client, COMPANY, + { entityType: "issue", entityId: "t2", action: "issue.created", actorType: "user", actorId: "user-1", details: { identifier: "T2" } }, ACTOR); + await settle(); + expect(client.getQueryData(INDEX_KEY)).toEqual([task("t1"), task("t2")]); + unmount(); + }); + + it("refreshes at once when this tab changes a task and invalidates the company task lists", async () => { + const index = serverList([task("t1")]); + const unmount = mount(client, INDEX_KEY, index.queryFn); + await settle(); + emit(client, { entityId: "t1", action: "issue.comment_added" }); + await settle(1_000); + index.set([task("t1", { status: "done" })]); + await client.invalidateQueries({ queryKey: queryKeys.issues.list(COMPANY) }); + await settle(); + expect(client.getQueryData(INDEX_KEY)).toEqual([task("t1", { status: "done" })]); + unmount(); + }); + + it("never loses an event inside the window: the last state shows when the window ends after activity stops", async () => { + const index = serverList([task("t1")]); + const unmount = mount(client, INDEX_KEY, index.queryFn); + await settle(WINDOW_MS); + for (let second = 0; second < 10; second += 1) { + index.set([task("t1", { title: `Edit ${second}` })]); + emit(client, { entityId: "t1", action: "issue.updated", details: { title: `Edit ${second}` } }); + await settle(1_000); + } + expect(client.getQueryData(INDEX_KEY)).not.toEqual([task("t1", { title: "Edit 9" })]); + await settle(WINDOW_MS); + expect(client.getQueryData(INDEX_KEY)).toEqual([task("t1", { title: "Edit 9" })]); + unmount(); + }); + + it("marks an unwatched index stale without fetching, and a later mount refetches it", async () => { + const index = serverList([task("t1")]); + const unmount = mount(client, INDEX_KEY, index.queryFn); + await settle(); + unmount(); + emit(client, { entityId: "t1", action: "issue.updated", details: { status: "done" } }); + await settle(WINDOW_MS); + expect(index.queryFn).toHaveBeenCalledTimes(1); + expect(client.getQueryState(INDEX_KEY)?.isInvalidated).toBe(true); + const remount = mount(client, INDEX_KEY, index.queryFn); + await settle(); + expect(index.queryFn).toHaveBeenCalledTimes(2); + remount(); + }); + + it("keeps the window when events arrive through the live-event batcher", async () => { + const index = serverList([task("t1")]); + const unmount = mount(client, INDEX_KEY, index.queryFn); + await settle(WINDOW_MS); + const batched = createCoalescingQueryClient(client, __liveUpdatesTestUtils.createLiveInvalidationBatcher(client, COMPANY)); + for (let second = 0; second < 20; second += 1) { + __liveUpdatesTestUtils.invalidateActivityQueries(batched, COMPANY, + { entityType: "issue", entityId: "t1", action: "issue.comment_added", actorType: "agent", actorId: "agent-1" }, ACTOR); + await settle(1_000); + } + await settle(WINDOW_MS); + expect(index.queryFn.mock.calls.length).toBeLessThanOrEqual(3); + expect(index.queryFn.mock.calls.length).toBeGreaterThanOrEqual(2); + unmount(); + }); +}); diff --git a/ui/src/context/LiveUpdatesProvider.tsx b/ui/src/context/LiveUpdatesProvider.tsx index e72a39c4fab..2c2eaebf6ce 100644 --- a/ui/src/context/LiveUpdatesProvider.tsx +++ b/ui/src/context/LiveUpdatesProvider.tsx @@ -315,9 +315,8 @@ function resolveIssueQueryRefs( const detailIssue = queryClient.getQueryData( queryKeys.issues.detail(issueId), ); - const listIssues = queryClient.getQueryData( - queryKeys.issues.list(companyId), - ); + const listIssues = queryClient.getQueryData(queryKeys.issues.list(companyId)) + ?? queryClient.getQueryData(queryKeys.issues.listIndex(companyId)); const detailsIdentifier = readString(details?.identifier) ?? readString(details?.issueIdentifier); @@ -375,8 +374,8 @@ function resolveIssueToastContext( const detailIssue = issueRefs .map((ref) => queryClient.getQueryData(queryKeys.issues.detail(ref))) .find((issue): issue is Issue => !!issue); - const listIssue = queryClient - .getQueryData(queryKeys.issues.list(companyId)) + const listIssue = (queryClient.getQueryData(queryKeys.issues.list(companyId)) + ?? queryClient.getQueryData(queryKeys.issues.listIndex(companyId))) ?.find((issue) => issueRefs.some((ref) => issue.id === ref || issue.identifier === ref), ); @@ -1066,6 +1065,42 @@ function refreshMentionPoolForActivity( }, wait); } +const TASK_INDEX_REFRESH_WINDOW_MS = 30_000; +const taskIndexRefreshes = new WeakMap< + object, + Map | null }> +>(); + +function refreshTaskIndexForActivity(queryClient: QueryClient, companyId: string, ownActivity: boolean) { + const queryKey = queryKeys.issues.listIndex(companyId); + if (queryClient.getQueryData(queryKey) === undefined) return; + if (ownActivity) { + queryClient.invalidateQueries({ queryKey, exact: true }); + return; + } + const query = queryClient.getQueryCache().find({ queryKey, exact: true }); + if (!query) return; + if (query.getObserversCount() === 0) { + queryClient.invalidateQueries({ queryKey, exact: true, refetchType: "none" }); + return; + } + let refresh = taskIndexRefreshes.get(queryClient); + if (!refresh) { + refresh = new Map(); + taskIndexRefreshes.set(queryClient, refresh); + } + const state = refresh.get(companyId) ?? { lastRequestedAt: 0, timer: null }; + refresh.set(companyId, state); + if (state.timer !== null) return; + const lastRefresh = Math.max(query.state.dataUpdatedAt, state.lastRequestedAt); + const wait = Math.max(0, lastRefresh + TASK_INDEX_REFRESH_WINDOW_MS - Date.now()); + state.timer = setTimeout(() => { + state.timer = null; + state.lastRequestedAt = Date.now(); + queryClient.invalidateQueries({ queryKey, exact: true }); + }, wait); +} + const ISSUE_DOCUMENT_ACTIVITY_ACTIONS = new Set([ "issue.document_created", "issue.document_updated", @@ -1644,8 +1679,9 @@ function invalidateActivityQueries( } queryClient.invalidateQueries({ queryKey: queryKeys.issues.list(companyId), - predicate: excludingQueryKeys(queryKeys.issues.mentionPool(companyId)), + predicate: excludingQueryKeys(queryKeys.issues.mentionPool(companyId), queryKeys.issues.listIndex(companyId)), }); + refreshTaskIndexForActivity(queryClient, companyId, ownActorActivity); refreshMentionPoolForActivity( queryClient, companyId, @@ -2320,6 +2356,7 @@ export const __liveUpdatesTestUtils = { createLiveInvalidationBatcher, isAmbientCompanyQueryKey, MENTION_POOL_REFRESH_WINDOW_MS, + TASK_INDEX_REFRESH_WINDOW_MS, buildActivityToast, applyRunLifecycleToCompanyLiveRuns, buildAgentStatusToast, diff --git a/ui/src/lib/queryKeys.ts b/ui/src/lib/queryKeys.ts index 302b1897e95..549457d5292 100644 --- a/ui/src/lib/queryKeys.ts +++ b/ui/src/lib/queryKeys.ts @@ -323,6 +323,7 @@ export const queryKeys = { ["issues", id, "recovery-actions", actionId, runId] as const, allRecoveryActions: (id: string) => ["issues", id, "recovery-actions"] as const, list: (companyId: string) => ["issues", companyId] as const, + listIndex: (companyId: string) => ["issues", companyId, "list-index"] as const, onboardingFirstTask: (companyId: string) => ["issues", companyId, "onboarding-first-task"] as const, starPromptDone: (companyId: string) => diff --git a/ui/src/pages/BoardChat.test.tsx b/ui/src/pages/BoardChat.test.tsx index a18800ddd1a..713c26f8778 100644 --- a/ui/src/pages/BoardChat.test.tsx +++ b/ui/src/pages/BoardChat.test.tsx @@ -20,7 +20,7 @@ import { BoardChat } from "./BoardChat"; const mockAgentsApi = vi.hoisted(() => ({ list: vi.fn() })); const mockGoalsApi = vi.hoisted(() => ({ list: vi.fn() })); const mockIssuesApi = vi.hoisted(() => ({ - list: vi.fn(), + listCompact: vi.fn(), listComments: vi.fn(), listFeedbackVotes: vi.fn(), })); @@ -127,7 +127,7 @@ describe("BoardChat staged typing intro", () => { mockGoalsApi.list.mockResolvedValue([ { id: "goal-1", title: "Build affordable robots", status: "active" }, ]); - mockIssuesApi.list.mockResolvedValue([BOARD_ISSUE]); + mockIssuesApi.listCompact.mockResolvedValue([BOARD_ISSUE]); mockIssuesApi.listComments.mockResolvedValue([]); mockIssuesApi.listFeedbackVotes.mockResolvedValue([]); }); diff --git a/ui/src/pages/BoardChat.tsx b/ui/src/pages/BoardChat.tsx index 3901c926e9f..9ddd34a0d36 100644 --- a/ui/src/pages/BoardChat.tsx +++ b/ui/src/pages/BoardChat.tsx @@ -286,8 +286,8 @@ export function BoardChat() { // Find or detect the board operations issue const { data: issues } = useQuery({ - queryKey: queryKeys.issues.list(selectedCompanyId!), - queryFn: () => issuesApi.list(selectedCompanyId!), + queryKey: queryKeys.issues.listIndex(selectedCompanyId!), + queryFn: ({ signal }) => issuesApi.listCompact(selectedCompanyId!, { projection: "index" }, { signal }), enabled: !!selectedCompanyId, }); diff --git a/ui/src/pages/Dashboard.test.ts b/ui/src/pages/Dashboard.test.ts index bc053d2e7b0..088ea2c27c6 100644 --- a/ui/src/pages/Dashboard.test.ts +++ b/ui/src/pages/Dashboard.test.ts @@ -1,6 +1,6 @@ -import type { Agent } from "@paperclipai/shared"; +import type { Agent, CompactIssue, IssueSummary } from "@paperclipai/shared"; import { describe, expect, it } from "vitest"; -import { derivePausedAgentBanner } from "./Dashboard"; +import { derivePausedAgentBanner, recentTaskIndexRows, withTaskSummaries } from "./Dashboard"; function agent(overrides: Partial): Agent { return { @@ -43,3 +43,36 @@ describe("derivePausedAgentBanner", () => { expect(banner).toBeNull(); }); }); + +describe("dashboard recent tasks from the task index", () => { + const row = (id: string, updatedAt: string): CompactIssue => + ({ id, identifier: id.toUpperCase(), title: id, status: "blocked", updatedAt, assigneeAgentId: "agent-1" }) as unknown as CompactIssue; + + it("keeps the ten most recently updated tasks, newest first", () => { + const rows = Array.from({ length: 14 }, (_, index) => row(`t${index}`, new Date(Date.UTC(2026, 10, 1 + index)).toISOString())); + expect(recentTaskIndexRows(rows.reverse()).map(issue => issue.id)).toEqual( + ["t13", "t12", "t11", "t10", "t9", "t8", "t7", "t6", "t5", "t4"]); + }); + + it("reads each task's update time once, in one pass over a large company", () => { + let reads = 0; + const rows = Array.from({ length: 5_000 }, (_, index) => { + const at = new Date(Date.UTC(2026, 0, 1) + ((index * 7919) % 5_000) * 60_000).toISOString(); + return Object.defineProperty({ id: `t${index}` }, "updatedAt", { get: () => { reads += 1; return at; } }) as unknown as CompactIssue; + }); + const recent = recentTaskIndexRows(rows); + expect(reads).toBe(5_000); + const sorted = [...rows].sort((a, b) => Date.parse(String(b.updatedAt)) - Date.parse(String(a.updatedAt))).slice(0, 10); + expect(recent.map(issue => issue.id)).toEqual(sorted.map(issue => issue.id)); + }); + + it("takes the status glyph fields from the task summary and the rest from the index row", () => { + const summary = { id: "t1", displayStatus: "in_queue", externalConversationState: "waiting", + blockerAttention: { state: "covered" } } as unknown as IssueSummary; + const [merged] = withTaskSummaries([row("t1", "2026-11-01T00:00:00.000Z")], [summary]); + expect(merged).toMatchObject({ id: "t1", title: "t1", assigneeAgentId: "agent-1", displayStatus: "in_queue", + externalConversationState: "waiting", blockerAttention: { state: "covered" } }); + const [plain] = withTaskSummaries([row("t2", "2026-11-01T00:00:00.000Z")], undefined); + expect(plain).not.toHaveProperty("displayStatus"); + }); +}); diff --git a/ui/src/pages/Dashboard.tsx b/ui/src/pages/Dashboard.tsx index 0d9a71537d9..0e160ce1deb 100644 --- a/ui/src/pages/Dashboard.tsx +++ b/ui/src/pages/Dashboard.tsx @@ -1,4 +1,5 @@ import { companyProjectsQuery } from "@/lib/company-projects-query"; +import { useAccountIdentity } from "@/api/companies-query"; import { AgentIdentity } from "../components/AgentIdentity"; import { TaskSummaryList } from "../components/TaskSummaryList"; import { useDesignSandbox } from "../context/DesignSandboxContext"; @@ -37,18 +38,43 @@ import { QueryErrorState, useQueryView } from "../components/QueryView"; import { Card } from "@/components/ui/card"; import { Button } from "@/components/ui/button"; import { InlineBanner } from "../components/InlineBanner"; -import type { Agent, Issue } from "@paperclipai/shared"; +import type { Agent, CompactIssue, Issue, IssueSummary } from "@paperclipai/shared"; import { PluginSlotOutlet } from "@/plugins/slots"; import { SmokeLabDashboardCard } from "../components/SmokeLabDashboardCard"; import { taskStatusIconProps } from "@/lib/task-status-presentation"; const DASHBOARD_ACTIVITY_LIMIT = 10; +const DASHBOARD_RECENT_TASK_LIMIT = 10; -function getRecentIssues(issues: Issue[]): Issue[] { - return [...issues] - .sort((a, b) => new Date(b.updatedAt).getTime() - new Date(a.updatedAt).getTime()); +export function recentTaskIndexRows(issues: readonly CompactIssue[]): CompactIssue[] { + const recent: Array<{ issue: CompactIssue; at: number }> = []; + for (const issue of issues) { + const at = new Date(issue.updatedAt).getTime(); + if (recent.length === DASHBOARD_RECENT_TASK_LIMIT && at <= recent[recent.length - 1]!.at) continue; + let index = recent.length; + while (index > 0 && recent[index - 1]!.at < at) index -= 1; + recent.splice(index, 0, { issue, at }); + if (recent.length > DASHBOARD_RECENT_TASK_LIMIT) recent.pop(); + } + return recent.map(entry => entry.issue); } +export function withTaskSummaries(rows: readonly CompactIssue[], summaries: readonly IssueSummary[] | undefined): Issue[] { + const byId = new Map((summaries ?? []).map((summary) => [summary.id, summary])); + return rows.map((row) => { + const summary = byId.get(row.id); + return { + ...row, + ...(summary ? { + displayStatus: summary.displayStatus, + externalConversationState: summary.externalConversationState, + ...(summary.blockerAttention ? { blockerAttention: summary.blockerAttention } : {}), + } : {}), + } as Issue; + }); +} + + export type PausedAgentBanner = | { kind: "imported"; pausedImportedAgentIds: string[] } | { kind: "all-paused" } @@ -196,10 +222,18 @@ export function Dashboard() { usePublishSharedQueryData(sharedActivity, activity, activityUpdatedAt); const { data: issues } = useQuery({ - queryKey: queryKeys.issues.list(selectedCompanyId!), - queryFn: () => issuesApi.list(selectedCompanyId!), + queryKey: queryKeys.issues.listIndex(selectedCompanyId!), + queryFn: ({ signal }) => issuesApi.listCompact(selectedCompanyId!, { projection: "index" }, { signal }), enabled: !!selectedCompanyId, }); + const { userId: accountUserId, settled: accountSettled } = useAccountIdentity(); + const recentIndexRows = useMemo(() => recentTaskIndexRows(issues ?? []), [issues]); + const recentIds = useMemo(() => recentIndexRows.map((issue) => issue.id).sort(), [recentIndexRows]); + const { data: recentSummaries } = useQuery({ + queryKey: queryKeys.issues.summaryBatch(selectedCompanyId!, accountUserId, recentIds), + queryFn: ({ signal }) => issuesApi.summaries(selectedCompanyId!, recentIds, { signal }), + enabled: !!selectedCompanyId && accountSettled && recentIds.length > 0, + }); const { data: projects } = useQuery({ ...companyProjectsQuery(selectedCompanyId!, { includeArchived: true }), @@ -217,7 +251,10 @@ export function Dashboard() { [companyMembers?.users], ); - const recentIssues = issues ? getRecentIssues(issues) : []; + const recentIssues = useMemo( + () => withTaskSummaries(recentIndexRows, recentSummaries), + [recentIndexRows, recentSummaries], + ); const recentActivity = useMemo(() => (activity ?? []).slice(0, 10), [activity]); useEffect(() => {