From 30c4d2f4a130a457186b38fd414c0e6b2e9f11ce Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 21:42:06 -0600 Subject: [PATCH] perf(server): read the issue detail with fewer database statements (#21713) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI loads the task page through `GET /api/issues/:id`, and the server runs all requests on one Node main thread > - A CPU profile of the server shows that the cost is per database statement: SQL building, parameter encoding, result parsing and one socket write each > - The issue detail read sent 73 statements for a task with two ancestors, one blocker and a project mention > - Several of those statements read the issue row again, only to get values that the route already loaded > - This pull request passes the loaded issue to those reads, walks the ancestor chain in one query, and skips reads whose result the response does not use > - The benefit is 9 fewer statements on each task page load (73 to 64), plus one fewer statement for each extra ancestor level, with a byte-identical response ## Linked Issues or Issue Description **What existing behavior does this improve?** `GET /api/issues/:id` reads the same issue row several times in one request, and walks the ancestor chain with one statement for each level. **Subsystem affected** Server: issue routes and the issue, issue reference and project services. **Current behavior** The route loads the issue, then `getRelationSummaries`, `getCurrentScheduledRetry`, `findMentionedProjectIds`, `listIssueReferenceSummary` and `getAncestors` each read the issue row again. `getAncestors` sends one statement for each ancestor. The inbox archive fields read two activity aggregates even when the issue has no archive row. The project embed reads the workspace provider catalog, but the compact embed does not return it. The test fixture request sends 73 statements. **Proposed behavior** The route passes the values it already has. The ancestor walk is one recursive query. The activity aggregates run only when an archive row exists. The project embed skips the provider catalog. The same fixture request sends 64 statements, and the JSON response is the same. **Reason and benefit** Fewer statements per request means less CPU on the main thread for every open task page. **Breaking changes** None. Other callers of the changed service functions keep the old behavior, because the new arguments are optional. ## What Changed - `issueService.getRelationSummaries` and `getCurrentScheduledRetry` accept an optional known `companyId` and then skip the issue read. - `issueService.findMentionedProjectIds` accepts the loaded issue (`companyId`, `title`, `description`). - `issueService.getAncestors` accepts the known `parentId` and walks the chain in one recursive query. It keeps the order (nearest first), the limit of 50, and the stop at a cycle or at a task that is its own parent. - `issueService.getActiveInboxArchiveFields` reads the archive row first and reads the activity aggregates only when an archive row exists. - `issueReferenceService.listIssueReferenceSummary` accepts an optional known `companyId`. - `issueThreadInteractionService.listForIssue` accepts the loaded issue (`status`, `companyId`, `executionWorkspaceId`). `GET /api/issues/:id/interactions`, which the board polls, passes it (11 to 10 statements). - `projectService.getById` accepts `{ workspaceProvider: false }`. The issue detail route uses it, because `compactIssueProject` never returns `workspaceProvider`. - Test helpers: `createStatementCensus` (statements per request, grouped by statement shape with the server callers), and a board route fixture with the real session actor middleware. `meterDatabaseStatements` can record a call stack for each statement. - New tests: `issue-detail-statements.test.ts` (statement budget, byte-identical normalized JSON against a fixture file, access rules, membership removal on the next request, ancestor walk with a cycle) plus an interactions poll test, and `board-route-statement-census.test.ts` (census of the polled and page-load board routes). ## Verification - Red: before the change, the fixture request sent 73 statements, and the budget test failed (`expected 73 to be less than or equal to 64`). - Green: after the change, the request sends 64 statements. The normalized JSON is byte-identical to the fixture file that I wrote before the change. - Access: another company's user gets 404, a missing issue gets 404, an anonymous request gets 404, a hidden issue stays readable for a member, and a removed membership gets 404 on the very next request. - `pnpm exec vitest run` on `issue-detail-statements`, `board-route-statement-census`, `issues-goal-context-routes`, `issues-service`, `issue-references-service`, `inbox-archive-routes`, `heartbeat-task-context`, `project-workspace-provider.integration`, `workspace-provider-failure`, `issue-telemetry-routes`, `issue-thread-interaction-routes`, `issue-document-restore-routes`, `heartbeat-referenced-projects`, `project-environments`: 14 files, 414 tests pass. - `pnpm --filter @paperclipai/server typecheck` passes. - Interactions: the polled route sends 10 statements instead of 11, with a response equal to the service result without the known issue. - Census: `PAPERCLIP_STATEMENT_CENSUS=/tmp/census.txt pnpm exec vitest run src/__tests__/board-route-statement-census.test.ts` writes statements per request and the top statement shapes for each board route. ## Risks - Low risk. The new arguments are optional, so other callers keep the old reads. - When the route passes a known `companyId`, a concurrent delete of the issue no longer gives a 404 from these sub-reads. The route already loaded and authorized the issue in the same request, so the response is the same in practice. - The JSON fixture file fails when someone changes the issue detail response. Regenerate it with `PAPERCLIP_WRITE_ISSUE_DETAIL_GOLDEN=1` and review the diff. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../board-route-statement-census.test.ts | 29 ++ .../fixtures/issue-detail-response.json | 280 ++++++++++++++++++ .../helpers/database-statement-meter.ts | 13 +- .../__tests__/helpers/route-census-fixture.ts | 87 ++++++ .../src/__tests__/helpers/statement-census.ts | 72 +++++ .../__tests__/issue-detail-statements.test.ts | 142 +++++++++ .../issue-thread-interaction-routes.test.ts | 2 +- .../issues-goal-context-routes.test.ts | 2 +- server/src/routes/issues.ts | 18 +- server/src/services/issue-references.ts | 8 +- .../src/services/issue-thread-interactions.ts | 7 +- server/src/services/issues.ts | 110 ++++--- server/src/services/projects.ts | 23 +- 13 files changed, 721 insertions(+), 72 deletions(-) create mode 100644 server/src/__tests__/board-route-statement-census.test.ts create mode 100644 server/src/__tests__/fixtures/issue-detail-response.json create mode 100644 server/src/__tests__/helpers/route-census-fixture.ts create mode 100644 server/src/__tests__/helpers/statement-census.ts create mode 100644 server/src/__tests__/issue-detail-statements.test.ts diff --git a/server/src/__tests__/board-route-statement-census.test.ts b/server/src/__tests__/board-route-statement-census.test.ts new file mode 100644 index 00000000000..e2c8d58bf28 --- /dev/null +++ b/server/src/__tests__/board-route-statement-census.test.ts @@ -0,0 +1,29 @@ +import { writeFileSync } from "node:fs"; +import request from "supertest"; +import { expect, it } from "vitest"; +import { boardCensusRoutes, boardSessionApp, seedBoardRouteFixture } from "./helpers/route-census-fixture.js"; +import { describeEmbeddedPostgres, useEmbeddedPostgres } from "./helpers/route-test-harness.js"; +import { createStatementCensus, formatStatementCensus } from "./helpers/statement-census.js"; + +describeEmbeddedPostgres("board route statement census", () => { + const ctx = useEmbeddedPostgres("board-route-census-"); + + it("answers every polled and page-load board route and reports statements per request", async () => { + const census = createStatementCensus(ctx.db); + try { + const fixture = await seedBoardRouteFixture(ctx.db, "Census"); + const app = boardSessionApp(ctx.db, () => fixture.userId); + const report: string[] = []; + for (const [label, path] of boardCensusRoutes(fixture)) { + const warm = await request(app).get(path); + expect(warm.status, `${label}: ${JSON.stringify(warm.body)}`).toBe(200); + const measured = await census.measure(() => request(app).get(path).expect(200), { stacks: true }); + expect(measured.statements, label).toBeGreaterThan(0); + report.push(formatStatementCensus(label, measured, Number(process.env.PAPERCLIP_STATEMENT_CENSUS_TOP ?? 5))); + } + if (process.env.PAPERCLIP_STATEMENT_CENSUS) writeFileSync(process.env.PAPERCLIP_STATEMENT_CENSUS, `${report.join("\n\n")}\n`); + } finally { + census.uninstall(); + } + }, 120_000); +}); diff --git a/server/src/__tests__/fixtures/issue-detail-response.json b/server/src/__tests__/fixtures/issue-detail-response.json new file mode 100644 index 00000000000..e1d98dc1f95 --- /dev/null +++ b/server/src/__tests__/fixtures/issue-detail-response.json @@ -0,0 +1,280 @@ +{ + "id": "", + "companyId": "", + "conversationAgentId": null, + "conversationUserId": null, + "conversationState": null, + "conversationSessionGeneration": 0, + "conversationBoundaryCommentId": null, + "projectId": "", + "projectWorkspaceId": null, + "goalId": null, + "parentId": "", + "visibility": "open", + "privacyRootIssueId": null, + "privacyParentIssueId": null, + "title": "Detail task", + "titleNeedsGeneration": false, + "description": "Uses [@Detail project](project://)", + "status": "in_progress", + "statusVersion": 0, + "lastStatusDecisionId": null, + "workMode": "standard", + "harnessKind": null, + "priority": "high", + "reviewPolicy": null, + "assigneeAgentId": "", + "assigneeUserId": null, + "checkoutRunId": null, + "executionRunId": null, + "executionAgentNameKey": null, + "executionLockedAt": null, + "createdByAgentId": null, + "createdByUserId": "", + "responsibleUserId": null, + "issueNumber": null, + "identifier": null, + "originKind": "manual", + "originId": null, + "originRunId": null, + "originIssueId": null, + "originIdentityContextId": null, + "continuationIdentityContextId": null, + "originFingerprint": "default", + "requestDepth": 0, + "billingCode": null, + "assigneeAdapterOverrides": null, + "executionPolicy": null, + "executionState": null, + "monitorNextCheckAt": null, + "monitorWakeRequestedAt": null, + "monitorLastTriggeredAt": null, + "monitorAttemptCount": 0, + "monitorNotes": null, + "monitorScheduledBy": null, + "executionWorkspaceId": null, + "workspaceBindingRevision": 0, + "workspaceSelection": null, + "workspacePendingSelection": null, + "executionWorkspacePreference": null, + "executionWorkspaceSettings": null, + "sourceTrust": null, + "unblockDescriptor": null, + "blockedTransitionAt": null, + "blockedOwnerNotifiedAt": null, + "startedAt": null, + "completedAt": null, + "cancelledAt": null, + "hiddenAt": null, + "createdAt": "