mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
perf: fetch recent task summaries in one request (#20389)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI sidebar shows the person's recent tasks on every page, including the Mine inbox > - The sidebar refreshes those rows through `GET /companies/:id/issues/summaries`, which accepted at most five IDs > - `useRecentTasks` sent the chunks one after another, so a 25-task history cost five sequential requests > - On staging each request took 3–6 s, so the recent-task rows settled about 20 s after load and every request competed with the inbox rows > - This pull request raises the per-request cap to 50 and fetches any remaining chunks in parallel > - The benefit is one summaries request for a typical history instead of a serial chain ## Linked Issues or Issue Description **What existing behavior does this improve?** The refresh of the sidebar's recent tasks on page load. **Subsystem affected** Board UI (`ui/src/hooks/useRecentTasks.ts`) and the `/issues/summaries` route bound (`packages/shared`, `server/src/routes/issues.ts`). **Current behavior** A history of N recent tasks sends ceil(N/5) summaries requests in series. Measured on staging on 2026-11-09 for a cold Mine load: five requests of five IDs, started at 1.0 s, 4.3 s, 8.1 s, 12.3 s and 18.4 s, each taking 3.3–6.1 s. **Proposed behavior** Up to 50 recent tasks are one request. Longer histories send their chunks at the same time. **Reason and benefit** The staging server is saturated, so each extra serial round trip delays the sidebar and the inbox rows. One request removes four round trips from a typical load. ## What Changed - `ISSUE_SUMMARIES_MAX_IDS` is now 50 (was 5). The shared validator, the route's 400 message, the service guard and the OpenAPI description use the new bound. - `useRecentTasks` builds all chunks first and fetches them at most three at a time (`RECENT_TASK_SUMMARY_CONCURRENCY`). The recent list has no size cap, so this keeps a long history from becoming a burst. The cache key, omission handling and pruning are unchanged. A failed chunk still fails the batch, so nothing is pruned on a partial failure. - Tests: the route test accepts exactly `MAX` IDs and rejects `MAX + 1`; a hook test checks that at most three chunks are in flight; the API test uses the constant; `useRecentTasks` tests cover a 25-task history in one request and a 103-task history with three chunks in flight together, with and without a failed chunk. - New browser check `tests/perf/collections/recent-task-summaries.spec.ts`: a Mine load with a 12-task history sends summaries requests that each carry all 12 IDs, and the server answers 200 with all 12 tasks. ## Verification - `pnpm verify` (selected Vitest files) passes. - `server`: `npx vitest run src/__tests__/issue-summaries-routes.test.ts src/__tests__/issue-display-status.test.ts` passes (embedded PostgreSQL). - `ui`: `npx vitest run src/hooks/useRecentTasks.test.tsx src/api/issue-summaries.test.ts src/components/task-chat/TaskChatBlockerLinks.test.tsx src/components/CoordinatorSidebar.test.tsx src/components/SidebarRecentTasks.test.tsx` passes. - Browser check against a disposable local instance: `PAPERCLIP_COLLECTION_PERF_BASE_URL=http://127.0.0.1:<port> pnpm exec playwright test --config tests/perf/collections/playwright.config.ts recent-task-summaries.spec.ts`. It passes on this branch. With `master`'s hook and constant it fails: three requests of 5, 5 and 2 IDs. ## Risks - Low risk. The route still validates UUIDs, deduplicates, and applies the same read authorization per row. Blocker attention is computed with set-based queries, so 50 IDs is the same number of queries as 5. - During a rolling deploy, a new UI against an old server would get a 400 for a history above five tasks. The server serves the UI bundle, so both change together on a normal deploy. The sidebar keeps its stored snapshot when the refresh fails. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, browser). 1M-token context window. Extended thinking was off. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2f47d0fadb
commit
7e93ccb4b3
9 files changed
+129
-20
No files matched your search
@@ -1,5 +1,9 @@
|
||||
export const COMPANY_STATUSES = ["active", "paused", "archived"] as const;
|
||||
export const ISSUE_SUMMARIES_MAX_IDS = 5;
|
||||
/**
|
||||
* IDs per `/issues/summaries` request. Large enough that a typical recent-task
|
||||
* list is one request; callers split longer lists into parallel chunks.
|
||||
*/
|
||||
export const ISSUE_SUMMARIES_MAX_IDS = 50;
|
||||
/** Tasks per chat bootstrap request. The list preloads larger sets in chunks. */
|
||||
export const ISSUE_CHAT_BOOTSTRAP_MAX_IDS = 10;
|
||||
export type CompanyStatus = (typeof COMPANY_STATUSES)[number];
|
||||
|
||||
@@ -6,6 +6,7 @@ import { afterAll, beforeAll, expect, it, vi } from "vitest";
|
||||
import {
|
||||
agents, chatConversations, chatEndpoints, heartbeatRuns, issueRelations, issues, toolApplications, toolConnections,
|
||||
} from "@paperclipai/db";
|
||||
import { ISSUE_SUMMARIES_MAX_IDS } from "@paperclipai/shared";
|
||||
import { issueRoutes } from "../routes/issues.js";
|
||||
import * as issueServices from "../services/issues.js";
|
||||
import { errorHandler } from "../middleware/index.js";
|
||||
@@ -59,12 +60,20 @@ describeEmbeddedPostgres("bounded issue summaries", () => {
|
||||
|
||||
it.each(["", "?ids=", "?ids=nope", "?ids=,", "?ids[]=nope", "?ids[x]=nope",
|
||||
`?ids=${randomUUID()}&ids[]=nope`, `?ids=${randomUUID()}&ids[x]=nope`,
|
||||
`?ids=${randomUUID()}&ids=${randomUUID()}`, `?ids=${Array(6).fill(randomUUID()).join(",")}`,
|
||||
`?ids=${randomUUID()}&ids=${randomUUID()}`, `?ids=${Array.from({ length: ISSUE_SUMMARIES_MAX_IDS + 1 }, () => randomUUID()).join(",")}`,
|
||||
])("rejects invalid query %s", async (query) => {
|
||||
const f = await fixture();
|
||||
await request(app(f.actor)).get(url(f.companyId) + query).expect(400);
|
||||
});
|
||||
|
||||
it("accepts exactly the maximum number of IDs", async () => {
|
||||
const f = await fixture();
|
||||
const ids = [f.rows[0]!.id, f.rows[1]!.id,
|
||||
...Array.from({ length: ISSUE_SUMMARIES_MAX_IDS - 2 }, () => randomUUID())];
|
||||
const res = await request(app(f.actor)).get(url(f.companyId)).query({ ids: ids.join(",") }).expect(200);
|
||||
expect(res.body.map((row: { id: string }) => row.id)).toEqual([f.rows[0]!.id, f.rows[1]!.id]);
|
||||
});
|
||||
|
||||
it("omits hidden, private, foreign and missing IDs without revealing them", async () => {
|
||||
const f = await fixture();
|
||||
const foreign = await fixture();
|
||||
|
||||
@@ -13,7 +13,7 @@ import { getHeartbeatRunReadDecisions, redactHeartbeatRunListRow } from "../serv
|
||||
import { taskInterruptionPresentation, taskInterruptionPresentations } from "../services/task-interruption-presentation.js";
|
||||
import { parseObject } from "../adapters/utils.js";
|
||||
import { prepareRestrictedWorkspaceRetry } from "../services/restricted-workspace-retry.js";
|
||||
import { issueChatBootstrapQuerySchema, issueSummariesQuerySchema, type IssueSummary } from "@paperclipai/shared";
|
||||
import { ISSUE_SUMMARIES_MAX_IDS, issueChatBootstrapQuerySchema, issueSummariesQuerySchema, type IssueSummary } from "@paperclipai/shared";
|
||||
import { readIssueChatBootstrapItems } from "../services/issue-chat-bootstrap.js";
|
||||
import { issueRunPresenceService } from "../services/issue-run-presence.js";
|
||||
import { activityService } from "../services/activity.js";
|
||||
@@ -8572,13 +8572,13 @@ export function issueRoutes(
|
||||
}
|
||||
const parsed = issueSummariesQuerySchema.safeParse(req.query);
|
||||
if (!parsed.success) {
|
||||
res.status(400).json({ error: "ids must be the only query parameter and contain one to five comma-separated UUIDs" });
|
||||
res.status(400).json({ error: `ids must be the only query parameter and contain one to ${ISSUE_SUMMARIES_MAX_IDS} comma-separated UUIDs` });
|
||||
return;
|
||||
}
|
||||
const rows = await listIssueSummaries(db, companyId, parsed.data.ids, await issueReadSqlCondition(db, req.actor));
|
||||
const visible = await filterIssuesForActor(req, rows);
|
||||
// Blocked rows carry glyph-only blocker state so every task view resolves
|
||||
// the same glyph (covered → "in queue") as the task pane. At most five IDs.
|
||||
// the same glyph (covered → "in queue") as the task pane. At most ISSUE_SUMMARIES_MAX_IDS IDs.
|
||||
const blockerAttention = await listIssueStatusBlockerAttention(db, companyId, visible, async (ids) => new Set(
|
||||
(await filterIssuesForActor(req, await listIssueSummaries(db, companyId, ids, await issueReadSqlCondition(db, req.actor))))
|
||||
.map((row) => row.id),
|
||||
|
||||
@@ -4367,10 +4367,10 @@ registry.registerPath({
|
||||
path: "/api/companies/{companyId}/issues/summaries",
|
||||
tags: ["issues"],
|
||||
summary: "Get authorized recent-task summaries",
|
||||
description: "Accepts only the ids query parameter: one to five comma-separated UUIDs, bounded before deduplication. IDs are trimmed and normalized to lowercase. Repeated, array and nested query syntax is rejected, including mixed forms. Returns accessible, visible tasks in request order; unavailable IDs are omitted. This private, no-store read excludes full task and workspace enrichment.",
|
||||
description: "Accepts only the ids query parameter: one to fifty comma-separated UUIDs, bounded before deduplication. IDs are trimmed and normalized to lowercase. Repeated, array and nested query syntax is rejected, including mixed forms. Returns accessible, visible tasks in request order; unavailable IDs are omitted. This private, no-store read excludes full task and workspace enrichment.",
|
||||
request: {
|
||||
params: z.object({ companyId: z.string() }),
|
||||
query: z.object({ ids: z.string().min(1).describe("One to five comma-separated task UUIDs; repeated query parameters are rejected") }),
|
||||
query: z.object({ ids: z.string().min(1).describe("One to fifty comma-separated task UUIDs; repeated query parameters are rejected") }),
|
||||
},
|
||||
responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden },
|
||||
});
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
import { expect, test } from "@playwright/test";
|
||||
import { collectionFixture, type CollectionFixture } from "./fixture";
|
||||
|
||||
let fixture: CollectionFixture;
|
||||
test.beforeAll(async ({ request }) => {
|
||||
fixture = await collectionFixture(request);
|
||||
});
|
||||
|
||||
// A 12-task history used to cost three serial five-ID summary requests on
|
||||
// every page load. It must be one request.
|
||||
test("Mine refreshes a recent-task history with one summaries request", async ({ page, request }) => {
|
||||
const session = await (await request.get("/api/auth/get-session")).json().catch(() => null);
|
||||
const userId: string | null = session?.user?.id ?? session?.session?.userId ?? null;
|
||||
const companyId = fixture.company.id;
|
||||
const taskIds = fixture.taskIds.slice(0, 12);
|
||||
await page.addInitScript(({ companyId, userId, taskIds }) => {
|
||||
const entries = taskIds.map((id, index) => ({
|
||||
id, companyId, title: `Recent ${index}`, identifier: null, status: "backlog", recordedAt: Date.now() - index * 1000,
|
||||
}));
|
||||
for (const owner of new Set([userId ?? "__local_board__", "__local_board__"])) {
|
||||
localStorage.setItem(`paperclip.recentTasks.v2:${companyId}:${owner}`, JSON.stringify(entries));
|
||||
}
|
||||
}, { companyId, userId, taskIds });
|
||||
const summaries: URL[] = [];
|
||||
page.on("request", item => {
|
||||
const url = new URL(item.url());
|
||||
if (url.pathname === `/api/companies/${companyId}/issues/summaries`) summaries.push(url);
|
||||
});
|
||||
const response = page.waitForResponse(item => new URL(item.url()).pathname === `/api/companies/${companyId}/issues/summaries`);
|
||||
await page.goto(`/${fixture.company.issuePrefix}/issues?view=mine`);
|
||||
// The server must accept the whole batch, not just receive it.
|
||||
const first = await response;
|
||||
expect(first.status()).toBe(200);
|
||||
const rows = await first.json() as Array<{ id: string }>;
|
||||
expect(rows.map(row => row.id).sort()).toEqual([...taskIds].sort());
|
||||
// Allow any follow-up chunk to start before counting.
|
||||
await page.waitForTimeout(1_500);
|
||||
// The live-updates socket may refresh the batch once after it opens; that
|
||||
// repeats the same request. Chunked requests would each carry fewer IDs.
|
||||
for (const url of summaries) expect(url.searchParams.get("ids")!.split(",")).toHaveLength(12);
|
||||
expect(summaries.length).toBeLessThanOrEqual(2);
|
||||
});
|
||||
@@ -1,4 +1,5 @@
|
||||
import { beforeEach, afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { ISSUE_SUMMARIES_MAX_IDS } from "@paperclipai/shared";
|
||||
import { issuesApi } from "./issues";
|
||||
|
||||
const first = "11111111-1111-4111-8111-111111111111";
|
||||
@@ -22,7 +23,7 @@ describe("issue summary HTTP requests", () => {
|
||||
|
||||
it("does not send empty or oversized batches", async () => {
|
||||
expect(await issuesApi.summaries("company", [])).toEqual([]);
|
||||
await expect(issuesApi.summaries("company", Array(6).fill(first))).rejects.toThrow("Too many");
|
||||
await expect(issuesApi.summaries("company", Array(ISSUE_SUMMARIES_MAX_IDS + 1).fill(first))).rejects.toThrow("Too many");
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@ function relation(id: string, identifier: string, status: IssueRelationIssueSumm
|
||||
|
||||
// The onboarding walk: BRA-1 waits on BRA-2 (running), todo tasks and
|
||||
// blocked tasks that are "covered" in the task list. Six steps plus a live
|
||||
// terminal blocker exceed one summaries request (at most five IDs).
|
||||
// terminal blocker; every summaries request stays within the endpoint limit.
|
||||
const terminal = relation("bra-9", "BRA-9", "in_progress");
|
||||
const blockers = [
|
||||
{ ...relation("bra-2", "BRA-2", "in_progress"), terminalBlockers: [terminal] },
|
||||
|
||||
@@ -4,6 +4,7 @@ import { act } from "react";
|
||||
import { createRoot, type Root } from "react-dom/client";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { ISSUE_SUMMARIES_MAX_IDS } from "@paperclipai/shared";
|
||||
import { ApiError } from "@/api/client";
|
||||
import { queryKeys } from "@/lib/queryKeys";
|
||||
import {
|
||||
@@ -12,7 +13,7 @@ import {
|
||||
recordRecentTask,
|
||||
RECENT_TASKS_UPDATED_EVENT,
|
||||
} from "@/lib/recent-tasks";
|
||||
import { useRecentTasks } from "./useRecentTasks";
|
||||
import { RECENT_TASK_SUMMARY_CONCURRENCY, useRecentTasks } from "./useRecentTasks";
|
||||
|
||||
const { getIssue, summaries } = vi.hoisted(() => ({ getIssue: vi.fn(), summaries: vi.fn() }));
|
||||
vi.mock("@/api/issues", () => ({ issuesApi: { get: getIssue, summaries } }));
|
||||
@@ -182,23 +183,62 @@ describe("recent task synchronization", () => {
|
||||
expect(summaries).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([false, true])("refreshes long histories in bounded requests without pruning on a partial failure (%s)", async fails => {
|
||||
const tasks = Array.from({ length: 13 }, (_, i) => ({ ...task(i + 1, `Task ${i}`), id: `issue-${i}` }));
|
||||
it("fetches a 25-task history in one request", async () => {
|
||||
const tasks = Array.from({ length: 25 }, (_, i) => ({ ...task(i + 1, `Task ${i}`), id: `issue-${String(i).padStart(2, "0")}` }));
|
||||
tasks.forEach(issue => recordRecentTask(issue, "user-1"));
|
||||
summaries.mockResolvedValue(tasks);
|
||||
await act(async () => root.render(<QueryClientProvider client={client()}><RecentTasks name="staging-size" /></QueryClientProvider>));
|
||||
await flush();
|
||||
expect(summaries).toHaveBeenCalledTimes(1);
|
||||
expect(summaries.mock.calls[0][1]).toHaveLength(25);
|
||||
});
|
||||
|
||||
it.each([false, true])("refreshes long histories in parallel bounded requests without pruning on a partial failure (%s)", async fails => {
|
||||
// Three chunks, all in flight together within the concurrency limit.
|
||||
const count = ISSUE_SUMMARIES_MAX_IDS * 2 + 3;
|
||||
const tasks = Array.from({ length: count }, (_, i) => ({ ...task(i + 1, `Task ${i}`), id: `issue-${String(i).padStart(3, "0")}` }));
|
||||
tasks.forEach(issue => recordRecentTask(issue, "user-1"));
|
||||
const pending: Array<() => void> = [];
|
||||
summaries.mockImplementation(async (_company, ids: string[]) => {
|
||||
expect(ids.length).toBeLessThanOrEqual(5);
|
||||
if (fails && summaries.mock.calls.length === 2) throw new ApiError("Unavailable", 500, null);
|
||||
return tasks.filter(issue => ids.includes(issue.id)).map(issue => ({ ...issue, title: `Refreshed ${issue.title}`, updatedAt: new Date(100) }));
|
||||
expect(ids.length).toBeLessThanOrEqual(ISSUE_SUMMARIES_MAX_IDS);
|
||||
const call = summaries.mock.calls.length;
|
||||
// Each chunk waits until the test releases it, so all three must be in flight together.
|
||||
await new Promise<void>(resolve => pending.push(resolve));
|
||||
if (fails && call === 2) throw new ApiError("Unavailable", 500, null);
|
||||
return tasks.filter(issue => ids.includes(issue.id)).map(issue => ({ ...issue, title: `Refreshed ${issue.title}`, updatedAt: new Date(10_000) }));
|
||||
});
|
||||
await act(async () => root.render(<QueryClientProvider client={client()}><RecentTasks name="long-history" /></QueryClientProvider>));
|
||||
await flush();
|
||||
expect(summaries).toHaveBeenCalledTimes(fails ? 2 : 3);
|
||||
expect(summaries).toHaveBeenCalledTimes(3);
|
||||
expect(pending).toHaveLength(3);
|
||||
pending.forEach(resolve => resolve());
|
||||
await flush();
|
||||
const entries = readRecentTasks(storageKey, "company-1");
|
||||
expect(entries).toHaveLength(13);
|
||||
expect(entries).toHaveLength(count);
|
||||
expect(entries.every(entry => entry.title.startsWith("Refreshed"))).toBe(!fails);
|
||||
expect(getIssue).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("limits how many summary chunks are in flight at once", async () => {
|
||||
const count = ISSUE_SUMMARIES_MAX_IDS * 5;
|
||||
const tasks = Array.from({ length: count }, (_, i) => ({ ...task(i + 1, `Task ${i}`), id: `issue-${String(i).padStart(3, "0")}` }));
|
||||
tasks.forEach(issue => recordRecentTask(issue, "user-1"));
|
||||
const pending: Array<() => void> = [];
|
||||
summaries.mockImplementation(async (_company, ids: string[]) => {
|
||||
await new Promise<void>(resolve => pending.push(resolve));
|
||||
return tasks.filter(issue => ids.includes(issue.id));
|
||||
});
|
||||
await act(async () => root.render(<QueryClientProvider client={client()}><RecentTasks name="many" /></QueryClientProvider>));
|
||||
await flush();
|
||||
expect(summaries).toHaveBeenCalledTimes(RECENT_TASK_SUMMARY_CONCURRENCY);
|
||||
while (pending.length) {
|
||||
pending.splice(0).forEach(resolve => resolve());
|
||||
await flush();
|
||||
}
|
||||
expect(summaries).toHaveBeenCalledTimes(5);
|
||||
expect(readRecentTasks(storageKey, "company-1")).toHaveLength(count);
|
||||
});
|
||||
|
||||
it("prunes successful omissions without resurrecting cached full details", async () => {
|
||||
const missing = task(10, "Removed");
|
||||
recordRecentTask(missing, "user-1");
|
||||
|
||||
@@ -20,6 +20,8 @@ type RecentTasksUpdatedDetail = {
|
||||
};
|
||||
|
||||
const RECENT_TASK_ORDER_DEBOUNCE_MS = 1_000;
|
||||
/** Summary chunks in flight at once for a history longer than one request. */
|
||||
export const RECENT_TASK_SUMMARY_CONCURRENCY = 3;
|
||||
|
||||
export function useRecentTasks({
|
||||
companyId,
|
||||
@@ -82,11 +84,22 @@ export function useRecentTasks({
|
||||
const summaryQuery = useQuery({
|
||||
queryKey: summaryKey,
|
||||
queryFn: async ({ signal }) => {
|
||||
const summaries: IssueSummary[] = [];
|
||||
// One request covers a typical list. Longer lists fetch their chunks a
|
||||
// few at a time rather than one after another or all at once.
|
||||
const chunks: string[][] = [];
|
||||
for (let offset = 0; offset < queryIds.length; offset += ISSUE_SUMMARIES_MAX_IDS) {
|
||||
summaries.push(...await issuesApi.summaries(companyId!, queryIds.slice(offset, offset + ISSUE_SUMMARIES_MAX_IDS), { signal }));
|
||||
chunks.push(queryIds.slice(offset, offset + ISSUE_SUMMARIES_MAX_IDS));
|
||||
}
|
||||
return summaries;
|
||||
const results: IssueSummary[][] = new Array(chunks.length);
|
||||
let next = 0;
|
||||
const worker = async () => {
|
||||
while (next < chunks.length) {
|
||||
const index = next++;
|
||||
results[index] = await issuesApi.summaries(companyId!, chunks[index]!, { signal });
|
||||
}
|
||||
};
|
||||
await Promise.all(Array.from({ length: Math.min(RECENT_TASK_SUMMARY_CONCURRENCY, chunks.length) }, worker));
|
||||
return results.flat();
|
||||
},
|
||||
enabled: Boolean(companyId && queryIds.length),
|
||||
staleTime: 30_000,
|
||||
|
||||
Reference in new issue
Block a user