mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person opens a task from the sidebar's Recent Tasks, the Inbox or the Tasks list, and the task chat shows its first frame from a `chat-bootstrap` read > - On staging, a task opened from Recent Tasks soon after a page load took 15.3 s to show its chat. The same task from the Inbox took 0.19 s > - Two defects cause this. Every task's bootstrap paid a fixed ~1.8 s for its latest runs. Recent Tasks preloaded ten tasks in one request (17.7 s), and an open waited for that whole request > - Recent Tasks also warmed its tasks only once per page load. Unread cache entries are dropped after five minutes, so its tasks were cold for the rest of the session > - This pull request makes the runs read use indexes, lets an open read its own task instead of waiting for a ten-task preload, and re-warms Recent Tasks on navigation > - The benefit is that a task from Recent Tasks opens as fast as from a task list ## Linked Issues or Issue Description Refs #19818, Refs #16713 (earlier proposals of the same `runsForIssue` rewrite; this PR adds a query-plan regression test and the client fixes). **What happened?** On staging, a click on a Recent Tasks link about 6 s after a page load took 15.3 s until the chat thread showed. The Inbox opened the same task in 0.19 s. **Steps to reproduce** 1. Load a page with a populated Recent Tasks sidebar. 2. Within a few seconds, click a recent task. 3. The chat stays on its loading state for many seconds. **Expected behavior** The task opens about as fast as from the Inbox or the Tasks list. **Root cause (measured on staging)** - `GET /issues/:id/runs?view=recent` took 1.7–1.9 s of server time for tasks with 1 to 24 runs. Every other first-frame read took 80–350 ms. The query filtered with `issue_id = $1 OR EXISTS (activity_log … run_id = heartbeat_runs.id)`, so Postgres walked every run of the company. - The bootstrap reads two tasks at a time, so the Recents preload of ten tasks took 17.7 s. Each task in a batch settles only with the whole request, and the page's reads wait for it (up to the 5 s share deadline, then read again under load). - `useRecentTaskOpenWarmup` remembered preloaded task ids for the life of the page and never preloaded them again. ## What Changed - `activityService.runsForIssue`: select the task's runs by `id IN (runs with issue_id = task UNION run ids from the task's activity)`. Each half uses its own index. The results are the same. - `startIssueChatBootstrap`: when the only request for a task is a background batch that is still loading, send a request for that task alone. All refs of the task point to the new request. The later batch answer only fills keys that are still empty or older. - `loadIssueChatBootstraps` (background): a task whose answer did not arrive is not held back for five minutes, so the next preload can retry it. - `useRecentTaskOpenWarmup`: use the background preload (five-minute repeat, the same as the cache GC time) and run again after each navigation (`navigationKey`). It no longer keeps a per-page-load set. `SidebarRecentTasks` and `CoordinatorSidebar` pass the pathname. ## Verification - `server/src/__tests__/issue-runs-read-cost.test.ts` (new): a company with 20,003 runs, and a task with 3 runs. EXPLAIN ANALYZE of the runs reads examined **40,011** run rows before the fix and **15** after (red/green). The test also checks the run order and the run that is linked only through activity. - `ui/src/lib/issue-chat-bootstrap.test.ts`: a new test opens a task while a ten-task preload is still loading. Before the fix it timed out waiting on the batch. After the fix it reads its own request. A second new test checks that a single-task request is still shared. - `ui/src/hooks/useRecentTaskOpenWarmup.test.tsx`: a new test re-warms Recents on a navigation after five minutes (red on the old hook). Another new test sends nothing for navigations inside five minutes. - Existing suites pass: activity service/routes, task cold open, agent conversations (197 tests); sidebar, Recents, navigation prefetch, issue detail cache, IssueDetail (341 tests). - `tsc --noEmit` is clean for `server` and `ui` (apart from the pre-existing Storybook index errors). - `pnpm verify` exits 2 (an opaque server setup dependency), so CI supplies the broad coverage. - After deploy: on staging, time a Recent Tasks click a few seconds after a cold load, and time `runs?view=recent`. ## Risks - Low risk. The SQL change keeps the same filter. It changes only how Postgres finds the candidate runs. The existing activity tests cover retry, credential and conversation cases. - An open during a background preload now sends one more bootstrap request for that task. This happens only on an open or a resting hover, never for each row. - Recents now sends at most one preload per five minutes while the person navigates. It sends nothing while the tab is idle or hidden. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser measurement on staging, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
82 lines
5.3 KiB
TypeScript
82 lines
5.3 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { sql } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
|
import { activityLog, agents, applyPendingMigrations, companies, createDb, heartbeatRuns, issues } from "@paperclipai/db";
|
|
import { activityService } from "../services/activity.js";
|
|
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
|
|
|
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_ISSUE_RUNS_COST_TEST_DATABASE_URL");
|
|
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
|
type Plan = { "Relation Name"?: string; "Actual Rows": number; "Actual Loops": number;
|
|
"Rows Removed by Filter"?: number; "Rows Removed by Index Recheck"?: number; Plans?: Plan[] };
|
|
|
|
// The task chat reads a task's two latest runs before its first frame (the
|
|
// chat bootstrap and `/issues/:id/runs?view=recent`). That read must cost the
|
|
// task's own runs, not every run the company ever had.
|
|
describe.skipIf(!support.supported)("issue runs read cost", () => {
|
|
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
|
let db: ReturnType<typeof createDb>;
|
|
let capture = false;
|
|
const queries: Array<{ query: string; parameters: unknown[] }> = [];
|
|
beforeAll(async () => {
|
|
if (external) await applyPendingMigrations(external);
|
|
else temporary = await startEmbeddedPostgresTestDatabase("issue-runs-cost-");
|
|
db = createDb(external ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
|
db.$client.options.onnotice = () => {};
|
|
db.$client.options.debug = (_connection, query, parameters) => {
|
|
if (capture && /from "heartbeat_runs"/.test(query)) queries.push({ query, parameters: [...parameters] });
|
|
};
|
|
}, 30_000);
|
|
afterEach(async () => { capture = false; queries.length = 0; await db.execute(sql`truncate companies cascade`); });
|
|
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
|
|
|
|
it("reads a task's latest runs without walking the company's run history", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Busy", issuePrefix: `BR${companyId.slice(0, 6)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", role: "engineer", status: "active",
|
|
adapterType: "codex_local", adapterConfig: {}, runtimeConfig: {}, permissions: {} });
|
|
const [task, other] = await db.insert(issues).values([
|
|
{ companyId, title: "Opened task", status: "in_progress" },
|
|
{ companyId, title: "Other task", status: "in_progress" },
|
|
]).returning();
|
|
// 20,000 runs of other work, newer than the task's own runs.
|
|
await db.execute(sql`insert into heartbeat_runs (company_id, agent_id, issue_id, status, created_at)
|
|
select ${companyId}::uuid, ${agentId}::uuid, ${other!.id}::uuid, 'succeeded',
|
|
'2026-06-01T00:00:00Z'::timestamptz + n * interval '1 second' from generate_series(1, 20000) n`);
|
|
// The task's runs: two linked directly, one only through its activity.
|
|
const [direct1, direct2, viaActivity] = await db.insert(heartbeatRuns).values([
|
|
{ companyId, agentId, issueId: task!.id, status: "succeeded", createdAt: new Date("2026-01-01T00:00:01Z") },
|
|
{ companyId, agentId, issueId: task!.id, status: "succeeded", createdAt: new Date("2026-01-01T00:00:02Z") },
|
|
{ companyId, agentId, status: "succeeded", createdAt: new Date("2026-01-01T00:00:03Z") },
|
|
]).returning();
|
|
await db.insert(activityLog).values({ companyId, actorType: "agent", actorId: agentId, action: "issue.updated",
|
|
entityType: "issue", entityId: task!.id, runId: viaActivity!.id });
|
|
await db.execute(sql`analyze heartbeat_runs`);
|
|
await db.execute(sql`analyze activity_log`);
|
|
|
|
capture = true;
|
|
const recent = await activityService(db).runsForIssue(companyId, task!.id, null, 2, undefined, { skipLivenessBackfill: true });
|
|
const all = await activityService(db).runsForIssue(companyId, task!.id, null, undefined, undefined, { skipLivenessBackfill: true });
|
|
capture = false;
|
|
|
|
expect(recent.map(run => run.runId)).toEqual([viaActivity!.id, direct2!.id]);
|
|
expect(all.map(run => run.runId)).toEqual([viaActivity!.id, direct2!.id, direct1!.id]);
|
|
|
|
expect(queries.length).toBeGreaterThan(0);
|
|
let runRowsExamined = 0;
|
|
for (const statement of queries) {
|
|
const result = await db.$client.unsafe("EXPLAIN (ANALYZE, FORMAT JSON) " + statement.query, statement.parameters as never[]);
|
|
const visit = (node: Plan) => {
|
|
if (node["Relation Name"] === "heartbeat_runs") runRowsExamined += node["Actual Loops"]
|
|
* (node["Actual Rows"] + (node["Rows Removed by Filter"] ?? 0) + (node["Rows Removed by Index Recheck"] ?? 0));
|
|
node.Plans?.forEach(visit);
|
|
};
|
|
visit((result[0]!["QUERY PLAN"] as Array<{ Plan: Plan }>)[0]!.Plan);
|
|
}
|
|
process.stdout.write(JSON.stringify({ issueRunsReadCost: { companyRuns: 20003, runRowsExamined, queryCount: queries.length } }) + "\n");
|
|
// Two reads of a three-run task. The other task's 20,000 runs must not be scanned.
|
|
expect(runRowsExamined).toBeLessThanOrEqual(100);
|
|
}, 120_000);
|
|
});
|