mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
fix(recovery): keep the resolved-dependency probes per task whatever the statistics (#21574)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The `resolved-dependencies` recovery queue loads the blocked tasks whose blockers are all done (#21517). It runs this load on recovery hints. > - The load has two correlated subqueries on `issue_relations`. PostgreSQL turns them into a semi join and an anti join, and lets the statistics choose how. > - After a bulk change in a company that the statistics have not seen (a large import, a restore, a big cleanup), the planner estimates one row for the company. It then picks nested loops whose inner side scans every relation of the company and filters by task id. On a local copy with 5,000 open tasks the query ran for more than 20 seconds (once 19 minutes) until the table was analyzed; after `ANALYZE` it takes 3.7 ms. > - This pull request keeps both subqueries as per-task subplans, so each one is an index probe by (company, task id) whatever the statistics say. > - The benefit is that the load stays proportional to the blocked tasks it reads, also right after bulk changes. This is prevention: the hosted instance does not show this query as slow today. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a local database that had been analyzed with a smaller dataset and then loaded with 5,000 open tasks in a new company, the `resolved-dependencies` load query did not finish within 20 seconds; in a benchmark run it stayed active for 19 minutes after its server stopped. **Steps to reproduce** 1. Load a company with about 1,500 tasks and run `ANALYZE issues, issue_relations`. 2. With autovacuum off for these tables, load another company with 5,000 tasks, 1,900 of them blocked with `blocks` relations. 3. Run the load query of `resolved-dependencies` for the new company. Plan with stale statistics (abridged): ``` Limit -> Nested Loop Semi Join (rows=1) Join Filter: (issues.id = r.related_issue_id) -> Nested Loop Anti Join (rows=1) Join Filter: (r_1.related_issue_id = issues.id) -> Index Scan using issues_company_id_uq on issues Index Cond: (company_id = ...) -> Nested Loop -> Index Scan using issue_relations_company_related_issue_idx on issue_relations r_1 Index Cond: (company_id = ...) -> Index Scan using issues_company_id_uq on issues blocker Index Cond: (company_id = ...) -> Index Scan using issue_relations_company_related_issue_idx on issue_relations r Index Cond: (company_id = ...) ``` The index condition holds only `company_id`. The task id is a join filter, so each blocked task rescans every relation (and every task, for the blocker) of the company. **Expected behavior** Each blocked task probes its own relations through `issue_relations_company_related_issue_idx` (company_id, related_issue_id) and its blockers through `issues_company_id_uq` (company_id, id). ## What Changed - `server/src/services/recovery/work.ts`: the candidate predicate is a named export, `resolvedDependencyCandidate`, and both subqueries end with `offset 0`. PostgreSQL does not pull a subquery with `OFFSET` up into a join, so it stays a correlated subplan. Its index condition then includes the outer task id. The predicate and its results are otherwise unchanged. - `server/src/__tests__/resolved-dependency-candidates.test.ts`: two tests. Indexes that the subplans use (all existing): `issue_relations_company_related_issue_idx` (company_id, related_issue_id) for both relation probes, and `issues_company_id_uq` (company_id, id) for the blocker status. `issue_relations_company_issue_idx` (company_id, issue_id) is not needed here. ## Verification - `selects the same tasks as the earlier predicate`: 40 blocked tasks with a done blocker, an open blocker, a done and a cancelled blocker, or no blocker. The new predicate returns the same 10 tasks as the earlier one. - `probes the relations of each task, whatever the planner statistics say`: analyzes a small company, then loads 3,000 blocked tasks in a new company. The plan (`EXPLAIN (FORMAT JSON)`) has two subplans and no semi or anti join, and the load returns 100 rows within a 5-second statement timeout. I saw it fail first with the earlier predicate: the plan had a semi join and an anti join. - Local plan on the stale database: EXPLAIN ANALYZE shows `SubPlan 1` and `SubPlan 2` with `Index Cond: ((company_id = issues.company_id) AND (related_issue_id = issues.id))`, 1,933 probes each, 4.9 ms. - `issue-recovery-deadlines.test.ts` and `issue-recovery-dependencies.test.ts` pass (14 tests). `pnpm check:module-boundaries` passes. ## Risks - With fresh statistics the planner could use a merge or hash join for a large page; the subplan form gives up that choice. Each blocked task now costs two index probes, which is the same order as the fresh-statistics plan (3.7 ms before, 4.9 ms after on 1,933 blocked tasks, measured once each on a loaded machine). - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6b6c94a4ea
commit
c1d3f90b37
2 files changed
+70
-6
No files matched your search
@@ -0,0 +1,62 @@
|
||||
import { and, eq, inArray, sql } from "drizzle-orm";
|
||||
import { expect, it } from "vitest";
|
||||
import { issueRelations, issues } from "@paperclipai/db";
|
||||
import { resolvedDependencyCandidate } from "../services/recovery/work.js";
|
||||
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
|
||||
|
||||
const earlierPredicate = sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId}
|
||||
and r.related_issue_id = ${issues.id} and r.type = 'blocks')
|
||||
and not exists (select 1 from ${issueRelations} r inner join ${issues} blocker
|
||||
on blocker.id = r.issue_id and blocker.company_id = r.company_id
|
||||
where r.company_id = ${issues.companyId} and r.related_issue_id = ${issues.id}
|
||||
and r.type = 'blocks' and blocker.status <> 'done')`;
|
||||
|
||||
describeIssueRecovery("resolved dependency candidates", () => {
|
||||
const state = issueRecoveryFixture();
|
||||
async function blockedGraph(companyId: string, agentId: string, count: number) {
|
||||
const blockers = await state.db.insert(issues).values([
|
||||
{ companyId, title: "Open blocker", status: "in_progress", assigneeAgentId: agentId },
|
||||
{ companyId, title: "Done blocker", status: "done" },
|
||||
{ companyId, title: "Cancelled blocker", status: "cancelled" },
|
||||
]).returning({ id: issues.id });
|
||||
const blocked = await state.db.insert(issues).values(Array.from({ length: count }, (_, index) => ({
|
||||
companyId, title: `Blocked ${index}`, status: "blocked", assigneeAgentId: agentId,
|
||||
}))).returning({ id: issues.id });
|
||||
const relations = blocked.flatMap((row, index) => {
|
||||
const shape = index % 4;
|
||||
if (shape === 0) return [{ blocker: 1, row }];
|
||||
if (shape === 1) return [{ blocker: 0, row }];
|
||||
if (shape === 2) return [{ blocker: 1, row }, { blocker: 2, row }];
|
||||
return [];
|
||||
}).map(({ blocker, row }) => ({ companyId, issueId: blockers[blocker].id, relatedIssueId: row.id, type: "blocks" as const }));
|
||||
for (let index = 0; index < relations.length; index += 500) await state.db.insert(issueRelations).values(relations.slice(index, index + 500));
|
||||
}
|
||||
const candidates = (companyId: string, predicate: ReturnType<typeof sql>) => state.db.select({ id: issues.id }).from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), inArray(issues.status, ["blocked"]), predicate)).orderBy(issues.id);
|
||||
it("selects the same tasks as the earlier predicate", async () => {
|
||||
const f = await state.fixture();
|
||||
await blockedGraph(f.companyId, f.agentId, 40);
|
||||
const current = await candidates(f.companyId, resolvedDependencyCandidate);
|
||||
expect(current).toEqual(await candidates(f.companyId, earlierPredicate));
|
||||
expect(current).toHaveLength(10);
|
||||
});
|
||||
it("probes the relations of each task, whatever the planner statistics say", async () => {
|
||||
const small = await state.fixture();
|
||||
await blockedGraph(small.companyId, small.agentId, 8);
|
||||
await state.db.execute(sql`analyze issues`);
|
||||
await state.db.execute(sql`analyze issue_relations`);
|
||||
const large = await state.fixture();
|
||||
await blockedGraph(large.companyId, large.agentId, 3000);
|
||||
const query = candidates(large.companyId, resolvedDependencyCandidate).limit(100).toSQL();
|
||||
const [{ "QUERY PLAN": plan }] = await state.db.$client.unsafe(`explain (format json) ${query.sql}`, query.params as never[]) as Array<{ "QUERY PLAN": unknown }>;
|
||||
const text = JSON.stringify(plan);
|
||||
expect(text).not.toMatch(/"Join Type":"(Semi|Anti)"/);
|
||||
expect(text.match(/"Subplan Name"/g)).toHaveLength(2);
|
||||
const rows = await state.db.transaction(async tx => {
|
||||
await tx.execute(sql`set local statement_timeout = '5s'`);
|
||||
return tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, large.companyId),
|
||||
inArray(issues.status, ["blocked"]), resolvedDependencyCandidate)).limit(100);
|
||||
});
|
||||
expect(rows).toHaveLength(100);
|
||||
});
|
||||
});
|
||||
@@ -11,6 +11,13 @@ type Entry = { id: string; companyId: string; dueAt: number };
|
||||
type Scope = { companyId: string; issueId: string; onDeferredDeadline: (at: number) => void };
|
||||
const PAGE_SIZE = 100;
|
||||
|
||||
export const resolvedDependencyCandidate = sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId}
|
||||
and r.related_issue_id = ${issues.id} and r.type = 'blocks' offset 0)
|
||||
and not exists (select 1 from ${issueRelations} r inner join ${issues} blocker
|
||||
on blocker.id = r.issue_id and blocker.company_id = r.company_id
|
||||
where r.company_id = ${issues.companyId} and r.related_issue_id = ${issues.id}
|
||||
and r.type = 'blocks' and blocker.status <> 'done' offset 0)`;
|
||||
|
||||
export function registerIssueRecoveryWork(
|
||||
db: Db,
|
||||
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
|
||||
@@ -155,12 +162,7 @@ export function registerIssueRecoveryWork(
|
||||
const dependencies = register(DELIVERY_QUEUES.resolvedDependencies, ["blocked"], async scope => {
|
||||
const result = await owner.dependencies(scope);
|
||||
if (result.enqueueFailed > 0) throw new Error("Resolved dependency wake remains undelivered");
|
||||
}, sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId}
|
||||
and r.related_issue_id = ${issues.id} and r.type = 'blocks')
|
||||
and not exists (select 1 from ${issueRelations} r inner join ${issues} blocker
|
||||
on blocker.id = r.issue_id and blocker.company_id = r.company_id
|
||||
where r.company_id = ${issues.companyId} and r.related_issue_id = ${issues.id}
|
||||
and r.type = 'blocks' and blocker.status <> 'done')`);
|
||||
}, resolvedDependencyCandidate);
|
||||
const locks = register(DELIVERY_QUEUES.staleIssueLocks, null, owner.locks);
|
||||
const assignedWork = owner.assignedWork ? registerAssignedWork(owner.assignedWork) : null;
|
||||
const watchdogs = registerTaskWatchdogWork(db, coordinator, {
|
||||
|
||||
Reference in new issue
Block a user