From c1d3f90b3753f91a666ba15599a30c8e6ba5f574 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 01:28:55 -0600 Subject: [PATCH] fix(recovery): keep the resolved-dependency probes per task whatever the statistics (#21574) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The `resolved-dependencies` recovery queue loads the blocked tasks whose blockers are all done (#21517). It runs this load on recovery hints. > - The load has two correlated subqueries on `issue_relations`. PostgreSQL turns them into a semi join and an anti join, and lets the statistics choose how. > - After a bulk change in a company that the statistics have not seen (a large import, a restore, a big cleanup), the planner estimates one row for the company. It then picks nested loops whose inner side scans every relation of the company and filters by task id. On a local copy with 5,000 open tasks the query ran for more than 20 seconds (once 19 minutes) until the table was analyzed; after `ANALYZE` it takes 3.7 ms. > - This pull request keeps both subqueries as per-task subplans, so each one is an index probe by (company, task id) whatever the statistics say. > - The benefit is that the load stays proportional to the blocked tasks it reads, also right after bulk changes. This is prevention: the hosted instance does not show this query as slow today. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a local database that had been analyzed with a smaller dataset and then loaded with 5,000 open tasks in a new company, the `resolved-dependencies` load query did not finish within 20 seconds; in a benchmark run it stayed active for 19 minutes after its server stopped. **Steps to reproduce** 1. Load a company with about 1,500 tasks and run `ANALYZE issues, issue_relations`. 2. With autovacuum off for these tables, load another company with 5,000 tasks, 1,900 of them blocked with `blocks` relations. 3. Run the load query of `resolved-dependencies` for the new company. Plan with stale statistics (abridged): ``` Limit -> Nested Loop Semi Join (rows=1) Join Filter: (issues.id = r.related_issue_id) -> Nested Loop Anti Join (rows=1) Join Filter: (r_1.related_issue_id = issues.id) -> Index Scan using issues_company_id_uq on issues Index Cond: (company_id = ...) -> Nested Loop -> Index Scan using issue_relations_company_related_issue_idx on issue_relations r_1 Index Cond: (company_id = ...) -> Index Scan using issues_company_id_uq on issues blocker Index Cond: (company_id = ...) -> Index Scan using issue_relations_company_related_issue_idx on issue_relations r Index Cond: (company_id = ...) ``` The index condition holds only `company_id`. The task id is a join filter, so each blocked task rescans every relation (and every task, for the blocker) of the company. **Expected behavior** Each blocked task probes its own relations through `issue_relations_company_related_issue_idx` (company_id, related_issue_id) and its blockers through `issues_company_id_uq` (company_id, id). ## What Changed - `server/src/services/recovery/work.ts`: the candidate predicate is a named export, `resolvedDependencyCandidate`, and both subqueries end with `offset 0`. PostgreSQL does not pull a subquery with `OFFSET` up into a join, so it stays a correlated subplan. Its index condition then includes the outer task id. The predicate and its results are otherwise unchanged. - `server/src/__tests__/resolved-dependency-candidates.test.ts`: two tests. Indexes that the subplans use (all existing): `issue_relations_company_related_issue_idx` (company_id, related_issue_id) for both relation probes, and `issues_company_id_uq` (company_id, id) for the blocker status. `issue_relations_company_issue_idx` (company_id, issue_id) is not needed here. ## Verification - `selects the same tasks as the earlier predicate`: 40 blocked tasks with a done blocker, an open blocker, a done and a cancelled blocker, or no blocker. The new predicate returns the same 10 tasks as the earlier one. - `probes the relations of each task, whatever the planner statistics say`: analyzes a small company, then loads 3,000 blocked tasks in a new company. The plan (`EXPLAIN (FORMAT JSON)`) has two subplans and no semi or anti join, and the load returns 100 rows within a 5-second statement timeout. I saw it fail first with the earlier predicate: the plan had a semi join and an anti join. - Local plan on the stale database: EXPLAIN ANALYZE shows `SubPlan 1` and `SubPlan 2` with `Index Cond: ((company_id = issues.company_id) AND (related_issue_id = issues.id))`, 1,933 probes each, 4.9 ms. - `issue-recovery-deadlines.test.ts` and `issue-recovery-dependencies.test.ts` pass (14 tests). `pnpm check:module-boundaries` passes. ## Risks - With fresh statistics the planner could use a merge or hash join for a large page; the subplan form gives up that choice. Each blocked task now costs two index probes, which is the same order as the fresh-statistics plan (3.7 ms before, 4.9 ms after on 1,933 blocked tasks, measured once each on a loaded machine). - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../resolved-dependency-candidates.test.ts | 62 +++++++++++++++++++ server/src/services/recovery/work.ts | 14 +++-- 2 files changed, 70 insertions(+), 6 deletions(-) create mode 100644 server/src/__tests__/resolved-dependency-candidates.test.ts diff --git a/server/src/__tests__/resolved-dependency-candidates.test.ts b/server/src/__tests__/resolved-dependency-candidates.test.ts new file mode 100644 index 00000000000..f133fab36fc --- /dev/null +++ b/server/src/__tests__/resolved-dependency-candidates.test.ts @@ -0,0 +1,62 @@ +import { and, eq, inArray, sql } from "drizzle-orm"; +import { expect, it } from "vitest"; +import { issueRelations, issues } from "@paperclipai/db"; +import { resolvedDependencyCandidate } from "../services/recovery/work.js"; +import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js"; + +const earlierPredicate = sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId} + and r.related_issue_id = ${issues.id} and r.type = 'blocks') + and not exists (select 1 from ${issueRelations} r inner join ${issues} blocker + on blocker.id = r.issue_id and blocker.company_id = r.company_id + where r.company_id = ${issues.companyId} and r.related_issue_id = ${issues.id} + and r.type = 'blocks' and blocker.status <> 'done')`; + +describeIssueRecovery("resolved dependency candidates", () => { + const state = issueRecoveryFixture(); + async function blockedGraph(companyId: string, agentId: string, count: number) { + const blockers = await state.db.insert(issues).values([ + { companyId, title: "Open blocker", status: "in_progress", assigneeAgentId: agentId }, + { companyId, title: "Done blocker", status: "done" }, + { companyId, title: "Cancelled blocker", status: "cancelled" }, + ]).returning({ id: issues.id }); + const blocked = await state.db.insert(issues).values(Array.from({ length: count }, (_, index) => ({ + companyId, title: `Blocked ${index}`, status: "blocked", assigneeAgentId: agentId, + }))).returning({ id: issues.id }); + const relations = blocked.flatMap((row, index) => { + const shape = index % 4; + if (shape === 0) return [{ blocker: 1, row }]; + if (shape === 1) return [{ blocker: 0, row }]; + if (shape === 2) return [{ blocker: 1, row }, { blocker: 2, row }]; + return []; + }).map(({ blocker, row }) => ({ companyId, issueId: blockers[blocker].id, relatedIssueId: row.id, type: "blocks" as const })); + for (let index = 0; index < relations.length; index += 500) await state.db.insert(issueRelations).values(relations.slice(index, index + 500)); + } + const candidates = (companyId: string, predicate: ReturnType) => state.db.select({ id: issues.id }).from(issues) + .where(and(eq(issues.companyId, companyId), inArray(issues.status, ["blocked"]), predicate)).orderBy(issues.id); + it("selects the same tasks as the earlier predicate", async () => { + const f = await state.fixture(); + await blockedGraph(f.companyId, f.agentId, 40); + const current = await candidates(f.companyId, resolvedDependencyCandidate); + expect(current).toEqual(await candidates(f.companyId, earlierPredicate)); + expect(current).toHaveLength(10); + }); + it("probes the relations of each task, whatever the planner statistics say", async () => { + const small = await state.fixture(); + await blockedGraph(small.companyId, small.agentId, 8); + await state.db.execute(sql`analyze issues`); + await state.db.execute(sql`analyze issue_relations`); + const large = await state.fixture(); + await blockedGraph(large.companyId, large.agentId, 3000); + const query = candidates(large.companyId, resolvedDependencyCandidate).limit(100).toSQL(); + const [{ "QUERY PLAN": plan }] = await state.db.$client.unsafe(`explain (format json) ${query.sql}`, query.params as never[]) as Array<{ "QUERY PLAN": unknown }>; + const text = JSON.stringify(plan); + expect(text).not.toMatch(/"Join Type":"(Semi|Anti)"/); + expect(text.match(/"Subplan Name"/g)).toHaveLength(2); + const rows = await state.db.transaction(async tx => { + await tx.execute(sql`set local statement_timeout = '5s'`); + return tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, large.companyId), + inArray(issues.status, ["blocked"]), resolvedDependencyCandidate)).limit(100); + }); + expect(rows).toHaveLength(100); + }); +}); diff --git a/server/src/services/recovery/work.ts b/server/src/services/recovery/work.ts index 5aa16dfc9f6..00f3dbd0eb4 100644 --- a/server/src/services/recovery/work.ts +++ b/server/src/services/recovery/work.ts @@ -11,6 +11,13 @@ type Entry = { id: string; companyId: string; dueAt: number }; type Scope = { companyId: string; issueId: string; onDeferredDeadline: (at: number) => void }; const PAGE_SIZE = 100; +export const resolvedDependencyCandidate = sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId} + and r.related_issue_id = ${issues.id} and r.type = 'blocks' offset 0) + and not exists (select 1 from ${issueRelations} r inner join ${issues} blocker + on blocker.id = r.issue_id and blocker.company_id = r.company_id + where r.company_id = ${issues.companyId} and r.related_issue_id = ${issues.id} + and r.type = 'blocks' and blocker.status <> 'done' offset 0)`; + export function registerIssueRecoveryWork( db: Db, coordinator: ReturnType, @@ -155,12 +162,7 @@ export function registerIssueRecoveryWork( const dependencies = register(DELIVERY_QUEUES.resolvedDependencies, ["blocked"], async scope => { const result = await owner.dependencies(scope); if (result.enqueueFailed > 0) throw new Error("Resolved dependency wake remains undelivered"); - }, sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId} - and r.related_issue_id = ${issues.id} and r.type = 'blocks') - and not exists (select 1 from ${issueRelations} r inner join ${issues} blocker - on blocker.id = r.issue_id and blocker.company_id = r.company_id - where r.company_id = ${issues.companyId} and r.related_issue_id = ${issues.id} - and r.type = 'blocks' and blocker.status <> 'done')`); + }, resolvedDependencyCandidate); const locks = register(DELIVERY_QUEUES.staleIssueLocks, null, owner.locks); const assignedWork = owner.assignedWork ? registerAssignedWork(owner.assignedWork) : null; const watchdogs = registerTaskWatchdogWork(db, coordinator, {