mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The `resolved-dependencies` recovery queue loads the blocked tasks whose blockers are all done (#21517). It runs this load on recovery hints. > - The load has two correlated subqueries on `issue_relations`. PostgreSQL turns them into a semi join and an anti join, and lets the statistics choose how. > - After a bulk change in a company that the statistics have not seen (a large import, a restore, a big cleanup), the planner estimates one row for the company. It then picks nested loops whose inner side scans every relation of the company and filters by task id. On a local copy with 5,000 open tasks the query ran for more than 20 seconds (once 19 minutes) until the table was analyzed; after `ANALYZE` it takes 3.7 ms. > - This pull request keeps both subqueries as per-task subplans, so each one is an index probe by (company, task id) whatever the statistics say. > - The benefit is that the load stays proportional to the blocked tasks it reads, also right after bulk changes. This is prevention: the hosted instance does not show this query as slow today. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a local database that had been analyzed with a smaller dataset and then loaded with 5,000 open tasks in a new company, the `resolved-dependencies` load query did not finish within 20 seconds; in a benchmark run it stayed active for 19 minutes after its server stopped. **Steps to reproduce** 1. Load a company with about 1,500 tasks and run `ANALYZE issues, issue_relations`. 2. With autovacuum off for these tables, load another company with 5,000 tasks, 1,900 of them blocked with `blocks` relations. 3. Run the load query of `resolved-dependencies` for the new company. Plan with stale statistics (abridged): ``` Limit -> Nested Loop Semi Join (rows=1) Join Filter: (issues.id = r.related_issue_id) -> Nested Loop Anti Join (rows=1) Join Filter: (r_1.related_issue_id = issues.id) -> Index Scan using issues_company_id_uq on issues Index Cond: (company_id = ...) -> Nested Loop -> Index Scan using issue_relations_company_related_issue_idx on issue_relations r_1 Index Cond: (company_id = ...) -> Index Scan using issues_company_id_uq on issues blocker Index Cond: (company_id = ...) -> Index Scan using issue_relations_company_related_issue_idx on issue_relations r Index Cond: (company_id = ...) ``` The index condition holds only `company_id`. The task id is a join filter, so each blocked task rescans every relation (and every task, for the blocker) of the company. **Expected behavior** Each blocked task probes its own relations through `issue_relations_company_related_issue_idx` (company_id, related_issue_id) and its blockers through `issues_company_id_uq` (company_id, id). ## What Changed - `server/src/services/recovery/work.ts`: the candidate predicate is a named export, `resolvedDependencyCandidate`, and both subqueries end with `offset 0`. PostgreSQL does not pull a subquery with `OFFSET` up into a join, so it stays a correlated subplan. Its index condition then includes the outer task id. The predicate and its results are otherwise unchanged. - `server/src/__tests__/resolved-dependency-candidates.test.ts`: two tests. Indexes that the subplans use (all existing): `issue_relations_company_related_issue_idx` (company_id, related_issue_id) for both relation probes, and `issues_company_id_uq` (company_id, id) for the blocker status. `issue_relations_company_issue_idx` (company_id, issue_id) is not needed here. ## Verification - `selects the same tasks as the earlier predicate`: 40 blocked tasks with a done blocker, an open blocker, a done and a cancelled blocker, or no blocker. The new predicate returns the same 10 tasks as the earlier one. - `probes the relations of each task, whatever the planner statistics say`: analyzes a small company, then loads 3,000 blocked tasks in a new company. The plan (`EXPLAIN (FORMAT JSON)`) has two subplans and no semi or anti join, and the load returns 100 rows within a 5-second statement timeout. I saw it fail first with the earlier predicate: the plan had a semi join and an anti join. - Local plan on the stale database: EXPLAIN ANALYZE shows `SubPlan 1` and `SubPlan 2` with `Index Cond: ((company_id = issues.company_id) AND (related_issue_id = issues.id))`, 1,933 probes each, 4.9 ms. - `issue-recovery-deadlines.test.ts` and `issue-recovery-dependencies.test.ts` pass (14 tests). `pnpm check:module-boundaries` passes. ## Risks - With fresh statistics the planner could use a merge or hash join for a large page; the subplan form gives up that choice. Each blocked task now costs two index probes, which is the same order as the fresh-statistics plan (3.7 ms before, 4.9 ms after on 1,933 blocked tasks, measured once each on a loaded machine). - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>