fix(recovery): keep the resolved-dependency probes per task whatever the statistics (#21574)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `resolved-dependencies` recovery queue loads the blocked tasks
whose blockers are all done (#21517). It runs this load on recovery
hints.
> - The load has two correlated subqueries on `issue_relations`.
PostgreSQL turns them into a semi join and an anti join, and lets the
statistics choose how.
> - After a bulk change in a company that the statistics have not seen
(a large import, a restore, a big cleanup), the planner estimates one
row for the company. It then picks nested loops whose inner side scans
every relation of the company and filters by task id. On a local copy
with 5,000 open tasks the query ran for more than 20 seconds (once 19
minutes) until the table was analyzed; after `ANALYZE` it takes 3.7 ms.
> - This pull request keeps both subqueries as per-task subplans, so
each one is an index probe by (company, task id) whatever the statistics
say.
> - The benefit is that the load stays proportional to the blocked tasks
it reads, also right after bulk changes. This is prevention: the hosted
instance does not show this query as slow today.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On a local database that had been analyzed with a smaller dataset and
then loaded with 5,000 open tasks in a new company, the
`resolved-dependencies` load query did not finish within 20 seconds; in
a benchmark run it stayed active for 19 minutes after its server
stopped.

**Steps to reproduce**

1. Load a company with about 1,500 tasks and run `ANALYZE issues,
issue_relations`.
2. With autovacuum off for these tables, load another company with 5,000
tasks, 1,900 of them blocked with `blocks` relations.
3. Run the load query of `resolved-dependencies` for the new company.

Plan with stale statistics (abridged):

```
Limit
  ->  Nested Loop Semi Join  (rows=1)
        Join Filter: (issues.id = r.related_issue_id)
        ->  Nested Loop Anti Join  (rows=1)
              Join Filter: (r_1.related_issue_id = issues.id)
              ->  Index Scan using issues_company_id_uq on issues  Index Cond: (company_id = ...)
              ->  Nested Loop
                    ->  Index Scan using issue_relations_company_related_issue_idx on issue_relations r_1
                          Index Cond: (company_id = ...)
                    ->  Index Scan using issues_company_id_uq on issues blocker  Index Cond: (company_id = ...)
        ->  Index Scan using issue_relations_company_related_issue_idx on issue_relations r  Index Cond: (company_id = ...)
```

The index condition holds only `company_id`. The task id is a join
filter, so each blocked task rescans every relation (and every task, for
the blocker) of the company.

**Expected behavior**

Each blocked task probes its own relations through
`issue_relations_company_related_issue_idx` (company_id,
related_issue_id) and its blockers through `issues_company_id_uq`
(company_id, id).

## What Changed

- `server/src/services/recovery/work.ts`: the candidate predicate is a
named export, `resolvedDependencyCandidate`, and both subqueries end
with `offset 0`. PostgreSQL does not pull a subquery with `OFFSET` up
into a join, so it stays a correlated subplan. Its index condition then
includes the outer task id. The predicate and its results are otherwise
unchanged.
- `server/src/__tests__/resolved-dependency-candidates.test.ts`: two
tests.

Indexes that the subplans use (all existing):
`issue_relations_company_related_issue_idx` (company_id,
related_issue_id) for both relation probes, and `issues_company_id_uq`
(company_id, id) for the blocker status.
`issue_relations_company_issue_idx` (company_id, issue_id) is not needed
here.

## Verification

- `selects the same tasks as the earlier predicate`: 40 blocked tasks
with a done blocker, an open blocker, a done and a cancelled blocker, or
no blocker. The new predicate returns the same 10 tasks as the earlier
one.
- `probes the relations of each task, whatever the planner statistics
say`: analyzes a small company, then loads 3,000 blocked tasks in a new
company. The plan (`EXPLAIN (FORMAT JSON)`) has two subplans and no semi
or anti join, and the load returns 100 rows within a 5-second statement
timeout. I saw it fail first with the earlier predicate: the plan had a
semi join and an anti join.
- Local plan on the stale database: EXPLAIN ANALYZE shows `SubPlan 1`
and `SubPlan 2` with `Index Cond: ((company_id = issues.company_id) AND
(related_issue_id = issues.id))`, 1,933 probes each, 4.9 ms.
- `issue-recovery-deadlines.test.ts` and
`issue-recovery-dependencies.test.ts` pass (14 tests). `pnpm
check:module-boundaries` passes.

## Risks

- With fresh statistics the planner could use a merge or hash join for a
large page; the subplan form gives up that choice. Each blocked task now
costs two index probes, which is the same order as the fresh-statistics
plan (3.7 ms before, 4.9 ms after on 1,933 blocked tasks, measured once
each on a loaded machine).
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 01:28:55 -06:00
1 parent 6b6c94a4ea
commit c1d3f90b37
2 files changed
+70 -6

No files matched your search

@@ -0,0 +1,62 @@
import { and, eq, inArray, sql } from "drizzle-orm";
import { expect, it } from "vitest";
import { issueRelations, issues } from "@paperclipai/db";
import { resolvedDependencyCandidate } from "../services/recovery/work.js";
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
const earlierPredicate = sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId}
and r.related_issue_id = ${issues.id} and r.type = 'blocks')
and not exists (select 1 from ${issueRelations} r inner join ${issues} blocker
on blocker.id = r.issue_id and blocker.company_id = r.company_id
where r.company_id = ${issues.companyId} and r.related_issue_id = ${issues.id}
and r.type = 'blocks' and blocker.status <> 'done')`;
describeIssueRecovery("resolved dependency candidates", () => {
const state = issueRecoveryFixture();
async function blockedGraph(companyId: string, agentId: string, count: number) {
const blockers = await state.db.insert(issues).values([
{ companyId, title: "Open blocker", status: "in_progress", assigneeAgentId: agentId },
{ companyId, title: "Done blocker", status: "done" },
{ companyId, title: "Cancelled blocker", status: "cancelled" },
]).returning({ id: issues.id });
const blocked = await state.db.insert(issues).values(Array.from({ length: count }, (_, index) => ({
companyId, title: `Blocked ${index}`, status: "blocked", assigneeAgentId: agentId,
}))).returning({ id: issues.id });
const relations = blocked.flatMap((row, index) => {
const shape = index % 4;
if (shape === 0) return [{ blocker: 1, row }];
if (shape === 1) return [{ blocker: 0, row }];
if (shape === 2) return [{ blocker: 1, row }, { blocker: 2, row }];
return [];
}).map(({ blocker, row }) => ({ companyId, issueId: blockers[blocker].id, relatedIssueId: row.id, type: "blocks" as const }));
for (let index = 0; index < relations.length; index += 500) await state.db.insert(issueRelations).values(relations.slice(index, index + 500));
}
const candidates = (companyId: string, predicate: ReturnType<typeof sql>) => state.db.select({ id: issues.id }).from(issues)
.where(and(eq(issues.companyId, companyId), inArray(issues.status, ["blocked"]), predicate)).orderBy(issues.id);
it("selects the same tasks as the earlier predicate", async () => {
const f = await state.fixture();
await blockedGraph(f.companyId, f.agentId, 40);
const current = await candidates(f.companyId, resolvedDependencyCandidate);
expect(current).toEqual(await candidates(f.companyId, earlierPredicate));
expect(current).toHaveLength(10);
});
it("probes the relations of each task, whatever the planner statistics say", async () => {
const small = await state.fixture();
await blockedGraph(small.companyId, small.agentId, 8);
await state.db.execute(sql`analyze issues`);
await state.db.execute(sql`analyze issue_relations`);
const large = await state.fixture();
await blockedGraph(large.companyId, large.agentId, 3000);
const query = candidates(large.companyId, resolvedDependencyCandidate).limit(100).toSQL();
const [{ "QUERY PLAN": plan }] = await state.db.$client.unsafe(`explain (format json) ${query.sql}`, query.params as never[]) as Array<{ "QUERY PLAN": unknown }>;
const text = JSON.stringify(plan);
expect(text).not.toMatch(/"Join Type":"(Semi|Anti)"/);
expect(text.match(/"Subplan Name"/g)).toHaveLength(2);
const rows = await state.db.transaction(async tx => {
await tx.execute(sql`set local statement_timeout = '5s'`);
return tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, large.companyId),
inArray(issues.status, ["blocked"]), resolvedDependencyCandidate)).limit(100);
});
expect(rows).toHaveLength(100);
});
});
+8 -6
View File
@@ -11,6 +11,13 @@ type Entry = { id: string; companyId: string; dueAt: number };
type Scope = { companyId: string; issueId: string; onDeferredDeadline: (at: number) => void };
const PAGE_SIZE = 100;
export const resolvedDependencyCandidate = sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId}
and r.related_issue_id = ${issues.id} and r.type = 'blocks' offset 0)
and not exists (select 1 from ${issueRelations} r inner join ${issues} blocker
on blocker.id = r.issue_id and blocker.company_id = r.company_id
where r.company_id = ${issues.companyId} and r.related_issue_id = ${issues.id}
and r.type = 'blocks' and blocker.status <> 'done' offset 0)`;
export function registerIssueRecoveryWork(
db: Db,
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
@@ -155,12 +162,7 @@ export function registerIssueRecoveryWork(
const dependencies = register(DELIVERY_QUEUES.resolvedDependencies, ["blocked"], async scope => {
const result = await owner.dependencies(scope);
if (result.enqueueFailed > 0) throw new Error("Resolved dependency wake remains undelivered");
}, sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId}
and r.related_issue_id = ${issues.id} and r.type = 'blocks')
and not exists (select 1 from ${issueRelations} r inner join ${issues} blocker
on blocker.id = r.issue_id and blocker.company_id = r.company_id
where r.company_id = ${issues.companyId} and r.related_issue_id = ${issues.id}
and r.type = 'blocks' and blocker.status <> 'done')`);
}, resolvedDependencyCandidate);
const locks = register(DELIVERY_QUEUES.staleIssueLocks, null, owner.locks);
const assignedWork = owner.assignedWork ? registerAssignedWork(owner.assignedWork) : null;
const watchdogs = registerTaskWatchdogWork(db, coordinator, {