mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
perf(recovery): check only blocked tasks whose blockers are done (#21517)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues in the background. One queue, `resolved-dependencies`, wakes a blocked task when all of its blockers are done. > - The queue reloads every blocked task of a company when the company gets a recovery hint. Every run dispatch and run state change sends that hint. > - Each reloaded task gets a dependency check of about five statements, also when its blockers are still open and nothing can happen. > - On a hosted instance with 577 blocked tasks and 5 live runs, this loop ran all the time: five statements at 7.3 per second each. > - This pull request loads only the blocked tasks whose every blocker is done. These are the only tasks the check can wake. > - The benefit is that the cost of this queue no longer grows with the number of blocked tasks that wait on open work. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The background check that wakes blocked tasks when their blockers are done (`resolved-dependencies` in `server/src/services/recovery/work.ts`). **Current behavior** A company recovery hint reloads every blocked, agent-assigned task of the company. Run dispatch and every run state change send this hint, so with live runs it comes all the time. Each task then gets `reconcileResolvedDependencyWakeBackstop` with its issue id. That reads the task, reads the dependency readiness, and then reads the stall wakes of the task (children, blockers, recovery actions). For a task with open blockers this finds nothing to do. The queue reloads at most every 30 seconds per hint and works 25 tasks per pass with 1 second between passes, so with hundreds of blocked tasks it never becomes idle. **Proposed behavior** The queue loads only blocked tasks that have at least one blocker and no blocker that is not done. The per-task check is unchanged, so the workspace-finalize barrier and the existing-wake dedup still apply. A blocked task enters the queue when its last blocker becomes done, because that change sends the company hint and the next reload now finds it. Stall wakes for tasks with open blockers come from the company pass of the `active-recovery-actions` queue (`reconcileActiveRecoveryActions`). That pass reads every stall of the company with the same predicate, and it gets the same company hint. **Reason and benefit** Background work must not grow with the backlog of open tasks. With 300 blocked tasks behind one open blocker, a company hint now checks 0 of them instead of 300. ## What Changed - `server/src/services/recovery/work.ts`: the shared `register` helper takes an optional extra SQL condition. The `resolved-dependencies` queue gives a condition that keeps only blocked tasks with at least one `blocks` relation and only `done` blockers. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: a new test with 300 blocked tasks behind an open blocker and 1 blocked task behind a done blocker. It runs the real coordinator, notification and SQL. ## Verification - `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts -t "blockers are done"` (in `server/`). Before the change it fails: the first pass checks all 301 tasks instead of the 1 ready task. After the change it passes. The test also shows that the 300 tasks are checked as soon as their blocker becomes done. - `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts src/__tests__/issue-recovery-work-lifecycle.test.ts src/__tests__/issue-blocker-stall-wakeups.test.ts` pass. The stall test file covers the stall wakes through the company pass. - `pnpm check:module-boundaries` passes. - Local benchmark: the real server against PostgreSQL with statement logging, one company, 60 agents, about 1,440 open tasks (780 in review, 580 blocked with blocker relations, 50 in progress, 30 to do), with run churn from the recovery queues. Results are in the table below. The benchmark table follows in an update of this description. ## Risks - Low risk. The check itself is unchanged. The change only removes tasks from the queue that the check skipped as not ready. - A blocked task whose only blocker is cancelled stays out of the queue. The check also treated it as not ready, because only a done blocker resolves a dependent. - Stall wakes for blocked tasks with open blockers now come only from the company recovery-actions pass. That pass uses a superset of the stall predicate. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e5ad224cc5
commit
de80299d2c
4 files changed
+85
-5
No files matched your search
@@ -59,6 +59,28 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
|
||||
return { companyId, parentAgent, childAgent, parent, child, action };
|
||||
}
|
||||
|
||||
it("continues a company scan past a full page of stalls, also after a restart", async () => {
|
||||
const f = await fixture();
|
||||
const more = await db.insert(issues).values([1, 2].map(index => ({ companyId: f.companyId, title: `Child ${index}`,
|
||||
status: "blocked", parentId: f.parent.id, assigneeAgentId: f.childAgent }))).returning();
|
||||
await db.insert(issueRecoveryActions).values(more.map(child => ({ companyId: f.companyId, sourceIssueId: child.id,
|
||||
kind: "run_recovery", status: "escalated", ownerType: "board", cause: "execution_recovery_budget_exhausted",
|
||||
fingerprint: `fixture-${child.id}`, nextAction: "Retry the run or reassign the task" })));
|
||||
const { calls, enqueueWakeup } = wakeRecorder();
|
||||
const stallWakes = () => new Set(calls.filter(call => call.input.reason === ISSUE_BLOCKER_STALLED_WAKE_REASON)
|
||||
.map(call => call.input.idempotencyKey));
|
||||
const first = await wakeWaitersOfStalledIssues(db, enqueueWakeup as never, { companyId: f.companyId, maxStalls: 2 });
|
||||
expect(stallWakes().size).toBe(2);
|
||||
expect(first.retryAt).not.toBeNull();
|
||||
vi.resetModules();
|
||||
const restarted = await import("../services/recovery/blocker-stall-wakeups.js");
|
||||
for (let pass = 0; pass < 5; pass++) {
|
||||
const result = await restarted.wakeWaitersOfStalledIssues(db, enqueueWakeup as never, { companyId: f.companyId, maxStalls: 2 });
|
||||
if (result.retryAt === null) break;
|
||||
}
|
||||
expect(stallWakes().size).toBe(3);
|
||||
});
|
||||
|
||||
it("wakes the parent's assignee once per stall, with the stalled child and its recovery action", async () => {
|
||||
const f = await fixture();
|
||||
const { calls, enqueueWakeup } = wakeRecorder();
|
||||
|
||||
@@ -110,6 +110,56 @@ describeIssueRecovery("issue recovery production producers and deadlines", () =>
|
||||
expect(new Set(checked).size).toBe(1000);
|
||||
await quiet();
|
||||
});
|
||||
it("checks only blocked tasks whose blockers are done, however many still wait", async () => {
|
||||
const f = await fixture();
|
||||
const [openBlocker, doneBlocker] = await state.db.insert(issues).values([
|
||||
{ companyId: f.companyId, title: "Open blocker", status: "in_progress", assigneeAgentId: f.agentId },
|
||||
{ companyId: f.companyId, title: "Done blocker", status: "done" },
|
||||
]).returning({ id: issues.id });
|
||||
const waiting = await state.db.insert(issues).values(Array.from({ length: 300 }, (_, index) => ({
|
||||
companyId: f.companyId, title: `Waiting ${index}`, status: "blocked", assigneeAgentId: f.agentId,
|
||||
}))).returning({ id: issues.id });
|
||||
const [ready] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Ready", status: "blocked",
|
||||
assigneeAgentId: f.agentId }).returning({ id: issues.id });
|
||||
await state.db.insert(issueRelations).values([
|
||||
...waiting.map(row => ({ companyId: f.companyId, issueId: openBlocker.id, relatedIssueId: row.id, type: "blocks" as const })),
|
||||
{ companyId: f.companyId, issueId: doneBlocker.id, relatedIssueId: ready.id, type: "blocks" as const },
|
||||
]);
|
||||
const checked: string[] = [];
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
||||
state.coordinator = createDeliveryWorkCoordinator({ owner: state.db, canRun: () => true, onError: error => state.errors.push(error) });
|
||||
const register = state.coordinator.register.bind(state.coordinator);
|
||||
state.coordinator.register = (queue, task) => register(queue, { ...task, run: signal => {
|
||||
const pass = task.run(signal); state.passes.add(pass);
|
||||
void pass.then(() => state.passes.delete(pass), () => state.passes.delete(pass));
|
||||
return pass;
|
||||
} });
|
||||
state.worker = registerIssueRecoveryWork(state.db, state.coordinator, {
|
||||
stranded: async () => {}, orphanBlockers: async () => {}, recoveryActions: async () => {},
|
||||
dependencies: async scope => { checked.push(scope.issueId); return { enqueueFailed: 0 }; }, locks: async () => {},
|
||||
watchdogs: async () => ({ incomplete: 0 }), schedulingSuppressed: async () => false,
|
||||
}, () => true);
|
||||
await state.worker.ready;
|
||||
async function settleWork() {
|
||||
for (let step = 0; step < 120; step++) {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await Promise.allSettled([...state.passes]); await immediate();
|
||||
if (state.coordinator.nextWakeAt() === null && idleWorkSnapshot().active === 0) return;
|
||||
}
|
||||
throw new Error("Recovery did not settle its committed work within two virtual minutes");
|
||||
}
|
||||
await settleWork();
|
||||
expect(checked).toEqual([ready.id]);
|
||||
checked.length = 0;
|
||||
await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
|
||||
await settleWork();
|
||||
expect(checked).toEqual([ready.id]);
|
||||
checked.length = 0;
|
||||
await issueService(state.db).update(openBlocker.id, { status: "done" });
|
||||
await settleWork();
|
||||
expect(new Set(checked)).toEqual(new Set([ready.id, ...waiting.map(row => row.id)]));
|
||||
await quiet();
|
||||
});
|
||||
it("uses ordinary watchdog configuration and subtree closure to dispatch a reusable review", async () => {
|
||||
const f = await fixture();
|
||||
const source = await issueService(state.db).create(f.companyId, { title: "Finished subtree", status: "done" });
|
||||
|
||||
@@ -288,8 +288,10 @@ export async function wakeWaitersOfStalledIssues(db: Db, enqueueWakeup: BlockerS
|
||||
} else {
|
||||
const cursorKey = opts.companyId ?? "*";
|
||||
stalled = await readStalls(db, { companyId: opts.companyId, afterActionId: companyCursors.get(cursorKey) ?? null }, maxStalls);
|
||||
if (stalled.length >= maxStalls) companyCursors.set(cursorKey, stalled.at(-1)!.actionId);
|
||||
else companyCursors.delete(cursorKey);
|
||||
if (stalled.length >= maxStalls) {
|
||||
companyCursors.set(cursorKey, stalled.at(-1)!.actionId);
|
||||
retryAt(Date.now());
|
||||
} else companyCursors.delete(cursorKey);
|
||||
}
|
||||
const access = authorizationService(db);
|
||||
const agentCanRead = async (agentId: string, onBehalfOfUserId: string | null, stall: StallRow) => (await access.decide({
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { registerTaskWatchdogWork } from "../task-watchdog-work.js";
|
||||
import { and, asc, eq, gt, inArray, isNotNull, isNull, or, sql } from "drizzle-orm";
|
||||
import { and, asc, eq, gt, inArray, isNotNull, isNull, or, sql, type SQL } from "drizzle-orm";
|
||||
import { issueRecoveryActions, issueRelations, issues, type Db } from "@paperclipai/db";
|
||||
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES, type DeliveryQueue } from "../delivery-work-notifications.js";
|
||||
@@ -31,7 +31,7 @@ export function registerIssueRecoveryWork(
|
||||
},
|
||||
canRun: () => boolean,
|
||||
) {
|
||||
function register(queue: DeliveryQueue, statuses: string[] | null, check: (scope: Scope) => Promise<unknown>) {
|
||||
function register(queue: DeliveryQueue, statuses: string[] | null, check: (scope: Scope) => Promise<unknown>, eligible?: SQL) {
|
||||
const cursors = new Map<string, string>();
|
||||
let enabled = false;
|
||||
return registerScopedDeadlineWork<Entry, IssueRecoveryScope>(coordinator, {
|
||||
@@ -52,6 +52,7 @@ export function registerIssueRecoveryWork(
|
||||
statuses ? isNull(issues.hiddenAt) : undefined,
|
||||
statuses ? isNull(issues.assigneeUserId) : undefined,
|
||||
statuses ? or(isNotNull(issues.assigneeAgentId), eq(issues.status, "in_review")) : undefined,
|
||||
eligible,
|
||||
)).orderBy(asc(issues.id)).limit(PAGE_SIZE);
|
||||
const complete = rows.length < PAGE_SIZE;
|
||||
if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id);
|
||||
@@ -154,7 +155,12 @@ export function registerIssueRecoveryWork(
|
||||
const dependencies = register(DELIVERY_QUEUES.resolvedDependencies, ["blocked"], async scope => {
|
||||
const result = await owner.dependencies(scope);
|
||||
if (result.enqueueFailed > 0) throw new Error("Resolved dependency wake remains undelivered");
|
||||
});
|
||||
}, sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId}
|
||||
and r.related_issue_id = ${issues.id} and r.type = 'blocks')
|
||||
and not exists (select 1 from ${issueRelations} r inner join ${issues} blocker
|
||||
on blocker.id = r.issue_id and blocker.company_id = r.company_id
|
||||
where r.company_id = ${issues.companyId} and r.related_issue_id = ${issues.id}
|
||||
and r.type = 'blocks' and blocker.status <> 'done')`);
|
||||
const locks = register(DELIVERY_QUEUES.staleIssueLocks, null, owner.locks);
|
||||
const assignedWork = owner.assignedWork ? registerAssignedWork(owner.assignedWork) : null;
|
||||
const watchdogs = registerTaskWatchdogWork(db, coordinator, {
|
||||
|
||||
Reference in new issue
Block a user