From 009c8884a4fee1ef7e406ea03fdadd4ac7c40d08 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 03:05:59 -0600 Subject: [PATCH] perf(attention): read blocker retention state once per page (#21595) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The `attention-retention` queue archives old attention items. One kind is `blocker_attention`: a blocked task whose terminal blocker needs a person. > - When the queue loads a page of blocked tasks, it reads `keep` and `archived_at` from `decision_retention` for each of them, one full-row `select` per task. > - A company hint walks every page. On a hosted instance with 2,112 blocker attention items and one active run, these per-task reads ran at about 16 per second. > - This pull request reads the retention state for the whole page in one narrow statement. It also scopes three remaining run-path attention hints to the task graph. > - The benefit is one retention read per page instead of one per blocked task, with the same decisions. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The `blocker_attention` eligibility read in `server/src/services/attention-retention.ts` (`eligible()`), and three run-path producers of company-wide attention hints. **Subsystem affected** Server background retention (`attention-retention.ts`, `native-runtime/native-provider-capacity-retry.ts`, `native-runtime/native-review-participant.ts`, `native-runtime/native-workspace-reconstruction.ts`). **Current behavior** - For each blocked task whose blocker attention is terminal (and for the owner variant), `eligible()` ran `select from decision_retention where company_id = $1 and source_kind = $2 and source_id = $3`. It did this during `load`, once per task, on every company hint. - The hosted statement is identified by that column prefix: `id, company_id, source_kind, source_id, source_activity_at, …`. - I checked every other full-row reader of `decision_retention`. None fires per run event: - `syncItems`: attention summary and list requests, which a board calls once per load; retention `check` and the sweep, which run for due entries only; Keep and Archive actions. - `getState`: user actions only. - `archiveReviewedManifest`: proposal acceptance only. - `decisions.list`: only when archive proposals exist. - `native-provider-capacity-retry`, `native-review-participant` and `native-workspace-reconstruction` sent `{ companyId }` attention hints, which make the queue walk every page. **Proposed behavior** - `pageEvidence` reads `source_id`, `keep` and `archived_at` for every blocker root and every terminal blocker on the page in one statement (ids passed as one JSON parameter). `eligible()` uses that map when it has page evidence. - A missing row means "not kept, not archived", exactly as the per-task read returning no row did. - The per-entry `check` path still reads one row, now only `keep` and `archived_at`. - The three producers send `notifyIssueAttentionRetentionWork` with their task (and run). That is the task, its parents and the tasks it blocks, with a company fallback past 100. - A capacity retry changes the failed-run items of its task. - A review participant claim changes the review of its task. - A workspace reconstruction resolves the recovery action of its task, which can also change the blocker attention of the tasks it blocks. - The configuration, lifecycle, import and fallback producers keep their company hints. **Reason and benefit** Background work must not grow with the backlog. A page now costs one retention read instead of up to 100. ## What Changed - `server/src/services/attention-retention.ts`: batched and narrowed blocker retention read; `PageEvidence.blockerRetention`. - `server/src/services/native-runtime/native-provider-capacity-retry.ts`, `native-review-participant.ts`, `native-workspace-reconstruction.ts`: scoped attention hints. - Tests in `server/src/__tests__/attention-retention-work.test.ts` and `server/src/services/native-runtime/native-workspace-reconstruction.test.ts`. ## Verification - `reads blocker retention state once per page, not once per blocked task` (I saw it fail first). 160 blocked tasks, each behind its own unassigned blocker; the blockers' retention rows are kept (40), archived (40), plain (40) or missing (40). The full-size response capture and the statement log measure the load of both pages: - before: one `decision_retention` read per task (160), 26,240 result bytes; - after: 2 reads (one per page), 5,480 bytes, none of them selecting `source_activity_at`. - The same tasks are listed: only plain and missing. - `check` still drops the kept and archived ones. - `scopes the attention retention hint of a reconstruction to the task graph` (I saw it fail first): before, the reconstruction sent `{ companyId }`; after, it sends a hint that names the task. - `batches 100 blocker graphs with exact terminal timestamps and fresh terminal visibility` pinned 200 retention reads per page (one per root and variant). It now expects 1; the rest of that test is unchanged and passes. - Not separately tested: the capacity-retry and review-participant hint changes. Each is one line that passes the task the function already holds, the same pattern as the reconstruction test. - `attention-retention-work` (58), `attention-retention-candidates`, `native-workspace-reconstruction`, `native-review-participant` and `heartbeat-run-dispatch-work` pass, each file run alone (in `server/`, against dedicated local `_test` databases). - `pnpm check:module-boundaries` passes. ## Risks - The page read happens once, before the per-entry decisions of that page. Before, each decision read its row just in time. A Keep or Archive that lands between the page read and an entry's decision is seen by the next load or by `check`, which re-reads under a row lock, as before. - A run-path hint that changes more than its task graph would now miss the wider effect. The 90-day archive delay makes a missed reload cheap. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../attention-retention-work.test.ts | 50 ++++++++++++++++++- server/src/services/attention-retention.ts | 15 ++++-- .../native-provider-capacity-retry.ts | 4 +- .../native-review-participant.ts | 4 +- .../native-workspace-reconstruction.test.ts | 12 +++++ .../native-workspace-reconstruction.ts | 4 +- 6 files changed, 79 insertions(+), 10 deletions(-) diff --git a/server/src/__tests__/attention-retention-work.test.ts b/server/src/__tests__/attention-retention-work.test.ts index fbbae5a232e..e13d8a1931c 100644 --- a/server/src/__tests__/attention-retention-work.test.ts +++ b/server/src/__tests__/attention-retention-work.test.ts @@ -290,6 +290,54 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; expect(bytes("issues")).toBeLessThan(60_000); }); + it("reads blocker retention state once per page, not once per blocked task", async () => { + const blockers = await db.insert(issues).values(Array.from({ length: 160 }, () => ({ companyId, title: randomUUID(), + status: "todo", createdByAgentId: agentId, createdAt: old(), updatedAt: old() }))).returning({ id: issues.id }); + const blocked = await db.insert(issues).values(blockers.map(() => ({ companyId, title: randomUUID(), status: "blocked", + createdByAgentId: agentId, createdAt: old(), updatedAt: old() }))).returning({ id: issues.id }); + await db.insert(issueRelations).values(blocked.map((row, index) => ({ companyId, issueId: blockers[index].id, + relatedIssueId: row.id, type: "blocks" as const }))); + const at = old(); + const kept = blockers.slice(0, 40), archived = blockers.slice(40, 80), plain = blockers.slice(80, 120), missing = blockers.slice(120); + await db.insert(decisionRetention).values([ + ...kept.map(row => ({ companyId, sourceKind: "blocker_attention", sourceId: row.id, sourceActivityAt: at, keep: true })), + ...archived.map(row => ({ companyId, sourceKind: "blocker_attention", sourceId: row.id, sourceActivityAt: at, archivedAt: new Date() })), + ...plain.map(row => ({ companyId, sourceKind: "blocker_attention", sourceId: row.id, sourceActivityAt: at })), + ]); + const service = attentionRetentionService(db), scope = { companyId, sourceKind: "blocker_attention" as const }; + const observed = createDb(externalUrl ?? temporary!.connectionString), events: Array> = [], now = Date.now(); + const capture = createFullDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event)); + installDatabaseResponseCapture(observed, { decisionRetention }, capture); + const retentionReads: string[] = []; + observed.$client.options.debug = (_connection, query) => { if (query.includes('from "decision_retention"')) retentionReads.push(query); }; + const entries: Array<{ id: string; sourceId: string; rootId: string; variant?: string }> = []; + let pages = 0; + try { + const observedService = attentionRetentionService(observed); + for (; pages < 4; pages++) { + const loaded = await observedService.load(scope); + entries.push(...(loaded.rows ?? [])); + if (loaded.complete) { pages++; break; } + } + capture.flush(); + } finally { capture.flush(); await observed.$client.end(); } + const retentionBytes = events.filter(event => event.table === "decision_retention") + .reduce((total, event) => total + Number(event.responseValueBytes), 0); + expect(retentionBytes).toBeLessThan(10_000); + expect(retentionReads.length).toBeLessThanOrEqual(pages); + expect(retentionReads.every(query => !query.includes('"source_activity_at"'))).toBe(true); + const listed = new Set(entries.filter(entry => entry.variant === "terminal").map(entry => entry.sourceId)); + expect(listed).toEqual(new Set([...plain, ...missing].map(row => row.id))); + const byRoot = new Map(blocked.map((row, index) => [row.id, blockers[index].id])); + for (const root of [blocked[0].id, blocked[40].id]) { + const candidate = { sourceKind: "blocker_attention" as const, sourceId: root, companyId, activityAt: at.toISOString(), + deadlineAt: Number.MAX_SAFE_INTEGER, issueId: root, id: `blocker_attention:${root}:terminal`, rootId: root, + variant: "terminal" as const, dueAt: Number.MAX_SAFE_INTEGER }; + expect(byRoot.get(root)).toBeTruthy(); + expect(await service.check(candidate)).toBeNull(); + } + }); + it("batches 100 failed runs and rechecks late visibility and supersession before archive", async () => { const origins = await db.insert(issues).values(Array.from({ length: 100 }, () => ({ companyId, title: randomUUID(), status: "todo", updatedAt: old() }))).returning(); @@ -332,7 +380,7 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; const queries = [...statements]; expect(page.rows).toHaveLength(200); expect(page.complete).toBe(false); const retentionReads = queries.filter(query => query.includes('from "decision_retention"')); - expect(retentionReads).toHaveLength(200); + expect(retentionReads).toHaveLength(1); expect(queries.length - retentionReads.length).toBeLessThanOrEqual(30); const terminal = page.rows!.find(entry => entry.rootId === roots[99].id && entry.variant === "terminal")!; expect(terminal.sourceId).toBe(terminals[99].id); diff --git a/server/src/services/attention-retention.ts b/server/src/services/attention-retention.ts index 930d27eeba6..8f980f785d8 100644 --- a/server/src/services/attention-retention.ts +++ b/server/src/services/attention-retention.ts @@ -44,6 +44,7 @@ type PageEvidence = { newerRuns: Set; deferredRoots: Set; blockers: Awaited["listBlockerAttention"]>>; + blockerRetention: Map; }; const sourceTables: Record = { @@ -247,11 +248,18 @@ export function attentionRetentionService(db: Db) { const terminalIds = [...blockers.values()].filter(terminalBlockerRequiresAttention) .map(attention => attention.terminalBlockerIssueId).filter((id): id is string => id != null && !visible.has(id)); for (const row of await readIssues(terminalIds)) visible.set(row.id, row); + const retentionIds = [...new Set([...blockerIds, ...[...blockers.values()].filter(terminalBlockerRequiresAttention) + .map(attention => attention.terminalBlockerIssueId).filter((id): id is string => id != null)])]; + const blockerRetention = new Map(retentionIds.length ? (await database.select({ sourceId: decisionRetention.sourceId, + keep: decisionRetention.keep, archivedAt: decisionRetention.archivedAt }).from(decisionRetention).where(and( + eq(decisionRetention.companyId, companyId), eq(decisionRetention.sourceKind, "blocker_attention"), + sql`${decisionRetention.sourceId} in (select jsonb_array_elements_text(${JSON.stringify(retentionIds)}::jsonb))`))) + .map(row => [row.sourceId, { keep: row.keep, archivedAt: row.archivedAt }]) : []); result.set(companyId, { decisions: new Map(decisionRows.map(row => [row.id, row])), recoveries: new Map(recoveryRows.map(row => [row.id, row])), runs: new Map(runRows.map(row => [row.id, row])), agents: new Set(runAgents.map(row => row.id)), visibleIssues: visible, newerRuns: await listFailedRunsWithNewerRun(database, companyId, - runRows), blockers, deferredRoots: blockerBatch.deferredRootIds, + runRows), blockers, deferredRoots: blockerBatch.deferredRootIds, blockerRetention, }); } return result; @@ -307,8 +315,9 @@ export function attentionRetentionService(db: Db) { where company_id = ${candidate.companyId}::uuid and id = ${sourceId}::uuid`) as unknown as Array<{ at: string; due: string }>)[0]; activityAt = activity.at; dueAt = Number(activity.due); } - const [retention] = await database.select().from(decisionRetention).where(and(eq(decisionRetention.companyId, candidate.companyId), - eq(decisionRetention.sourceKind, candidate.sourceKind), eq(decisionRetention.sourceId, sourceId))); + const retention = evidence ? evidence.blockerRetention.get(sourceId) : (await database.select({ keep: decisionRetention.keep, + archivedAt: decisionRetention.archivedAt }).from(decisionRetention).where(and(eq(decisionRetention.companyId, candidate.companyId), + eq(decisionRetention.sourceKind, candidate.sourceKind), eq(decisionRetention.sourceId, sourceId))))[0]; if (retention?.keep || retention?.archivedAt) return null; } else if (candidate.sourceKind === "failed_run") { const run: RunProbe | undefined = evidence ? evidence.runs.get(sourceId) : (await database.select(runProbeColumns).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, candidate.companyId), eq(heartbeatRuns.id, sourceId))))[0]; diff --git a/server/src/services/native-runtime/native-provider-capacity-retry.ts b/server/src/services/native-runtime/native-provider-capacity-retry.ts index 334e5880226..a472054ee17 100644 --- a/server/src/services/native-runtime/native-provider-capacity-retry.ts +++ b/server/src/services/native-runtime/native-provider-capacity-retry.ts @@ -1,4 +1,4 @@ -import { notifyAttentionRetentionWork } from "../retention-work-notifications.js"; +import { notifyIssueAttentionRetentionWork } from "../retention-work-notifications.js"; import { notifyScheduledRetryDispatch } from "../run-dispatch-notifications.js"; import { randomUUID } from "node:crypto"; import { and, eq } from "drizzle-orm"; @@ -51,7 +51,7 @@ export async function materializeNativeProviderCapacityRetry(input: { scheduledRetryAt: dueAt.toISOString(), executionRetryAccounting: accountingForScheduledRetry(run, NATIVE_PROVIDER_OVERLOADED_CODE, attempt), }; - await notifyAttentionRetentionWork(input.tx, { companyId: input.companyId }); + await notifyIssueAttentionRetentionWork(input.tx, { companyId: input.companyId, issueId: input.issueId, runId: run.id }); await notifyScheduledRetryDispatch(input.tx, { companyId: run.companyId, runId: run.id }); const [wake] = await input.tx.insert(agentWakeupRequests).values({ companyId: run.companyId, agentId: run.agentId, source: "automation", triggerDetail: "system", diff --git a/server/src/services/native-runtime/native-review-participant.ts b/server/src/services/native-runtime/native-review-participant.ts index b51627d1352..ccd832f3ecb 100644 --- a/server/src/services/native-runtime/native-review-participant.ts +++ b/server/src/services/native-runtime/native-review-participant.ts @@ -1,5 +1,5 @@ import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; -import { notifyAttentionRetentionWork } from "../retention-work-notifications.js"; +import { notifyIssueAttentionRetentionWork } from "../retention-work-notifications.js"; import { agents, heartbeatRuns, @@ -243,7 +243,7 @@ export async function claimNativeReviewExecutionLock(db: Db, input: { || admitted.nativeReviewInteractionId !== expected.nativeReviewInteractionId || admitted.nativeReviewDecisionId !== expected.nativeReviewDecisionId) return false; await notifyStatusCardIssueChange(tx, input.companyId); - await notifyAttentionRetentionWork(tx, { companyId: input.companyId }); + await notifyIssueAttentionRetentionWork(tx, { companyId: input.companyId, issueId: input.issueId, runId: input.runId }); const claimed = await tx.update(issues).set({ executionRunId: input.runId, executionAgentNameKey: input.agentNameKey, diff --git a/server/src/services/native-runtime/native-workspace-reconstruction.test.ts b/server/src/services/native-runtime/native-workspace-reconstruction.test.ts index 8afb9820bee..08b26099219 100644 --- a/server/src/services/native-runtime/native-workspace-reconstruction.test.ts +++ b/server/src/services/native-runtime/native-workspace-reconstruction.test.ts @@ -161,6 +161,18 @@ describe("reconstruction after confirmed native workspace loss", () => { } finally { unsubscribe(); unsubscribeRoutine(); } }); + it("scopes the attention retention hint of a reconstruction to the task graph", async () => { + const f = await seed(); + const hints: string[] = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.attentionRetention, (event, values) => { + if (event === "settled") hints.push(...values ?? []); + }); + try { await db.transaction(tx => reconstructNativeWorkspace({ ...f, db: tx as unknown as Db })); } finally { unsubscribe(); } + expect(hints).not.toContain(JSON.stringify({ companyId: f.companyId })); + expect(hints.some(hint => (JSON.parse(hint) as { issueIds?: string[]; issueId?: string }).issueIds?.includes(f.issueId) + || (JSON.parse(hint) as { issueId?: string }).issueId === f.issueId)).toBe(true); + }); + it("pins a historical cancelled workspace through public heartbeat startup without replaying its turn", async () => { const f = await seed(); const later = await interveningCancelledRun(f, false); diff --git a/server/src/services/native-runtime/native-workspace-reconstruction.ts b/server/src/services/native-runtime/native-workspace-reconstruction.ts index aa20b852e14..9d95f5b22db 100644 --- a/server/src/services/native-runtime/native-workspace-reconstruction.ts +++ b/server/src/services/native-runtime/native-workspace-reconstruction.ts @@ -1,5 +1,5 @@ import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; -import { notifyAttentionRetentionWork } from "../retention-work-notifications.js"; +import { notifyIssueAttentionRetentionWork } from "../retention-work-notifications.js"; import { notifyRoutineExecutionStatus } from "../routine-execution-status-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; @@ -183,7 +183,7 @@ export async function reconstructNativeWorkspace(input: { ...(source.interveningExecutions.length ? { interveningExecutions: source.interveningExecutions } : {}) } }; await markExecutionReconciliation(db, source.action, decision, input.actorId); const now = new Date(); - await notifyAttentionRetentionWork(tx, { companyId: input.companyId }); + await notifyIssueAttentionRetentionWork(tx, { companyId: input.companyId, issueId: input.issueId }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.issueId }); await tx.update(issueRecoveryActions).set({ status: "resolved", outcome: "handed_back", resolvedAt: now, updatedAt: now, resolutionNote: "workspace_reconstruction_authorized", nextAction: "Continue from saved sources. The original unexported edits are lost; verify the selected attachments before rebuilding. Do not repeat completed external actions or reset the original work limits.",