From 61c315ebab762ace03225130734c49826d7b2578 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 23:56:53 -0600 Subject: [PATCH] perf(chat): keep GitHub review check deadlines on company-wide rebuilds (#21725) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel shows a check run on each pull request it reviews. The review-check sweep (`review-check-work.ts`) keeps one `github_review_check` action per check. It republishes the check when its status changes (queued, running, completed) and polls the pull request head on a saved deadline. > - Each time the sweep loads its checks, it computes the current check status of every `received` check (`githubReviewCheckState`: 3–5 statements) to see whether the status changed. > - Run lifecycle changes anywhere in a company send a company-wide hint. So the sweep rebuilds every company's checks every 30 s and recomputes the status of every waiting check, even when nothing about those checks changed. > - This pull request keeps the saved poll deadline on a company-wide rebuild and reads the status only on targeted hints, at startup, and in `check()`. > - The benefit is that an idle company with many waiting checks costs one page read per rebuild instead of 3–5 statements per check. ## Linked Issues or Issue Description Refs #21705, #21711 (the same class of idle reload in the GitHub response sweeps). No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On paperclip.staging at 89de6cbe3e (2026-11-15, 04:47–04:51Z, near idle at about 0.4 HTTP requests/s), the statements of `githubReviewCheckState` each ran at about 3/s: - `select … from chat_github_reviews where endpoint_id … and repository_id … and pull_number … and head_sha … order by created_at desc limit 1` (all columns: id, company_id, endpoint_id, issue_id, run_id, repository_id, …) at 3.04/s; - `select chat_message_links.comment_id from chat_deliveries join chat_message_links …` at 3.04/s; - `select id, status from heartbeat_runs where company_id = $1 and agent_id = $2 and context_snapshot->>'wakeCommentId' in (…)` at 3.04/s. Staging has 84 GitHub review checks waiting on runs that have not started. 84 checks reloaded every 30 s is 2.8 status computations per second. That matches the observed rate. **Root cause** `registerGitHubReviewCheckWork` loads with `githubReviewCheckReconstructionIntervalMs`, which allows one company-wide rebuild every 30 s. Run lifecycle changes elsewhere in the company keep sending `{ companyId }` hints, so the rebuild runs every 30 s. `entry()` calls `githubReviewCheckState` for every `received` row on every load, only to see whether the status differs from the published one. A waiting check's status does not change until its review, its run, its delivery or its connection changes. **Expected behavior** A company-wide rebuild reads the waiting checks without recomputing each status. A change to a check's run or pull request still publishes promptly. **Steps to reproduce** 1. Create 30 `received` GitHub review checks whose published status is `queued`, with a poll deadline 1 hour ahead. 2. Register the review-check sweep. Send a company-wide hint every 30 s. 3. On master, each rebuild runs one page read plus 2–5 statements per check. **Paperclip version or commit** master at 5acf27bc3b. ## What Changed - `server/src/services/chat-channels/github/review-check-work.ts`: - A company-wide load (company scope with no endpoint, repository, number or task) does not call `githubReviewCheckState`. It keeps each check's saved deadline: the head-poll `retryAt`, or the lease expiry. Dispatch reads the current status anyway when that deadline comes. - Startup (no scope) and targeted loads (endpoint, repository, pull request or task) still read every status, as before. So do `check()` and dispatch. - Run lifecycle changes send `{ companyId, issueId }` for the run's task, and publication lease releases send the pull request scope. Both still publish a status change at once. - New `mergeEntry`: an entry from a rebuild that did not read the status keeps an earlier due time that a status read found. A company-wide rebuild never postpones a pending publication. `check()` reads the status again before it dispatches. - `server/src/services/chat-channels/github/review-check-work.test.ts` (new, real PostgreSQL): - `does not recompute the status of every waiting check on each company rebuild` - `still publishes a check promptly when its review run starts` (a task-scoped hint after a review run starts makes only that check due) - `server/src/__tests__/chat-channels.integration.test.ts`: - `publishes a local change at once and still recovers a missed head after backing off` used a company-wide hint to stand for the review run starting. It now sends the task-scoped hint that a run milestone sends, and still expects the `in_progress` publication at once. - `rebuilds GitHub review checks for company-wide hints at most once per interval` is now `reads no GitHub review check status for company-wide hints and still reads it for the check's task`. Company-wide hints read 0 statuses. Hints for the check's task still read 1, and hints for another task or company read 0. No GitHub check content changes. The published status, title and conclusion are computed as before. ## Verification - `pnpm vitest run src/services/chat-channels/github/review-check-work.test.ts` (server), 30 waiting checks, 4 company rebuilds: - master: 244 statements, with 120 check status computations (red). - this branch: 4 statements (one page read each), with 0 status computations (green). A task-scoped hint still makes the started check due at once. - `pnpm vitest run src/services/chat-channels/github/`: 8 files, 62 passed. - Related suites (`chat-channels.integration.test.ts`, `app-private-hostname-gate.test.ts`, `idle-completed-history.test.ts`, `chat-github-review-repair.test.ts`) filtered to review-check, GitHub, history and hostname tests: 702 passed before the integration-test updates below; after updating the two review-check tests, the 15 review-check tests pass. - `pnpm typecheck` (server): passed. ## Risks - A status change that sends only a company-wide hint is now published at the check's next poll deadline, not at the next company rebuild. The poll backs off to at most 5 minutes. Run lifecycle changes for a review run carry its task, and connection or configuration changes are also checked by dispatch, so this affects only hints that name no task or pull request. - Startup still reads every status, so recovery after a restart is unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL tests) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../chat-channels.integration.test.ts | 11 +- .../github/review-check-work.test.ts | 132 ++++++++++++++++++ .../chat-channels/github/review-check-work.ts | 23 ++- 3 files changed, 157 insertions(+), 9 deletions(-) create mode 100644 server/src/services/chat-channels/github/review-check-work.test.ts diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index 5c6e071c913..268f13836f5 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -7850,7 +7850,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { await f.pass("*"); expect(f.pulls()).toBe(pulls); await db.update(chatGitHubReviews).set({ runId: f.run.id }).where(eq(chatGitHubReviews.id, f.review.id)); - await f.pass(JSON.stringify({ companyId: f.companyId })); + // A run change notifies its task, as run milestones do. + await f.pass(JSON.stringify({ companyId: f.companyId, issueId: f.review.issueId })); expect(f.writes.at(-1)).toMatchObject({ status: "in_progress" }); expect(await f.action()).toMatchObject({ status: "received", result: { state: "running", unchangedPolls: 0 } }); await f.window(400); @@ -7860,7 +7861,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { expect((await db.select().from(chatGitHubReviews).where(eq(chatGitHubReviews.id, f.review.id)))[0].state).toBe("superseded"); }, 120_000); - it("rebuilds GitHub review checks for company-wide hints at most once per interval", async () => { + it("reads no GitHub review check status for company-wide hints and still reads it for the check's task", async () => { const f = await unresolvedReviewCheckPasses(); await f.window(400); const reads = async (hint: string) => { @@ -7868,12 +7869,14 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { await f.pass(hint); return f.states() - before; }; + // A company-wide rebuild keeps the saved poll deadline; dispatch reads + // the status when it comes. const company = JSON.stringify({ companyId: f.companyId }); const repeated = []; for (let index = 0; index < 10; index++) repeated.push(await reads(company)); - expect(repeated).toEqual([1, 0, 0, 0, 0, 0, 0, 0, 0, 0]); + expect(repeated).toEqual([0, 0, 0, 0, 0, 0, 0, 0, 0, 0]); for (let index = 0; index < GITHUB_REVIEW_CHECK_COMPANY_RECONSTRUCTION_MS / 1000 - 10; index++) await f.pass(); - expect(await reads(company)).toBe(1); + expect(await reads(company)).toBe(0); expect(await reads(JSON.stringify({ companyId: f.companyId, issueId: f.review.issueId }))).toBe(1); expect(await reads(JSON.stringify({ companyId: f.companyId, issueId: f.review.issueId }))).toBe(1); expect(await reads(JSON.stringify({ companyId: f.companyId, issueId: randomUUID() }))).toBe(0); diff --git a/server/src/services/chat-channels/github/review-check-work.test.ts b/server/src/services/chat-channels/github/review-check-work.test.ts new file mode 100644 index 00000000000..2bfb016f9e7 --- /dev/null +++ b/server/src/services/chat-channels/github/review-check-work.test.ts @@ -0,0 +1,132 @@ +import { randomUUID } from "node:crypto"; +import { setTimeout as delay } from "node:timers/promises"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { chatActions, chatGitHubReviews, createDb, heartbeatRuns, type Db } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js"; +import { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../delivery-work-notifications.js"; +import { idleWorkSnapshot } from "../../task-admission.js"; +import { registerGitHubReviewCheckWork } from "./review-check-work.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +// Staging (2026-11-15) polled 84 GitHub review checks whose runs had not +// started. Their check status reads ran at about 3/s at idle. +const CHECKS = 30; + +describe.skipIf(!support.supported)("GitHub review check sweep", () => { + let temporary: Awaited>; + let db: Db; + let coordinator: ReturnType | undefined; + const errors: unknown[] = []; + const dispatched: string[] = []; + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("github-review-check-work-"); + db = createDb(temporary.connectionString); + }, 60_000); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; dispatched.length = 0; + vi.useRealTimers(); + expect(errors.splice(0)).toEqual([]); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + + async function seed() { + const client = db.$client; + const [{ company }] = await client.unsafe(`insert into companies (name, issue_prefix) values ('Checks', $1) returning id as company`, + [`C${randomUUID().slice(0, 7)}`]); + const [{ agent }] = await client.unsafe(`insert into agents (company_id, name) values ($1, 'Reviewer') returning id as agent`, [company]); + const [{ application }] = await client.unsafe(`insert into tool_applications (company_id, application_key, name, type) + values ($1, 'github', 'GitHub', 'chat') returning id as application`, [company]); + const [{ connection }] = await client.unsafe(`insert into tool_connections (company_id, application_id, name, uid, transport, connection_purpose) + values ($1, $2, 'GitHub', $3, 'chat_sdk', 'channel') returning id as connection`, [company, application, `github-${company}`]); + const [{ endpoint }] = await client.unsafe(`insert into chat_endpoints (company_id, connection_id, provider, public_id, assigned_agent_id, status) + values ($1, $2, 'github', $3, $4, 'active') returning id as endpoint`, [company, connection, `checks-${company}`, agent]); + await client.unsafe(`insert into chat_github_configurations (endpoint_id, company_id, configuration, updated_by_user_id) + values ($1, $2, $3::jsonb, 'owner')`, [endpoint, company, JSON.stringify({ toolsEnabled: true, defaults: {}, repositories: {} })]); + await client.unsafe(`insert into chat_endpoint_resources (company_id, endpoint_id, type, provider_resource_id, label, availability, enabled, metadata) + values ($1, $2, 'repository', 'paperclipai/paperclip', 'paperclipai/paperclip', 'available', true, $3::jsonb)`, + [company, endpoint, JSON.stringify({ providerRepositoryId: "418" })]); + const checks: Array<{ id: string; number: number; headSha: string }> = []; + const retryAt = new Date(Date.now() + 3_600_000).toISOString(); + for (let number = 1; number <= CHECKS; number++) { + const headSha = randomUUID().replaceAll("-", "").padEnd(40, "0"); + const [action] = await db.insert(chatActions).values({ companyId: company, endpointId: endpoint, kind: "github_review_check", + providerActionId: `github_review_check:${number}`, status: "received", + payload: { admitted: true, runtimeGeneration: 0, event: { repository: "paperclipai/paperclip", repositoryId: "418", + pullNumber: number, headSha, deliveryId: randomUUID() } }, + result: { state: "queued", retryAt } }).returning(); + checks.push({ id: action!.id, number, headSha }); + } + return { companyId: company as string, agentId: agent as string, endpointId: endpoint as string, checks }; + } + + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, + onError: error => errors.push(error) }); + await registerGitHubReviewCheckWork(db, coordinator, async id => { dispatched.push(id); }, () => true).ready; + } + async function advance(ms: number) { + const until = Date.now() + ms; + while (Date.now() < until) { + await vi.advanceTimersByTimeAsync(Math.min(1_000, until - Date.now())); await delay(1); + while (idleWorkSnapshot().active > 0) await delay(5); + } + } + async function hint(scope: Record) { + await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubReviewChecks, JSON.stringify(scope)); + }); + } + async function observe(work: () => Promise) { + const statements: string[] = [], prior = db.$client.options.debug; + db.$client.options.debug = (_connection, query) => { statements.push(query); }; + try { await work(); } finally { db.$client.options.debug = prior; } + return { + statements: statements.filter(query => /"chat_actions"|chat_github_reviews|heartbeat_runs|chat_deliveries/.test(query)).length, + // The first read of the check status of one check. + states: statements.filter(query => /from "chat_endpoints" inner join "companies"/.test(query)).length, + }; + } + + it("does not recompute the status of every waiting check on each company rebuild", async () => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const f = await seed(); + await start(); await advance(5_000); + const rebuilds = await observe(async () => { + for (let pass = 0; pass < 4; pass++) { await hint({ companyId: f.companyId }); await advance(31_000); } + }); + console.info(`review check sweep: 4 company rebuilds cost ${rebuilds.statements} statements ` + + `(${rebuilds.states} check status reads) for ${CHECKS} waiting checks`); + expect(rebuilds.states).toBe(0); + expect(rebuilds.statements).toBeLessThanOrEqual(4); + expect(dispatched).toEqual([]); + }, 300_000); + + it("still publishes a check promptly when its review run starts", async () => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const f = await seed(); + await start(); await advance(5_000); + const target = f.checks[4]!; + const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running", + contextSnapshot: {} }).returning(); + const [{ issue }] = await db.$client.unsafe(`insert into issues (company_id, title, status) values ($1, 'Review', 'in_progress') + returning id as issue`, [f.companyId]); + await db.insert(chatGitHubReviews).values({ companyId: f.companyId, endpointId: f.endpointId, issueId: issue, + runId: run!.id, repositoryId: "418", repository: "paperclipai/paperclip", pullNumber: target.number, + headSha: target.headSha, state: "running", deliveryId: randomUUID(), configurationRevision: 1, + policySnapshot: {}, event: {} } as typeof chatGitHubReviews.$inferInsert); + // A run lifecycle change sends a hint for its task. + await hint({ companyId: f.companyId, issueId: issue }); + await advance(5_000); + // The stand-in dispatch saves nothing, so the check stays due; only it is due. + expect(new Set(dispatched)).toEqual(new Set([target.id])); + // A later company-wide rebuild, which does not read the status, keeps the + // changed check due instead of moving it to its saved poll deadline. + dispatched.length = 0; + await hint({ companyId: f.companyId }); + await advance(5_000); + expect(new Set(dispatched)).toEqual(new Set([target.id])); + }, 300_000); +}); diff --git a/server/src/services/chat-channels/github/review-check-work.ts b/server/src/services/chat-channels/github/review-check-work.ts index 49ebb3ab8f5..5fe7f567087 100644 --- a/server/src/services/chat-channels/github/review-check-work.ts +++ b/server/src/services/chat-channels/github/review-check-work.ts @@ -7,7 +7,7 @@ import { registerScopedDeadlineWork } from "../../scoped-deadline-work.js"; import { githubReviewCheckState } from "./review-check-state.js"; type Scope = { companyId: string; endpointId?: string; repositoryId?: string; number?: number; issueId?: string }; -type Entry = Scope & { id: string; dueAt: number }; +type Entry = Scope & { id: string; dueAt: number; stateRead: boolean }; export const GITHUB_REVIEW_CHECK_COMPANY_RECONSTRUCTION_MS = 30_000; @@ -48,18 +48,21 @@ export function registerGitHubReviewCheckWork( cursor ? gt(chatActions.id, cursor) : undefined, actionId ? eq(chatActions.id, actionId) : undefined, )).orderBy(asc(chatActions.id)).limit(100); } - async function entry(row: Awaited>[number]): Promise { + /** `readState` compares the current check status with the published one, so + * a change is published before the saved poll deadline. */ + async function entry(row: Awaited>[number], readState = true): Promise { const { action } = row; const event = action.payload.event as GitHubReviewEventContext; let dueAt = action.result?.retryAt ? Date.parse(String(action.result.retryAt)) : 0; if (!Number.isFinite(dueAt)) throw new Error("GitHub review check has an invalid retry deadline"); - if (action.status === "received") { + if (action.status === "received" && readState) { const current = await githubReviewCheckState(db, action); if (current.disposition !== "publish" || current.state !== action.result?.state) dueAt = 0; } if (row.leaseExpiresAt) dueAt = Math.max(dueAt, Date.parse(row.leaseExpiresAt)); return { id: action.id, companyId: action.companyId, endpointId: action.endpointId, - repositoryId: event.repositoryId, number: event.pullNumber, issueId: row.reviewIssueId ?? undefined, dueAt }; + repositoryId: event.repositoryId, number: event.pullNumber, issueId: row.reviewIssueId ?? undefined, dueAt, + stateRead: readState || action.status !== "received" }; } return registerScopedDeadlineWork(coordinator, { queue: DELIVERY_QUEUES.chatGitHubReviewChecks, @@ -69,14 +72,24 @@ export function registerGitHubReviewCheckWork( (!scope.repositoryId || value.repositoryId === scope.repositoryId) && (!scope.number || value.number === scope.number) && (!scope.issueId || value.issueId === scope.issueId)), reconstructionIntervalMs: githubReviewCheckReconstructionIntervalMs, + // A rebuild that did not read the status must not postpone a change that + // an earlier read found due. check() reads the status again. + mergeEntry: (pending: Entry, loaded: Entry) => loaded.stateRead ? loaded + : { ...loaded, dueAt: Math.min(pending.dueAt, loaded.dueAt) }, load: async (scope?: Scope, signal?: AbortSignal) => { const entries: Entry[] = []; let cursor: string | undefined; + // A company-wide hint says only that something in the company changed. + // Its rebuild keeps each check's saved poll deadline, where dispatch reads + // the current status anyway. A change to a check's run or pull request + // sends a hint for its task or pull request, and startup (no scope) reads + // every status, so both still publish a change at once. + const readState = !scope || !!(scope.endpointId || scope.repositoryId || scope.number || scope.issueId); while (canRun() && !signal?.aborted) { const rows = await page(scope, cursor); for (const row of rows) { if (!canRun() || signal?.aborted) throw signal?.reason ?? new Error("GitHub review check recovery stopped"); - entries.push(await entry(row)); + entries.push(await entry(row, readState)); } if (rows.length < 100) return entries; cursor = rows[rows.length - 1].action.id;