From a7453eba604bc8439e79e0ec852a59ffbd119d32 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 05:22:35 -0600 Subject: [PATCH] perf(chat): back off GitHub review check head polls while the check cannot change (#21608) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel publishes a "Paperclip Review" check for each admitted pull request head. A background queue (`chat-github-review-checks`) keeps each unresolved check in step with its task and run > - While a check is queued or running, the queue polls GitHub for the pull request head every 5 seconds. This finds a push whose webhook was lost > - The poll never stops while the check is unresolved. A check whose run never comes, or whose run the check cannot find, stays queued forever. It re-reads its whole state and calls GitHub every 5 seconds, also when nothing changes > - On a hosted test database with zero active runs, the five state statements of this check ran 10.3 times a second each > - This pull request doubles the head poll interval for each poll that finds the same state, up to 5 minutes. A state change resets it to 5 seconds > - The benefit is that an idle unresolved check costs about one poll each 5 minutes, not one each 5 seconds. Local changes still publish at once ## Linked Issues or Issue Description Refs #17662 (added the 5-second head reconciliation for unresolved checks). Refs #21580 (the same kind of fix for the GitHub response sweeps). **What happened?** On a hosted test database (Neon, 488-second window, zero running or queued runs, 2 runs created in 30 minutes), these five statements ran at exactly 10.3 a second each: the `chat_endpoints` source read, the `chat_github_reviews` read, the `chat_message_links` comment read, the `heartbeat_runs` `wakeCommentId` lookup and the `chat_deliveries` state read. All five come from `githubReviewCheckState` (`server/src/services/chat-channels/github/review-check-state.ts`). **Expected behavior** A check whose state cannot change makes no recurring database reads or GitHub calls until something that can change it happens: a local hint, a lease expiry, or a saved deadline. The missed-webhook head check stays, at a bounded cost. **Steps to reproduce** 1. Admit a GitHub review check. Bind a task review to it, but give it no run that the check can find (for example, the run was never created, or its wake comment is not the delivery's comment). 2. Start the review check queue and leave the instance idle. 3. Count statements. The check is due again 5 seconds after each poll, for ever. Each poll evaluates the check state about 3 to 4 times (before dispatch, in dispatch, after dispatch, and in the rebuild that the publication lease release starts). Each poll also asks GitHub for an installation token and the pull request. **Cause** `dispatch` (`server/src/services/chat-github-checks.ts`) saved `retryAt = now + 5s` after each poll. `entry()` in `review-check-work.ts` uses that deadline when the state is unchanged. So an unresolved check was due every 5 seconds without end. ## What Changed - `chat-github-checks.ts`: save `unchangedPolls` in the check result. A poll that finds the same state as the saved state adds one (up to 6). A changed state, a first publication and a recovered failure set it to 0. The next head poll is due after `min(5 minutes, 5 seconds × 2^unchangedPolls)`. - Export `GITHUB_REVIEW_CHECK_HEAD_POLL_MS` (5 seconds) and `GITHUB_REVIEW_CHECK_MAX_HEAD_POLL_MS` (5 minutes). - `chat-channels.integration.test.ts`: two regression tests. They use the real review check queue, the real `dispatch`, embedded PostgreSQL, a controlled clock and one pass each second. The fixture has the hosted state: a review with no run, a delivery with a linked comment, and no run for that comment. ## Verification Reproduction with the real delivery coordinator, the real queue and embedded PostgreSQL. One queued check with no findable run, 30 seconds of real time, before this change: | Company-wide hints | Statements a second | Check state reads a second | GitHub pull reads a second | | --- | --- | --- | --- | | None | 8.4 | 0.70 | 0.17 | | One each second (`notifyChatRunMilestones`) | 15.2 | 1.67 | 0.17 | So one stuck check costs about 0.7 state reads a second from the head poll alone. The hosted rate (10.3 a second) is consistent with about 15 such checks, or fewer checks with frequent company-wide hints. Company-wide hints are a separate cause and a separate pull request. Red/green regression test, 600 simulated seconds with one pass each second: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t "unresolved GitHub review check cannot change|publishes a local change at once"` - Red (without the fix): `expected 120 to be less than or equal to 7`. Over 600 seconds: 5,160 statements, 360 check state reads, 120 GitHub pull reads. - Green: 258 statements, 18 check state reads, 6 GitHub pull reads (8.6 → 0.43 statements a second, a 95% drop). The saved interval stops at 5 minutes. - Correctness (green): after the back-off, a restarted queue makes no early GitHub call. A run bound to the review publishes `in_progress` at the next pass and resets the interval. A pushed head is still found within 5 minutes, the check is cancelled and the review is superseded. Other checks: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed, including all 12 review check queue tests. - `cd server && npx vitest run src/services/chat-github-review-repair.test.ts src/__tests__/idle-completed-history.test.ts src/__tests__/chat-admission-condition.test.ts src/__tests__/app-private-hostname-gate.test.ts`: 109 passed. - `cd server && node ../scripts/run-tsc.mjs` and `node scripts/typecheck-layers.mjs` (with test layers): passed. - `pnpm verify --explain` retains 2,718 test files conservatively, so I did not run it locally. Required CI supplies that coverage. ## Risks - A push whose webhook is lost is found later: after up to 5 minutes, not 5 seconds, when the check has not changed for a while. The new head's own webhook still creates its new check at once. Only the old head's check stays queued or running on GitHub a little longer before it is cancelled. - Local changes are not delayed. `entry()` still makes a check due at once when its state differs from the saved state, and run, review, enqueue and lease hints still rebuild it. - Saved actions from an older release have no `unchangedPolls`. They start at 0. An invalid saved value also starts at 0. No migration. - Restart recovery is unchanged: the deadline is saved on the action, and startup rebuilds all unfinished checks. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude Code with tool use (shell, file edits, and local embedded PostgreSQL reproductions). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../chat-channels.integration.test.ts | 77 ++++++++++++++++++- server/src/services/chat-github-checks.ts | 11 ++- 2 files changed, 86 insertions(+), 2 deletions(-) diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index 707beb57534..fa0fea2c7f6 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -81,7 +81,7 @@ import { GITHUB_INLINE_RETRY_WINDOW_MS } from "../services/chat-channels/github/ import { githubChatReviewService } from "../services/chat-github-reviews.js"; import { resolveGitHubReviewInstructions } from "../services/chat-channels/github/review-instructions.js"; import { githubResponseCommentService } from "../services/chat-github-response-comments.js"; -import { githubReviewCheckService } from "../services/chat-github-checks.js"; +import { GITHUB_REVIEW_CHECK_MAX_HEAD_POLL_MS, githubReviewCheckService } from "../services/chat-github-checks.js"; import { githubAutomaticReviewEvent } from "../services/chat-github-events.js"; import { githubBotToolsForSession, syncGitHubBotTools } from "../services/chat-github-tools.js"; import { buildPaperclipRuntimeMcpServers } from "../services/heartbeat/run-preparation.js"; @@ -7740,6 +7740,81 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { } finally { release(); await coordinator.stop(); } }); + async function unresolvedReviewCheckPasses() { + const f = await reviewCheckWorkerFixture("running"); + await db.update(chatGitHubReviews).set({ runId: null }).where(eq(chatGitHubReviews.id, f.review.id)); + const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.endpointId, f.endpoint.id)); + await db.update(chatDeliveries).set({ normalizedEvent: { ...delivery.normalizedEvent, + githubAutomatic: { context: { deliveryId: f.event.deliveryId } } } }).where(eq(chatDeliveries.id, delivery.id)); + await db.update(heartbeatRuns).set({ contextSnapshot: { wakeCommentId: randomUUID() } }).where(eq(heartbeatRuns.id, f.run.id)); + await f.enqueue(); + let clock = Date.now(); + const now = vi.spyOn(Date, "now").mockImplementation(() => clock); + onTestFinished(() => now.mockRestore()); + const queries: string[] = []; + const unsafe = db.$client.unsafe; + db.$client.unsafe = ((query: string, ...rest: unknown[]) => { + queries.push(query); + return (unsafe as (...args: unknown[]) => unknown).call(db.$client, query, ...rest); + }) as typeof unsafe; + onTestFinished(() => { db.$client.unsafe = unsafe; }); + const register = () => { + const tasks: Array<{ run(signal: AbortSignal): Promise; onSettled?(hints?: ReadonlySet): void }> = []; + f.checks.registerCheckWork({ register: (_queue: unknown, task: typeof tasks[number]) => { + tasks.push(task); return { wake() {}, stop: async () => {} }; + } } as never, () => true); + return tasks[0]; + }; + let task = register(); + const pulls = () => f.requests.filter(url => url.endsWith("/pulls/418")).length; + const pass = async (hint?: string) => { + clock += 1000; + if (hint) task.onSettled?.(new Set([hint])); + await task.run(new AbortController().signal); + }; + const window = async (seconds: number) => { + const before = { statements: queries.length, pulls: pulls(), + states: queries.filter(query => query.includes('from "chat_github_reviews"')).length }; + for (let second = 0; second < seconds; second++) await pass(); + return { statements: queries.length - before.statements, pulls: pulls() - before.pulls, + states: queries.filter(query => query.includes('from "chat_github_reviews"')).length - before.states }; + }; + return { ...f, pass, window, pulls, restart: () => { task = register(); }, now: () => clock }; + } + + it("backs off remote head polls while an unresolved GitHub review check cannot change", async () => { + const f = await unresolvedReviewCheckPasses(); + await f.pass(); + expect(f.writes).toEqual([expect.objectContaining({ status: "queued" })]); + const idle = await f.window(600); + expect(idle.pulls).toBeLessThanOrEqual(7); + expect(idle.states).toBeLessThanOrEqual(21); + expect(f.writes).toHaveLength(1); + const saved = await f.action(); + expect(saved).toMatchObject({ status: "received", result: { state: "queued", unchangedPolls: 6 } }); + expect(Date.parse(String(saved.result?.retryAt)) - f.now()).toBeLessThanOrEqual(GITHUB_REVIEW_CHECK_MAX_HEAD_POLL_MS); + }, 120_000); + + it("publishes a local change at once and still recovers a missed head after backing off", async () => { + const f = await unresolvedReviewCheckPasses(); + await f.window(400); + const backedOff = await f.action(); + expect(backedOff.result?.unchangedPolls).toBeGreaterThanOrEqual(5); + const pulls = f.pulls(); + f.restart(); + await f.pass("*"); + expect(f.pulls()).toBe(pulls); + await db.update(chatGitHubReviews).set({ runId: f.run.id }).where(eq(chatGitHubReviews.id, f.review.id)); + await f.pass(JSON.stringify({ companyId: f.companyId })); + expect(f.writes.at(-1)).toMatchObject({ status: "in_progress" }); + expect(await f.action()).toMatchObject({ status: "received", result: { state: "running", unchangedPolls: 0 } }); + await f.window(400); + f.setRemoteHead("c".repeat(40)); + await f.window(GITHUB_REVIEW_CHECK_MAX_HEAD_POLL_MS / 1000 + 1); + expect((await f.action()).status).toBe("cancelled"); + expect((await db.select().from(chatGitHubReviews).where(eq(chatGitHubReviews.id, f.review.id)))[0].state).toBe("superseded"); + }, 120_000); + async function reviewProjectionFixture(cancelFirst = true) { const f = await reviewBotFixture(); const deliveryId = randomUUID(); diff --git a/server/src/services/chat-github-checks.ts b/server/src/services/chat-github-checks.ts index 63e1f9321f8..be799375918 100644 --- a/server/src/services/chat-github-checks.ts +++ b/server/src/services/chat-github-checks.ts @@ -21,6 +21,10 @@ import { githubBotRequest, } from "./chat-github-client.js"; +export const GITHUB_REVIEW_CHECK_HEAD_POLL_MS = 5_000; +export const GITHUB_REVIEW_CHECK_MAX_HEAD_POLL_MS = 300_000; +const UNCHANGED_HEAD_POLL_LIMIT = Math.ceil(Math.log2(GITHUB_REVIEW_CHECK_MAX_HEAD_POLL_MS / GITHUB_REVIEW_CHECK_HEAD_POLL_MS)); + /** Check status follows the ordinary chat delivery/task/run. This outbox never * schedules an agent. Assessments are published by the governed review tool. */ export function githubReviewCheckService(db: Db, fetchImpl = fetch) { @@ -248,6 +252,10 @@ export function githubReviewCheckService(db: Db, fetchImpl = fetch) { ); }); } + const savedPolls = Number(fresh.result?.unchangedPolls); + const unchangedPolls = fresh.status === "received" && fresh.result?.state === state + ? Math.min(UNCHANGED_HEAD_POLL_LIMIT, (Number.isSafeInteger(savedPolls) && savedPolls > 0 ? savedPolls : 0) + 1) + : 0; await lease.commit(async (tx) => { await tx .update(chatActions) @@ -257,7 +265,8 @@ export function githubReviewCheckService(db: Db, fetchImpl = fetch) { state, attempts: 0, retryable: true, - retryAt: new Date(Date.now() + 5000).toISOString(), + unchangedPolls, + retryAt: new Date(Date.now() + Math.min(GITHUB_REVIEW_CHECK_MAX_HEAD_POLL_MS, GITHUB_REVIEW_CHECK_HEAD_POLL_MS * 2 ** unchangedPolls)).toISOString(), }, updatedAt: new Date(), })