Files
PaperClipAI/server
DottaandClaude Opus 5.5 a7453eba60 perf(chat): back off GitHub review check head polls while the check cannot change (#21608)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel publishes a "Paperclip Review" check for
each admitted pull request head. A background queue
(`chat-github-review-checks`) keeps each unresolved check in step with
its task and run
> - While a check is queued or running, the queue polls GitHub for the
pull request head every 5 seconds. This finds a push whose webhook was
lost
> - The poll never stops while the check is unresolved. A check whose
run never comes, or whose run the check cannot find, stays queued
forever. It re-reads its whole state and calls GitHub every 5 seconds,
also when nothing changes
> - On a hosted test database with zero active runs, the five state
statements of this check ran 10.3 times a second each
> - This pull request doubles the head poll interval for each poll that
finds the same state, up to 5 minutes. A state change resets it to 5
seconds
> - The benefit is that an idle unresolved check costs about one poll
each 5 minutes, not one each 5 seconds. Local changes still publish at
once

## Linked Issues or Issue Description

Refs #17662 (added the 5-second head reconciliation for unresolved
checks). Refs #21580 (the same kind of fix for the GitHub response
sweeps).

**What happened?**

On a hosted test database (Neon, 488-second window, zero running or
queued runs, 2 runs created in 30 minutes), these five statements ran at
exactly 10.3 a second each: the `chat_endpoints` source read, the
`chat_github_reviews` read, the `chat_message_links` comment read, the
`heartbeat_runs` `wakeCommentId` lookup and the `chat_deliveries` state
read. All five come from `githubReviewCheckState`
(`server/src/services/chat-channels/github/review-check-state.ts`).

**Expected behavior**

A check whose state cannot change makes no recurring database reads or
GitHub calls until something that can change it happens: a local hint, a
lease expiry, or a saved deadline. The missed-webhook head check stays,
at a bounded cost.

**Steps to reproduce**

1. Admit a GitHub review check. Bind a task review to it, but give it no
run that the check can find (for example, the run was never created, or
its wake comment is not the delivery's comment).
2. Start the review check queue and leave the instance idle.
3. Count statements. The check is due again 5 seconds after each poll,
for ever. Each poll evaluates the check state about 3 to 4 times (before
dispatch, in dispatch, after dispatch, and in the rebuild that the
publication lease release starts). Each poll also asks GitHub for an
installation token and the pull request.

**Cause**

`dispatch` (`server/src/services/chat-github-checks.ts`) saved `retryAt
= now + 5s` after each poll. `entry()` in `review-check-work.ts` uses
that deadline when the state is unchanged. So an unresolved check was
due every 5 seconds without end.

## What Changed

- `chat-github-checks.ts`: save `unchangedPolls` in the check result. A
poll that finds the same state as the saved state adds one (up to 6). A
changed state, a first publication and a recovered failure set it to 0.
The next head poll is due after `min(5 minutes, 5 seconds ×
2^unchangedPolls)`.
- Export `GITHUB_REVIEW_CHECK_HEAD_POLL_MS` (5 seconds) and
`GITHUB_REVIEW_CHECK_MAX_HEAD_POLL_MS` (5 minutes).
- `chat-channels.integration.test.ts`: two regression tests. They use
the real review check queue, the real `dispatch`, embedded PostgreSQL, a
controlled clock and one pass each second. The fixture has the hosted
state: a review with no run, a delivery with a linked comment, and no
run for that comment.

## Verification

Reproduction with the real delivery coordinator, the real queue and
embedded PostgreSQL. One queued check with no findable run, 30 seconds
of real time, before this change:

| Company-wide hints | Statements a second | Check state reads a second
| GitHub pull reads a second |
| --- | --- | --- | --- |
| None | 8.4 | 0.70 | 0.17 |
| One each second (`notifyChatRunMilestones`) | 15.2 | 1.67 | 0.17 |

So one stuck check costs about 0.7 state reads a second from the head
poll alone. The hosted rate (10.3 a second) is consistent with about 15
such checks, or fewer checks with frequent company-wide hints.
Company-wide hints are a separate cause and a separate pull request.

Red/green regression test, 600 simulated seconds with one pass each
second:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "unresolved GitHub
review check cannot change|publishes a local change at once"`
- Red (without the fix): `expected 120 to be less than or equal to 7`.
Over 600 seconds: 5,160 statements, 360 check state reads, 120 GitHub
pull reads.
- Green: 258 statements, 18 check state reads, 6 GitHub pull reads (8.6
→ 0.43 statements a second, a 95% drop). The saved interval stops at 5
minutes.
- Correctness (green): after the back-off, a restarted queue makes no
early GitHub call. A run bound to the review publishes `in_progress` at
the next pass and resets the interval. A pushed head is still found
within 5 minutes, the check is cancelled and the review is superseded.

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed,
including all 12 review check queue tests.
- `cd server && npx vitest run
src/services/chat-github-review-repair.test.ts
src/__tests__/idle-completed-history.test.ts
src/__tests__/chat-admission-condition.test.ts
src/__tests__/app-private-hostname-gate.test.ts`: 109 passed.
- `cd server && node ../scripts/run-tsc.mjs` and `node
scripts/typecheck-layers.mjs` (with test layers): passed.
- `pnpm verify --explain` retains 2,718 test files conservatively, so I
did not run it locally. Required CI supplies that coverage.

## Risks

- A push whose webhook is lost is found later: after up to 5 minutes,
not 5 seconds, when the check has not changed for a while. The new
head's own webhook still creates its new check at once. Only the old
head's check stays queued or running on GitHub a little longer before it
is cancelled.
- Local changes are not delayed. `entry()` still makes a check due at
once when its state differs from the saved state, and run, review,
enqueue and lease hints still rebuild it.
- Saved actions from an older release have no `unchangedPolls`. They
start at 0. An invalid saved value also starts at 0. No migration.
- Restart recovery is unchanged: the deadline is saved on the action,
and startup rebuilds all unfinished checks.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local embedded PostgreSQL
reproductions).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:22:35 -06:00
..