perf(chat): rebuild GitHub response sweeps for company-wide hints at most every 30 seconds (#21580)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel has two background sweeps for request
response comments: acknowledgements, and terminal responses. Each sweep
rebuilds its candidate set when a hint arrives, and it reads the latest
run of each candidate request (`readGitHubResponseRun`) on each rebuild
> - A run milestone for a run with no task sends a company-wide hint to
both sweeps. Examples: each status change of a run without a task
(`setRunStatus`), and a comment merged into an active run in the wake
queue. In a busy company these hints arrive more than once a second
> - So both sweeps rebuild the whole company about once a second (their
minimum pass interval). Each rebuild reads every parked request again,
although nothing about those requests changed. On a hosted test
database, `readGitHubResponseRun` ran 6.7 times a second
> - This pull request rebuilds a GitHub response sweep for a
company-wide hint at most once every 30 seconds. Issue, request,
conversation, endpoint and lease hints still rebuild immediately.
`external-objects.ts` uses the same rule for its refresh sweep
> - The benefit is that a steady stream of company-wide hints no longer
makes each sweep read every parked request every second

## Linked Issues or Issue Description

Refs #21577 (makes each `readGitHubResponseRun` call cheap; this pull
request reduces how often the sweeps call it). Refs #18384 (the same
reconstruction interval for company-wide external-object hints).

**What existing behavior does this improve?**
The candidate rebuild of `registerGitHubAcknowledgementWork`
(`acknowledgement-work.ts`) and `registerGitHubTerminalResponseWork`
(`terminal-response-work.ts`) for company-wide hints.

**Subsystem affected**
GitHub chat channel: request acknowledgement and terminal-response
recovery sweeps.

**Current behavior**
`notifyChatRunMilestones` calls `notifyChatResponseAvailability` with
its scope. When the scope has no issue (only a company, or a company and
a run), the hint for both GitHub response queues is company-wide
(`{"companyId": ...}`), because the response scope has no run field.
Each company-wide hint makes each sweep load every candidate of the
company and call `readGitHubResponseRun` for each one. Passes are
limited to one a second, so with a steady hint stream the cost is about
(parked candidates) reads a second.

**Proposed behavior**
Both sweeps set `reconstructionIntervalMs`. A company-wide scope
(company ID only) waits 30 seconds after its last rebuild before it
rebuilds again. All narrower scopes, the startup scope (`*`), and saved
deadlines are not changed. Entries that are already loaded stay
available for dispatch while a company-wide hint waits, and each check
still reads the request again from the database.

**Reason and benefit**
On the hosted test database, `readGitHubResponseRun` ran 6.7 times a
second for no change in state. In a local reproduction, the call rate
drops from about one call for each parked request each second to one
call for each parked request each 30 seconds.

**Breaking changes**
None.

## What Changed

- Add `response-sweep-scope.ts` with the shared scope type of the two
GitHub response sweeps and `githubResponseReconstructionIntervalMs`: 30
seconds for a company-wide scope, 0 for all other scopes.
- Use it as `reconstructionIntervalMs` in the acknowledgement sweep and
the terminal-response sweep. Both files now import the shared scope type
instead of declaring it.
- Add `response-sweep-scope.test.ts`. It registers both real sweeps on a
real database with 6 parked requests (3 for each sweep) and a controlled
clock. It sends company-wide hints and issue hints, and counts
`readGitHubResponseRun` statements on each pass.

## Verification

Cause, from a local reproduction with the real delivery coordinator and
both real sweeps on embedded PostgreSQL 18.1, with 8 parked requests (4
for each sweep), each with a succeeded native run that waits for its
presentation:

| Hints | Hints a second | `readGitHubResponseRun` calls a second |
| --- | --- | --- |
| None | 0 | 0 |
| Company-wide run milestone (`{ companyId, runId }`) | 9.7 | 8.75 |
| Company-wide run milestone | 1.0 | 7.98 |
| Run milestone for an unrelated issue | 9.7 | 0 |

So the call rate is about one call for each parked request each second,
from company-wide hints only. The hosted rate (6.7 a second) is
consistent with about 7 parked GitHub requests in one company.

Red/green regression test:

- `cd server && npx vitest run
src/services/chat-channels/github/response-sweep-scope.test.ts`
- Red (without the two `reconstructionIntervalMs` lines): `expected [ 6,
6, 6, 6, 6 ] to deeply equal [ 6, +0, +0, +0, +0 ]`. Each of 5
company-wide hints rebuilt all 6 parked requests.
- Green: 2 of 2 pass. The first company-wide hint rebuilds once (6
reads). The next 4 hints read nothing. After 30 seconds, the waiting
hint rebuilds (6 reads). Issue hints still read exactly the one request
of that issue at once, and a hint for another company reads nothing. No
acknowledgement is sent for a parked request.

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed
(all GitHub chat-channel integration tests, including the
acknowledgement and terminal-response sweeps).
- `cd server && npx tsc --noEmit`: passed.
- After the rebase onto #21577: both new test files pass (4 of 4), and
the server typecheck passes.
- `pnpm verify` selected most of the server suite, because the sweep
modules are imported widely. I stopped that broad local run. Required CI
supplies that coverage.

## Risks

- A change that only a company-wide hint reports can take up to 30
seconds to rebuild a GitHub response sweep. I checked the callers.
Changes to one request use issue, delivery, conversation or lease hints:
run status changes with a task, native presentation commits
(`status-decision-committer.ts`, `native-run-finalizer.ts`,
`native-chat-review-presentation.ts`), publications, fast responses, and
the publication lease. Those stay immediate. Company-wide hints come
from runs with no task, from a comment merged into an active run (the
wake queue sends only the run ID), and from company authority changes. A
run with no task cannot change the latest run of a GitHub request's
task. For a merged comment, the active run's next status change sends an
issue hint, and staging the request's response sends a delivery hint;
both rebuild at once. Only a mid-run ownership update can wait up to 30
seconds. For an authority change, each check reads the request and its
authority again before it acts.
- The deadline reconciliation snapshot reports these two sweeps as not
reconciled while a company-wide hint waits (up to 30 seconds). This is
the same as the external-object refresh sweep.
- The startup rebuild (`*`) and failure retries do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local PostgreSQL
reproductions).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 02:05:45 -06:00
1 parent 340157d1d6
commit a1c37b5cc4
4 files changed
+132 -6

No files matched your search

@@ -5,10 +5,8 @@ import { chatActions, chatConversations, chatEndpointLeases, chatEndpointResourc
chatPublications, fastResponseRequests, chatEndpoints, chatMessageLinks, type Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../../delivery-work-notifications.js";
import type { ChatResponseScope } from "../../chat-response-availability.js";
import { registerScopedDeadlineWork } from "../../scoped-deadline-work.js";
type Scope = ChatResponseScope & { endpointId?: string; repositoryId?: string; number?: number };
import { githubResponseReconstructionIntervalMs, type GitHubResponseSweepScope as Scope } from "./response-sweep-scope.js";
type Entry = { id: string; companyId: string; endpointId: string; deliveryId: string;
issueId: string | null; conversationId: string | null; repositoryId: string | null; number: number | null; dueAt: number };
type FastResponse = { status: string; expiresAt: string; publicationState: string | null };
@@ -83,6 +81,7 @@ export function registerGitHubAcknowledgementWork(
}
return registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.chatGitHubAcknowledgements,
reconstructionIntervalMs: githubResponseReconstructionIntervalMs,
decodeScope: (hint: string): Scope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (value: Entry, scope?: Scope) => !scope || (value.companyId === scope.companyId &&
(!scope.issueId || value.issueId === scope.issueId) && (!scope.deliveryId || value.deliveryId === scope.deliveryId) &&
@@ -0,0 +1,118 @@
import { randomUUID } from "node:crypto";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { createDb, type Db } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js";
import { registerGitHubAcknowledgementWork } from "./acknowledgement-work.js";
import { registerGitHubTerminalResponseWork } from "./terminal-response-work.js";
import { GITHUB_RESPONSE_COMPANY_RECONSTRUCTION_MS } from "./response-sweep-scope.js";
const support = await getEmbeddedPostgresTestSupport();
const suite = support.supported ? describe : describe.skip;
type Task = { run(signal: AbortSignal): Promise<unknown>; onSettled?(hints?: ReadonlySet<string>): void };
suite("GitHub response sweep scopes", () => {
let db!: Db;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
let companyId = "";
const parkedIssues: string[] = [];
let clock = Date.now();
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("paperclip-github-response-sweep-scope-");
db = createDb(temporary.connectionString);
const client = db.$client;
const [{ company }] = await client.unsafe(`insert into companies (name, issue_prefix) values ('Sweep', 'SW') returning id as company`);
companyId = company;
const [{ agent }] = await client.unsafe(`insert into agents (company_id, name) values ($1, 'GitHub agent') returning id as agent`, [company]);
const [{ application }] = await client.unsafe(`insert into tool_applications (company_id, application_key, name, type)
values ($1, 'github', 'GitHub', 'chat') returning id as application`, [company]);
const [{ connection }] = await client.unsafe(`insert into tool_connections (company_id, application_id, name, uid, transport, connection_purpose)
values ($1, $2, 'GitHub', 'github-sweep', 'chat_sdk', 'channel') returning id as connection`, [company, application]);
const [{ endpoint }] = await client.unsafe(`insert into chat_endpoints (company_id, connection_id, provider, public_id, assigned_agent_id, status)
values ($1, $2, 'github', 'sweep', $3, 'active') returning id as endpoint`, [company, connection, agent]);
for (let n = 1; n <= 6; n++) {
const [{ issue }] = await client.unsafe(`insert into issues (company_id, title, status) values ($1, $2, 'in_progress') returning id as issue`,
[company, `Issue ${n}`]);
parkedIssues.push(issue);
const [{ conversation }] = await client.unsafe(`insert into chat_conversations (company_id, endpoint_id, issue_id,
external_conversation_id, external_label, external_thread_id) values ($1, $2, $3, $4, $4, $4) returning id as conversation`,
[company, endpoint, issue, `github:paperclipai/paperclip:${n}`]);
const [{ delivery }] = await client.unsafe(`insert into chat_deliveries (company_id, endpoint_id, provider_event_id, deduplication_key,
event_kind, normalized_event, conversation_id, state) values ($1, $2, $3, $3, 'message', '{}', $4, 'processed') returning id as delivery`,
[company, endpoint, `event-${n}`, conversation]);
const [{ comment }] = await client.unsafe(`insert into issue_comments (company_id, issue_id, body) values ($1, $2, 'Request')
returning id as comment`, [company, issue]);
await client.unsafe(`insert into chat_message_links (company_id, endpoint_id, conversation_id, delivery_id, comment_id,
provider_message_id, direction) values ($1, $2, $3, $4, $5, $6, 'inbound')`,
[company, endpoint, conversation, delivery, comment, `in-${n}`]);
await client.unsafe(`insert into chat_actions (company_id, endpoint_id, delivery_id, conversation_id, kind, provider_action_id, status, result)
values ($1, $2, $3, $4, 'github_response_comment', $5, $6, $7::jsonb)`,
[company, endpoint, delivery, conversation, `github_response:${delivery}`, n % 2 ? "received" : "processed",
n % 2 ? null : JSON.stringify({ id: `IC_${n}`, final: false })]);
await client.unsafe(`insert into heartbeat_runs (company_id, agent_id, status, runtime_mode, scope_kind, issue_id, native_issue_id,
context_snapshot, finished_at) values ($1, $2, 'succeeded', 'native', 'issue', $3, $3, $4::jsonb, now())`,
[company, agent, issue, JSON.stringify({ issueId: issue, wakeCommentId: comment })]);
}
}, 120_000);
afterEach(() => { vi.restoreAllMocks(); });
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
function sweeps() {
let reads = 0;
const counted = new Proxy(db, { get(target, key) {
if (key !== "select") return Reflect.get(target, key);
return (...fields: Parameters<Db["select"]>) => {
const builder = target.select(...fields), from = builder.from.bind(builder);
builder.from = ((...args: Parameters<typeof from>) => {
const selection = from(...args), execute = selection.execute.bind(selection);
selection.execute = (async () => {
if (selection.toSQL().sql.includes(`'wakeCommentIds', '[]'::jsonb) ?`)) reads += 1;
return execute();
}) as typeof selection.execute;
return selection;
}) as typeof builder.from;
return builder;
};
} }) as Db;
const tasks: Task[] = [];
const coordinator = { register: (_queue: unknown, task: Task) => { tasks.push(task); return { wake() {}, stop: async () => {} }; } };
const acknowledge = vi.fn(async () => undefined);
registerGitHubAcknowledgementWork(counted, coordinator as never, acknowledge, () => true);
registerGitHubTerminalResponseWork(counted, coordinator as never, acknowledge, () => true);
vi.spyOn(Date, "now").mockImplementation(() => clock);
const pass = async (hint?: string) => {
clock += 2_000;
const before = reads;
for (const task of tasks) {
if (hint) task.onSettled?.(new Set([hint]));
await task.run(new AbortController().signal);
}
return reads - before;
};
return { pass, acknowledge };
}
it("rebuilds parked requests once per interval for company-wide hints", async () => {
const { pass, acknowledge } = sweeps();
expect(await pass()).toBe(parkedIssues.length);
const companyHint = JSON.stringify({ companyId });
const repeated = [];
for (let index = 0; index < 5; index++) repeated.push(await pass(companyHint));
expect(repeated).toEqual([parkedIssues.length, 0, 0, 0, 0]);
clock += GITHUB_RESPONSE_COMPANY_RECONSTRUCTION_MS;
expect(await pass()).toBe(parkedIssues.length);
expect(acknowledge).not.toHaveBeenCalled();
}, 60_000);
it("still rebuilds an issue, request or other company immediately", async () => {
const { pass } = sweeps();
await pass();
await pass(JSON.stringify({ companyId }));
expect(await pass(JSON.stringify({ companyId, issueId: parkedIssues[0] }))).toBe(1);
expect(await pass(JSON.stringify({ companyId, issueId: parkedIssues[1] }))).toBe(1);
expect(await pass(JSON.stringify({ companyId, issueId: parkedIssues[0] }))).toBe(1);
expect(await pass(JSON.stringify({ companyId: randomUUID() }))).toBe(0);
}, 60_000);
});
@@ -0,0 +1,10 @@
import type { ChatResponseScope } from "../../chat-response-availability.js";
export type GitHubResponseSweepScope = ChatResponseScope & { endpointId?: string; repositoryId?: string; number?: number };
export const GITHUB_RESPONSE_COMPANY_RECONSTRUCTION_MS = 30_000;
export function githubResponseReconstructionIntervalMs(scope: GitHubResponseSweepScope | undefined) {
return scope && !scope.issueId && !scope.deliveryId && !scope.conversationId && !scope.endpointId
&& !scope.repositoryId && !scope.number ? GITHUB_RESPONSE_COMPANY_RECONSTRUCTION_MS : 0;
}
@@ -3,12 +3,10 @@ import { chatActions, chatConversations, chatDeliveries, chatEndpointLeases, cha
chatEndpoints, chatMessageLinks, heartbeatRuns, issues, type Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../../delivery-work-notifications.js";
import type { ChatResponseScope } from "../../chat-response-availability.js";
import { registerScopedDeadlineWork } from "../../scoped-deadline-work.js";
import { githubResponseExecution } from "./response-execution.js";
import { githubResponseReconstructionIntervalMs, type GitHubResponseSweepScope as Scope } from "./response-sweep-scope.js";
import { HttpError } from "../../../errors.js";
type Scope = ChatResponseScope & { endpointId?: string; repositoryId?: string; number?: number };
type Entry = { id: string; companyId: string; endpointId: string; deliveryId: string;
issueId: string; conversationId: string; repositoryId: string | null; number: number | null; dueAt: number };
@@ -87,6 +85,7 @@ export function registerGitHubTerminalResponseWork(
return registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.chatGitHubTerminalResponses,
reconstructionIntervalMs: githubResponseReconstructionIntervalMs,
decodeScope: (hint: string): Scope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (value: Entry, scope?: Scope) => !scope || (value.companyId === scope.companyId &&
(!scope.issueId || value.issueId === scope.issueId) && (!scope.deliveryId || value.deliveryId === scope.deliveryId) &&