From a1c37b5cc4d758c8194a8102f0250a7ff08f574b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 02:05:45 -0600 Subject: [PATCH] perf(chat): rebuild GitHub response sweeps for company-wide hints at most every 30 seconds (#21580) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel has two background sweeps for request response comments: acknowledgements, and terminal responses. Each sweep rebuilds its candidate set when a hint arrives, and it reads the latest run of each candidate request (`readGitHubResponseRun`) on each rebuild > - A run milestone for a run with no task sends a company-wide hint to both sweeps. Examples: each status change of a run without a task (`setRunStatus`), and a comment merged into an active run in the wake queue. In a busy company these hints arrive more than once a second > - So both sweeps rebuild the whole company about once a second (their minimum pass interval). Each rebuild reads every parked request again, although nothing about those requests changed. On a hosted test database, `readGitHubResponseRun` ran 6.7 times a second > - This pull request rebuilds a GitHub response sweep for a company-wide hint at most once every 30 seconds. Issue, request, conversation, endpoint and lease hints still rebuild immediately. `external-objects.ts` uses the same rule for its refresh sweep > - The benefit is that a steady stream of company-wide hints no longer makes each sweep read every parked request every second ## Linked Issues or Issue Description Refs #21577 (makes each `readGitHubResponseRun` call cheap; this pull request reduces how often the sweeps call it). Refs #18384 (the same reconstruction interval for company-wide external-object hints). **What existing behavior does this improve?** The candidate rebuild of `registerGitHubAcknowledgementWork` (`acknowledgement-work.ts`) and `registerGitHubTerminalResponseWork` (`terminal-response-work.ts`) for company-wide hints. **Subsystem affected** GitHub chat channel: request acknowledgement and terminal-response recovery sweeps. **Current behavior** `notifyChatRunMilestones` calls `notifyChatResponseAvailability` with its scope. When the scope has no issue (only a company, or a company and a run), the hint for both GitHub response queues is company-wide (`{"companyId": ...}`), because the response scope has no run field. Each company-wide hint makes each sweep load every candidate of the company and call `readGitHubResponseRun` for each one. Passes are limited to one a second, so with a steady hint stream the cost is about (parked candidates) reads a second. **Proposed behavior** Both sweeps set `reconstructionIntervalMs`. A company-wide scope (company ID only) waits 30 seconds after its last rebuild before it rebuilds again. All narrower scopes, the startup scope (`*`), and saved deadlines are not changed. Entries that are already loaded stay available for dispatch while a company-wide hint waits, and each check still reads the request again from the database. **Reason and benefit** On the hosted test database, `readGitHubResponseRun` ran 6.7 times a second for no change in state. In a local reproduction, the call rate drops from about one call for each parked request each second to one call for each parked request each 30 seconds. **Breaking changes** None. ## What Changed - Add `response-sweep-scope.ts` with the shared scope type of the two GitHub response sweeps and `githubResponseReconstructionIntervalMs`: 30 seconds for a company-wide scope, 0 for all other scopes. - Use it as `reconstructionIntervalMs` in the acknowledgement sweep and the terminal-response sweep. Both files now import the shared scope type instead of declaring it. - Add `response-sweep-scope.test.ts`. It registers both real sweeps on a real database with 6 parked requests (3 for each sweep) and a controlled clock. It sends company-wide hints and issue hints, and counts `readGitHubResponseRun` statements on each pass. ## Verification Cause, from a local reproduction with the real delivery coordinator and both real sweeps on embedded PostgreSQL 18.1, with 8 parked requests (4 for each sweep), each with a succeeded native run that waits for its presentation: | Hints | Hints a second | `readGitHubResponseRun` calls a second | | --- | --- | --- | | None | 0 | 0 | | Company-wide run milestone (`{ companyId, runId }`) | 9.7 | 8.75 | | Company-wide run milestone | 1.0 | 7.98 | | Run milestone for an unrelated issue | 9.7 | 0 | So the call rate is about one call for each parked request each second, from company-wide hints only. The hosted rate (6.7 a second) is consistent with about 7 parked GitHub requests in one company. Red/green regression test: - `cd server && npx vitest run src/services/chat-channels/github/response-sweep-scope.test.ts` - Red (without the two `reconstructionIntervalMs` lines): `expected [ 6, 6, 6, 6, 6 ] to deeply equal [ 6, +0, +0, +0, +0 ]`. Each of 5 company-wide hints rebuilt all 6 parked requests. - Green: 2 of 2 pass. The first company-wide hint rebuilds once (6 reads). The next 4 hints read nothing. After 30 seconds, the waiting hint rebuilds (6 reads). Issue hints still read exactly the one request of that issue at once, and a hint for another company reads nothing. No acknowledgement is sent for a parked request. Other checks: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed (all GitHub chat-channel integration tests, including the acknowledgement and terminal-response sweeps). - `cd server && npx tsc --noEmit`: passed. - After the rebase onto #21577: both new test files pass (4 of 4), and the server typecheck passes. - `pnpm verify` selected most of the server suite, because the sweep modules are imported widely. I stopped that broad local run. Required CI supplies that coverage. ## Risks - A change that only a company-wide hint reports can take up to 30 seconds to rebuild a GitHub response sweep. I checked the callers. Changes to one request use issue, delivery, conversation or lease hints: run status changes with a task, native presentation commits (`status-decision-committer.ts`, `native-run-finalizer.ts`, `native-chat-review-presentation.ts`), publications, fast responses, and the publication lease. Those stay immediate. Company-wide hints come from runs with no task, from a comment merged into an active run (the wake queue sends only the run ID), and from company authority changes. A run with no task cannot change the latest run of a GitHub request's task. For a merged comment, the active run's next status change sends an issue hint, and staging the request's response sends a delivery hint; both rebuild at once. Only a mid-run ownership update can wait up to 30 seconds. For an authority change, each check reads the request and its authority again before it acts. - The deadline reconciliation snapshot reports these two sweeps as not reconciled while a company-wide hint waits (up to 30 seconds). This is the same as the external-object refresh sweep. - The startup rebuild (`*`) and failure retries do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude Code with tool use (shell, file edits, and local PostgreSQL reproductions). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .../github/acknowledgement-work.ts | 5 +- .../github/response-sweep-scope.test.ts | 118 ++++++++++++++++++ .../github/response-sweep-scope.ts | 10 ++ .../github/terminal-response-work.ts | 5 +- 4 files changed, 132 insertions(+), 6 deletions(-) create mode 100644 server/src/services/chat-channels/github/response-sweep-scope.test.ts create mode 100644 server/src/services/chat-channels/github/response-sweep-scope.ts diff --git a/server/src/services/chat-channels/github/acknowledgement-work.ts b/server/src/services/chat-channels/github/acknowledgement-work.ts index 6bd60b0ed78..60e0db1cb7a 100644 --- a/server/src/services/chat-channels/github/acknowledgement-work.ts +++ b/server/src/services/chat-channels/github/acknowledgement-work.ts @@ -5,10 +5,8 @@ import { chatActions, chatConversations, chatEndpointLeases, chatEndpointResourc chatPublications, fastResponseRequests, chatEndpoints, chatMessageLinks, type Db } from "@paperclipai/db"; import type { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js"; import { DELIVERY_QUEUES } from "../../delivery-work-notifications.js"; -import type { ChatResponseScope } from "../../chat-response-availability.js"; import { registerScopedDeadlineWork } from "../../scoped-deadline-work.js"; - -type Scope = ChatResponseScope & { endpointId?: string; repositoryId?: string; number?: number }; +import { githubResponseReconstructionIntervalMs, type GitHubResponseSweepScope as Scope } from "./response-sweep-scope.js"; type Entry = { id: string; companyId: string; endpointId: string; deliveryId: string; issueId: string | null; conversationId: string | null; repositoryId: string | null; number: number | null; dueAt: number }; type FastResponse = { status: string; expiresAt: string; publicationState: string | null }; @@ -83,6 +81,7 @@ export function registerGitHubAcknowledgementWork( } return registerScopedDeadlineWork(coordinator, { queue: DELIVERY_QUEUES.chatGitHubAcknowledgements, + reconstructionIntervalMs: githubResponseReconstructionIntervalMs, decodeScope: (hint: string): Scope | undefined => hint === "*" ? undefined : JSON.parse(hint), matches: (value: Entry, scope?: Scope) => !scope || (value.companyId === scope.companyId && (!scope.issueId || value.issueId === scope.issueId) && (!scope.deliveryId || value.deliveryId === scope.deliveryId) && diff --git a/server/src/services/chat-channels/github/response-sweep-scope.test.ts b/server/src/services/chat-channels/github/response-sweep-scope.test.ts new file mode 100644 index 00000000000..f162a1689ec --- /dev/null +++ b/server/src/services/chat-channels/github/response-sweep-scope.test.ts @@ -0,0 +1,118 @@ +import { randomUUID } from "node:crypto"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { createDb, type Db } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js"; +import { registerGitHubAcknowledgementWork } from "./acknowledgement-work.js"; +import { registerGitHubTerminalResponseWork } from "./terminal-response-work.js"; +import { GITHUB_RESPONSE_COMPANY_RECONSTRUCTION_MS } from "./response-sweep-scope.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const suite = support.supported ? describe : describe.skip; + +type Task = { run(signal: AbortSignal): Promise; onSettled?(hints?: ReadonlySet): void }; + +suite("GitHub response sweep scopes", () => { + let db!: Db; + let temporary: Awaited> | null = null; + let companyId = ""; + const parkedIssues: string[] = []; + let clock = Date.now(); + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("paperclip-github-response-sweep-scope-"); + db = createDb(temporary.connectionString); + const client = db.$client; + const [{ company }] = await client.unsafe(`insert into companies (name, issue_prefix) values ('Sweep', 'SW') returning id as company`); + companyId = company; + const [{ agent }] = await client.unsafe(`insert into agents (company_id, name) values ($1, 'GitHub agent') returning id as agent`, [company]); + const [{ application }] = await client.unsafe(`insert into tool_applications (company_id, application_key, name, type) + values ($1, 'github', 'GitHub', 'chat') returning id as application`, [company]); + const [{ connection }] = await client.unsafe(`insert into tool_connections (company_id, application_id, name, uid, transport, connection_purpose) + values ($1, $2, 'GitHub', 'github-sweep', 'chat_sdk', 'channel') returning id as connection`, [company, application]); + const [{ endpoint }] = await client.unsafe(`insert into chat_endpoints (company_id, connection_id, provider, public_id, assigned_agent_id, status) + values ($1, $2, 'github', 'sweep', $3, 'active') returning id as endpoint`, [company, connection, agent]); + for (let n = 1; n <= 6; n++) { + const [{ issue }] = await client.unsafe(`insert into issues (company_id, title, status) values ($1, $2, 'in_progress') returning id as issue`, + [company, `Issue ${n}`]); + parkedIssues.push(issue); + const [{ conversation }] = await client.unsafe(`insert into chat_conversations (company_id, endpoint_id, issue_id, + external_conversation_id, external_label, external_thread_id) values ($1, $2, $3, $4, $4, $4) returning id as conversation`, + [company, endpoint, issue, `github:paperclipai/paperclip:${n}`]); + const [{ delivery }] = await client.unsafe(`insert into chat_deliveries (company_id, endpoint_id, provider_event_id, deduplication_key, + event_kind, normalized_event, conversation_id, state) values ($1, $2, $3, $3, 'message', '{}', $4, 'processed') returning id as delivery`, + [company, endpoint, `event-${n}`, conversation]); + const [{ comment }] = await client.unsafe(`insert into issue_comments (company_id, issue_id, body) values ($1, $2, 'Request') + returning id as comment`, [company, issue]); + await client.unsafe(`insert into chat_message_links (company_id, endpoint_id, conversation_id, delivery_id, comment_id, + provider_message_id, direction) values ($1, $2, $3, $4, $5, $6, 'inbound')`, + [company, endpoint, conversation, delivery, comment, `in-${n}`]); + await client.unsafe(`insert into chat_actions (company_id, endpoint_id, delivery_id, conversation_id, kind, provider_action_id, status, result) + values ($1, $2, $3, $4, 'github_response_comment', $5, $6, $7::jsonb)`, + [company, endpoint, delivery, conversation, `github_response:${delivery}`, n % 2 ? "received" : "processed", + n % 2 ? null : JSON.stringify({ id: `IC_${n}`, final: false })]); + await client.unsafe(`insert into heartbeat_runs (company_id, agent_id, status, runtime_mode, scope_kind, issue_id, native_issue_id, + context_snapshot, finished_at) values ($1, $2, 'succeeded', 'native', 'issue', $3, $3, $4::jsonb, now())`, + [company, agent, issue, JSON.stringify({ issueId: issue, wakeCommentId: comment })]); + } + }, 120_000); + + afterEach(() => { vi.restoreAllMocks(); }); + afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); }); + + function sweeps() { + let reads = 0; + const counted = new Proxy(db, { get(target, key) { + if (key !== "select") return Reflect.get(target, key); + return (...fields: Parameters) => { + const builder = target.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const selection = from(...args), execute = selection.execute.bind(selection); + selection.execute = (async () => { + if (selection.toSQL().sql.includes(`'wakeCommentIds', '[]'::jsonb) ?`)) reads += 1; + return execute(); + }) as typeof selection.execute; + return selection; + }) as typeof builder.from; + return builder; + }; + } }) as Db; + const tasks: Task[] = []; + const coordinator = { register: (_queue: unknown, task: Task) => { tasks.push(task); return { wake() {}, stop: async () => {} }; } }; + const acknowledge = vi.fn(async () => undefined); + registerGitHubAcknowledgementWork(counted, coordinator as never, acknowledge, () => true); + registerGitHubTerminalResponseWork(counted, coordinator as never, acknowledge, () => true); + vi.spyOn(Date, "now").mockImplementation(() => clock); + const pass = async (hint?: string) => { + clock += 2_000; + const before = reads; + for (const task of tasks) { + if (hint) task.onSettled?.(new Set([hint])); + await task.run(new AbortController().signal); + } + return reads - before; + }; + return { pass, acknowledge }; + } + + it("rebuilds parked requests once per interval for company-wide hints", async () => { + const { pass, acknowledge } = sweeps(); + expect(await pass()).toBe(parkedIssues.length); + const companyHint = JSON.stringify({ companyId }); + const repeated = []; + for (let index = 0; index < 5; index++) repeated.push(await pass(companyHint)); + expect(repeated).toEqual([parkedIssues.length, 0, 0, 0, 0]); + clock += GITHUB_RESPONSE_COMPANY_RECONSTRUCTION_MS; + expect(await pass()).toBe(parkedIssues.length); + expect(acknowledge).not.toHaveBeenCalled(); + }, 60_000); + + it("still rebuilds an issue, request or other company immediately", async () => { + const { pass } = sweeps(); + await pass(); + await pass(JSON.stringify({ companyId })); + expect(await pass(JSON.stringify({ companyId, issueId: parkedIssues[0] }))).toBe(1); + expect(await pass(JSON.stringify({ companyId, issueId: parkedIssues[1] }))).toBe(1); + expect(await pass(JSON.stringify({ companyId, issueId: parkedIssues[0] }))).toBe(1); + expect(await pass(JSON.stringify({ companyId: randomUUID() }))).toBe(0); + }, 60_000); +}); diff --git a/server/src/services/chat-channels/github/response-sweep-scope.ts b/server/src/services/chat-channels/github/response-sweep-scope.ts new file mode 100644 index 00000000000..b4d0290c48e --- /dev/null +++ b/server/src/services/chat-channels/github/response-sweep-scope.ts @@ -0,0 +1,10 @@ +import type { ChatResponseScope } from "../../chat-response-availability.js"; + +export type GitHubResponseSweepScope = ChatResponseScope & { endpointId?: string; repositoryId?: string; number?: number }; + +export const GITHUB_RESPONSE_COMPANY_RECONSTRUCTION_MS = 30_000; + +export function githubResponseReconstructionIntervalMs(scope: GitHubResponseSweepScope | undefined) { + return scope && !scope.issueId && !scope.deliveryId && !scope.conversationId && !scope.endpointId + && !scope.repositoryId && !scope.number ? GITHUB_RESPONSE_COMPANY_RECONSTRUCTION_MS : 0; +} diff --git a/server/src/services/chat-channels/github/terminal-response-work.ts b/server/src/services/chat-channels/github/terminal-response-work.ts index cb1f240bb9e..fd60d7538e5 100644 --- a/server/src/services/chat-channels/github/terminal-response-work.ts +++ b/server/src/services/chat-channels/github/terminal-response-work.ts @@ -3,12 +3,10 @@ import { chatActions, chatConversations, chatDeliveries, chatEndpointLeases, cha chatEndpoints, chatMessageLinks, heartbeatRuns, issues, type Db } from "@paperclipai/db"; import type { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js"; import { DELIVERY_QUEUES } from "../../delivery-work-notifications.js"; -import type { ChatResponseScope } from "../../chat-response-availability.js"; import { registerScopedDeadlineWork } from "../../scoped-deadline-work.js"; import { githubResponseExecution } from "./response-execution.js"; +import { githubResponseReconstructionIntervalMs, type GitHubResponseSweepScope as Scope } from "./response-sweep-scope.js"; import { HttpError } from "../../../errors.js"; - -type Scope = ChatResponseScope & { endpointId?: string; repositoryId?: string; number?: number }; type Entry = { id: string; companyId: string; endpointId: string; deliveryId: string; issueId: string; conversationId: string; repositoryId: string | null; number: number | null; dueAt: number }; @@ -87,6 +85,7 @@ export function registerGitHubTerminalResponseWork( return registerScopedDeadlineWork(coordinator, { queue: DELIVERY_QUEUES.chatGitHubTerminalResponses, + reconstructionIntervalMs: githubResponseReconstructionIntervalMs, decodeScope: (hint: string): Scope | undefined => hint === "*" ? undefined : JSON.parse(hint), matches: (value: Entry, scope?: Scope) => !scope || (value.companyId === scope.companyId && (!scope.issueId || value.issueId === scope.issueId) && (!scope.deliveryId || value.deliveryId === scope.deliveryId) &&