Files
PaperClipAI/server/src/services/chat-channels/github/acknowledgement-work.ts
T
DottaandClaude Opus 5.5 f4f4902707 perf(chat): read the run state of a GitHub response sweep page in one batch (#21711)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel has two sweeps for request response
comments: the acknowledgement sweep (`acknowledgement-work.ts`) and the
terminal-response sweep (`terminal-response-work.ts`). Each reads pages
of up to 100 requests.
> - For each loaded request, both sweeps call `readGitHubResponseRun`
one row at a time. That is one latest-run statement, plus the run
identity context read and the message link read of
`chatRunSourceMessageId`. A page of 100 requests costs about 300
statements.
> - On paperclip.staging, those two provenance reads ran at about 13/s
each. #21705 stops the idle backlog from being reloaded. This PR makes
the reads that remain cost a constant number of statements per page.
> - The per-row read also had a failure mode. A 403 from one run's
provenance check aborted the acknowledgement sweep's whole page, so no
request of that company was acknowledged until the 403 went away.
> - This pull request adds a batched `readGitHubResponseRuns` and uses
it once per page in both sweeps.
> - The benefit is a constant number of statements per page, and a
provenance failure that affects only its own request.

## Linked Issues or Issue Description

Refs #21705 (the same sweeps; that PR stops reloading requests that
cannot progress).

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On paperclip.staging (2026-11-15, about 01:45–02:25Z),
`provenanceReads.context` (`select … from run_identity_contexts where id
= $1 and run_id = $2 and company_id = $3 and status = $4 limit $5`) ran
at 13.37/s, and `provenanceReads.link` (`select
chat_conversations.issue_id, chat_endpoints.provider from
chat_message_links join … where chat_message_links.company_id = $3 and
comment_id = $4 …`) ran at 13.20/s. Both GitHub response sweeps reach
them through `readGitHubResponseRun` → `chatRunSourceMessageId(db, run,
{ provider: "github" })`, once for each loaded row. `load()` costs one
page read plus three statements per row. `check()` read the run state
twice for each due request (before and after the acknowledgement).

In the acknowledgement sweep, a 403 from `chatRunSourceMessageId` (for
example, a run whose identity context is no longer accepted) threw out
of `entry()` and aborted the whole page. The terminal sweep already kept
such a row due, so the acknowledgement owner could revoke it.

**Expected behavior**

A page of requests costs a constant number of statements. A failure for
one run affects only the requests of that run.

**Steps to reproduce**

1. Create 120 GitHub response requests on 30 tasks. Give each task an
ended run with an accepted identity context.
2. Register the terminal-response sweep. On master, the first pass costs
one page read plus three statements per row.
3. Mark the requests `received`, revoke one run's identity context, and
register the acknowledgement sweep. On master, the load fails with a
403, and none of the 150 requests is acknowledged.

**Paperclip version or commit**

master at ea8c734cb9 (includes #21705).

## What Changed

- `server/src/services/chat-channels/github/response-run-state.ts`:
- The latest-run subquery and the "woke for this request" check are
built once, from literals (single read) or from the columns of a request
row (batch read). The single read keeps the same query shape.
- New `readGitHubResponseRuns(db, inputs)`. It reads the latest run of
every request of a page in one statement: `unnest(…) with ordinality`
and a lateral latest-run subquery, then a join to `heartbeat_runs` by id
and company. It then calls the existing `chatRunSourceMessageIds(db,
runs, { allowLegacyWakeComment: true, provider: "github" })` once for
the page's distinct runs. That helper reuses `provenanceReads`, so the
scope, status filters and joins are the same as the single check.
Results map back by position. A run whose provenance check is forbidden
gives `{ forbidden }` for its requests only.
- `server/src/services/chat-channels/github/response-execution.ts`:
`githubResponseExecution` takes an optional pre-read state. It throws a
`{ forbidden }` state as the same 403 that the single read throws.
- `server/src/services/chat-channels/github/terminal-response-work.ts`:
`load()` reads each page's run state with one `readGitHubResponseRuns`
call. `check()` reads the run state once: after the acknowledgement it
reads the row only, and keeps a request that is still owed for the next
check, which reads its run state again first.
- `server/src/services/chat-channels/github/acknowledgement-work.ts`:
the same batching. A forbidden provenance check keeps that request due,
so the acknowledgement owner revokes it (`authorization_changed`), and
the rest of the page continues. After an acknowledgement, a still-owed
request keeps the deadline from its own row (retry time, fast-response
expiry or lease).
- Tests:
- `terminal-response-work.test.ts`: `reads a page of waiting responses
in a constant number of statements` and `acknowledges every owed request
of a page when one run's requester provenance changed`.
- `response-run-state.test.ts`: `reads a page of requests in one
latest-run statement with the state each single read returns` compares
the batch with the single read for every seeded binding and wake link.
- `response-sweep-scope.test.ts`: `rebuilds parked requests once per
interval for company-wide hints` counted run-state *statements* as a
proxy for requests reloaded. With batching, one statement reads a whole
page, so the count dropped from 6 to 2 and the test failed in CI. The
behavior it guards must stay: a company-wide rebuild reloads every
eligible request once per interval, and repeated hints inside the
interval reload none. The test now counts the requests that each
run-state read names (the length of a batched read's request arrays, or
1 for a single read). The expectations stay 6 / `[6, 0, 0, 0, 0]` / 6,
so the test still proves the same rebuild behavior.

No GitHub comment behavior changes.

## Verification

- `pnpm vitest run
src/services/chat-channels/github/terminal-response-work.test.ts`
(server), staging-shaped fixtures (120 waiting requests on 30 tasks;
codes null / `acknowledgement_unconfirmed` /
`acknowledgement_superseded`):
- before (#21705 only): the first pass costs 364 statements, with 120
run identity reads. With one revoked context, the acknowledgement sweep
acknowledges 0 of 150 owed requests. The 2 new tests fail (red).
- after: the first pass costs 10 statements, with 2 run identity reads
(per page: page, latest runs, context, link, wait keys). All 150 owed
requests are acknowledged, including the 5 whose provenance is
forbidden. 8 of 8 tests pass (green). All 7 files under
`chat-channels/github/`: 60 passed. `idle-completed-history.test.ts`: 75
passed.
- On master before #21705, the first pass was 362 statements, and every
company rebuild repeated it.
- `pnpm vitest run
src/services/chat-channels/github/response-run-state.test.ts`: 3 passed.
- `pnpm vitest run src/__tests__/chat-channels.integration.test.ts -t
"terminal|acknowledg|GitHub response|response comment|GitHub review"`:
192 passed.
- `pnpm typecheck` (server): passed.
- `pnpm verify` on #21705 selected 2,738 of 3,565 files (close to the
full suite). I relied on the focused runs and required CI, per
AGENTS.md.

## Risks

- The batched latest-run statement must return the same run as the
single read. The equivalence test covers native and legacy bindings,
ties, other agents and other companies.
- `check()` no longer reads the run state again right after an
acknowledgement. A request that is still owed keeps the deadline from
its own row and is read in full at its next check. The acknowledgement
still revalidates everything before it writes.
- A database error still rejects the whole page, as before. Only the 403
provenance decision is isolated per request.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL tests) and
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:15:08 -06:00

131 lines
8.8 KiB
TypeScript

import { readGitHubResponseRuns, type GitHubResponseRunRead } from "./response-run-state.js";
import { githubRunReplyState } from "../../chat-run-publications.js";
import { and, asc, eq, gt, isNotNull, or, sql } from "drizzle-orm";
import { chatActions, chatConversations, chatEndpointLeases, chatEndpointResources,
chatPublications, fastResponseRequests, chatEndpoints, chatMessageLinks, type Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../../delivery-work-notifications.js";
import { registerScopedDeadlineWork } from "../../scoped-deadline-work.js";
import { githubResponseReconstructionIntervalMs, type GitHubResponseSweepScope as Scope } from "./response-sweep-scope.js";
type Entry = { id: string; companyId: string; endpointId: string; deliveryId: string;
issueId: string | null; conversationId: string | null; repositoryId: string | null; number: number | null; dueAt: number };
type FastResponse = { status: string; expiresAt: string; publicationState: string | null };
export function registerGitHubAcknowledgementWork(
db: Db,
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
acknowledge: (deliveryId: string, canContinue: () => boolean) => Promise<unknown>,
canRun: () => boolean,
) {
const repositoryId = sql<string | null>`${chatEndpointResources.metadata}->>'providerRepositoryId'`;
const number = sql<string | null>`substring(${chatConversations.externalThreadId}
from '^github:[^:]+:(?:issue:)?([1-9][0-9]*)(?::rc:[1-9][0-9]*)?$')`;
const selection = {
id: chatActions.id, companyId: chatActions.companyId, endpointId: chatActions.endpointId,
deliveryId: chatActions.deliveryId, conversationId: chatActions.conversationId, issueId: chatConversations.issueId,
agentId: chatEndpoints.assignedAgentId,
sourceMessageId: sql<string | null>`(select m.comment_id from ${chatMessageLinks} m
where m.company_id = ${chatActions.companyId} and m.endpoint_id = ${chatActions.endpointId}
and m.delivery_id = ${chatActions.deliveryId} and m.direction = 'inbound' limit 1)`,
repositoryId, number, status: chatActions.status, result: chatActions.result,
leaseExpiresAt: sql<string | null>`(select max(l.expires_at) from ${chatEndpointLeases} l
where l.company_id = ${chatActions.companyId} and l.endpoint_id = ${chatActions.endpointId}
and l.lease_key = 'github-publication:' || (${repositoryId}) || ':' || ${number})`,
fast: sql<FastResponse | null>`(select jsonb_build_object('status', f.status, 'expiresAt', f.expires_at,
'publicationState', p.state) from ${fastResponseRequests} f left join ${chatPublications} p
on p.company_id = f.company_id and p.comment_id = f.comment_id
and p.endpoint_id = f.endpoint_id and p.conversation_id = f.conversation_id
where f.company_id = ${chatActions.companyId} and f.endpoint_id = ${chatActions.endpointId}
and f.delivery_id = ${chatActions.deliveryId} limit 1)`,
};
async function page(scope?: Scope, cursor?: string, actionId?: string) {
return db.select(selection).from(chatActions)
.innerJoin(chatEndpoints, and(eq(chatEndpoints.id, chatActions.endpointId), eq(chatEndpoints.companyId, chatActions.companyId)))
.leftJoin(chatConversations, and(eq(chatConversations.id, chatActions.conversationId), eq(chatConversations.companyId, chatActions.companyId)))
.leftJoin(chatEndpointResources, and(eq(chatEndpointResources.id, chatConversations.resourceId), eq(chatEndpointResources.companyId, chatActions.companyId)))
.where(and(eq(chatActions.kind, "github_response_comment"), isNotNull(chatActions.deliveryId),
sql`coalesce(${chatActions.result}->>'final', 'false') <> 'true'`,
or(eq(chatActions.status, "received"), and(eq(chatActions.status, "failed"), sql`${chatActions.result}->>'retryable' = 'true'`)),
scope ? eq(chatActions.companyId, scope.companyId) : undefined,
scope?.issueId ? eq(chatConversations.issueId, scope.issueId) : undefined,
scope?.conversationId ? eq(chatActions.conversationId, scope.conversationId) : undefined,
scope?.deliveryId ? eq(chatActions.deliveryId, scope.deliveryId) : undefined,
scope?.endpointId ? eq(chatActions.endpointId, scope.endpointId) : undefined,
scope?.repositoryId ? eq(repositoryId, scope.repositoryId) : undefined,
scope?.number ? eq(number, String(scope.number)) : undefined,
cursor ? gt(chatActions.id, cursor) : undefined,
actionId ? eq(chatActions.id, actionId) : undefined,
)).orderBy(asc(chatActions.id)).limit(100);
}
type Row = Awaited<ReturnType<typeof page>>[number];
async function entry(row: Row, read: GitHubResponseRunRead): Promise<Entry | null> {
// A request whose requester provenance changed stays due, so the
// acknowledgement owner revokes it. It must not stop the rest of the page.
if (!("forbidden" in read)) {
const { latestRun, ownsLatestRequest } = read;
// Execution success alone does not decide whether a final response exists.
// Presentation/publication commits invalidate this parked request.
if (row.issueId && ownsLatestRequest && latestRun?.nativePresentationPending &&
await githubRunReplyState(db, { companyId: row.companyId, endpointId: row.endpointId,
issueId: row.issueId, runId: latestRun.id }) === "none") return null;
}
return schedule(row);
}
/** The entries of a page, by position, from one batched run-state read. */
async function entries(rows: Row[]) {
const reads = await readGitHubResponseRuns(db, rows.map(row => ({ companyId: row.companyId, endpointId: row.endpointId,
agentId: row.agentId, issueId: row.issueId, deliveryId: row.deliveryId!, sourceMessageId: row.sourceMessageId })));
const values: Array<Entry | null> = [];
for (const [index, row] of rows.entries()) values.push(await entry(row, reads[index]!));
return values;
}
async function current(actionId: string) {
const [row] = await page(undefined, undefined, actionId);
return row ? (await entries([row]))[0] ?? null : null;
}
/** When the request is due, from its own row. */
function schedule(row: Row): Entry | null {
let dueAt = row.status === "failed" ? Date.parse(String(row.result?.retryAt)) : 0;
if (!Number.isFinite(dueAt)) throw new Error("GitHub acknowledgement has an invalid retry deadline");
const fast = row.fast;
if (fast?.publicationState === "streaming") return null;
if (fast && !["published", "delivery_unknown"].includes(fast.publicationState ?? "") &&
(["pending", "running"].includes(fast.status) || ["pending", "retry"].includes(fast.publicationState ?? ""))) {
dueAt = Math.max(dueAt, Date.parse(fast.expiresAt));
}
if (row.leaseExpiresAt) dueAt = Math.max(dueAt, Date.parse(row.leaseExpiresAt));
return { id: row.id, companyId: row.companyId, endpointId: row.endpointId, deliveryId: row.deliveryId!,
issueId: row.issueId, conversationId: row.conversationId, repositoryId: row.repositoryId, number: row.number ? Number(row.number) : null, dueAt };
}
return registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.chatGitHubAcknowledgements,
reconstructionIntervalMs: githubResponseReconstructionIntervalMs,
decodeScope: (hint: string): Scope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (value: Entry, scope?: Scope) => !scope || (value.companyId === scope.companyId &&
(!scope.issueId || value.issueId === scope.issueId) && (!scope.deliveryId || value.deliveryId === scope.deliveryId) &&
(!scope.conversationId || value.conversationId === scope.conversationId) &&
(!scope.endpointId || value.endpointId === scope.endpointId) && (!scope.repositoryId || value.repositoryId === scope.repositoryId) &&
(!scope.number || value.number === scope.number)),
load: async (scope?: Scope, signal?: AbortSignal) => {
const loaded: Entry[] = [];
let cursor: string | undefined;
while (canRun() && !signal?.aborted) {
const rows = await page(scope, cursor);
for (const value of await entries(rows)) if (value) loaded.push(value);
if (rows.length < 100) return loaded;
cursor = rows[rows.length - 1].id;
}
throw signal?.reason ?? new Error("GitHub acknowledgement recovery stopped");
},
check: async (value: Entry, signal?: AbortSignal) => {
const due = await current(value.id);
if (!due || due.dueAt > Date.now() || !canRun() || signal?.aborted) return due;
await acknowledge(due.deliveryId, () => canRun() && !signal?.aborted);
// A request that is still owed keeps its saved deadline. The next check
// reads its run state again before it acknowledges.
const [remaining] = await page(undefined, undefined, value.id);
return remaining ? schedule(remaining) : null;
},
}, canRun);
}