Files
PaperClipAI/server/src/services/chat-channels/github/response-run-state.ts
T
DottaandClaude Opus 5.5 f4f4902707 perf(chat): read the run state of a GitHub response sweep page in one batch (#21711)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel has two sweeps for request response
comments: the acknowledgement sweep (`acknowledgement-work.ts`) and the
terminal-response sweep (`terminal-response-work.ts`). Each reads pages
of up to 100 requests.
> - For each loaded request, both sweeps call `readGitHubResponseRun`
one row at a time. That is one latest-run statement, plus the run
identity context read and the message link read of
`chatRunSourceMessageId`. A page of 100 requests costs about 300
statements.
> - On paperclip.staging, those two provenance reads ran at about 13/s
each. #21705 stops the idle backlog from being reloaded. This PR makes
the reads that remain cost a constant number of statements per page.
> - The per-row read also had a failure mode. A 403 from one run's
provenance check aborted the acknowledgement sweep's whole page, so no
request of that company was acknowledged until the 403 went away.
> - This pull request adds a batched `readGitHubResponseRuns` and uses
it once per page in both sweeps.
> - The benefit is a constant number of statements per page, and a
provenance failure that affects only its own request.

## Linked Issues or Issue Description

Refs #21705 (the same sweeps; that PR stops reloading requests that
cannot progress).

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.

**What happened?**

On paperclip.staging (2026-11-15, about 01:45–02:25Z),
`provenanceReads.context` (`select … from run_identity_contexts where id
= $1 and run_id = $2 and company_id = $3 and status = $4 limit $5`) ran
at 13.37/s, and `provenanceReads.link` (`select
chat_conversations.issue_id, chat_endpoints.provider from
chat_message_links join … where chat_message_links.company_id = $3 and
comment_id = $4 …`) ran at 13.20/s. Both GitHub response sweeps reach
them through `readGitHubResponseRun` → `chatRunSourceMessageId(db, run,
{ provider: "github" })`, once for each loaded row. `load()` costs one
page read plus three statements per row. `check()` read the run state
twice for each due request (before and after the acknowledgement).

In the acknowledgement sweep, a 403 from `chatRunSourceMessageId` (for
example, a run whose identity context is no longer accepted) threw out
of `entry()` and aborted the whole page. The terminal sweep already kept
such a row due, so the acknowledgement owner could revoke it.

**Expected behavior**

A page of requests costs a constant number of statements. A failure for
one run affects only the requests of that run.

**Steps to reproduce**

1. Create 120 GitHub response requests on 30 tasks. Give each task an
ended run with an accepted identity context.
2. Register the terminal-response sweep. On master, the first pass costs
one page read plus three statements per row.
3. Mark the requests `received`, revoke one run's identity context, and
register the acknowledgement sweep. On master, the load fails with a
403, and none of the 150 requests is acknowledged.

**Paperclip version or commit**

master at ea8c734cb9 (includes #21705).

## What Changed

- `server/src/services/chat-channels/github/response-run-state.ts`:
- The latest-run subquery and the "woke for this request" check are
built once, from literals (single read) or from the columns of a request
row (batch read). The single read keeps the same query shape.
- New `readGitHubResponseRuns(db, inputs)`. It reads the latest run of
every request of a page in one statement: `unnest(…) with ordinality`
and a lateral latest-run subquery, then a join to `heartbeat_runs` by id
and company. It then calls the existing `chatRunSourceMessageIds(db,
runs, { allowLegacyWakeComment: true, provider: "github" })` once for
the page's distinct runs. That helper reuses `provenanceReads`, so the
scope, status filters and joins are the same as the single check.
Results map back by position. A run whose provenance check is forbidden
gives `{ forbidden }` for its requests only.
- `server/src/services/chat-channels/github/response-execution.ts`:
`githubResponseExecution` takes an optional pre-read state. It throws a
`{ forbidden }` state as the same 403 that the single read throws.
- `server/src/services/chat-channels/github/terminal-response-work.ts`:
`load()` reads each page's run state with one `readGitHubResponseRuns`
call. `check()` reads the run state once: after the acknowledgement it
reads the row only, and keeps a request that is still owed for the next
check, which reads its run state again first.
- `server/src/services/chat-channels/github/acknowledgement-work.ts`:
the same batching. A forbidden provenance check keeps that request due,
so the acknowledgement owner revokes it (`authorization_changed`), and
the rest of the page continues. After an acknowledgement, a still-owed
request keeps the deadline from its own row (retry time, fast-response
expiry or lease).
- Tests:
- `terminal-response-work.test.ts`: `reads a page of waiting responses
in a constant number of statements` and `acknowledges every owed request
of a page when one run's requester provenance changed`.
- `response-run-state.test.ts`: `reads a page of requests in one
latest-run statement with the state each single read returns` compares
the batch with the single read for every seeded binding and wake link.
- `response-sweep-scope.test.ts`: `rebuilds parked requests once per
interval for company-wide hints` counted run-state *statements* as a
proxy for requests reloaded. With batching, one statement reads a whole
page, so the count dropped from 6 to 2 and the test failed in CI. The
behavior it guards must stay: a company-wide rebuild reloads every
eligible request once per interval, and repeated hints inside the
interval reload none. The test now counts the requests that each
run-state read names (the length of a batched read's request arrays, or
1 for a single read). The expectations stay 6 / `[6, 0, 0, 0, 0]` / 6,
so the test still proves the same rebuild behavior.

No GitHub comment behavior changes.

## Verification

- `pnpm vitest run
src/services/chat-channels/github/terminal-response-work.test.ts`
(server), staging-shaped fixtures (120 waiting requests on 30 tasks;
codes null / `acknowledgement_unconfirmed` /
`acknowledgement_superseded`):
- before (#21705 only): the first pass costs 364 statements, with 120
run identity reads. With one revoked context, the acknowledgement sweep
acknowledges 0 of 150 owed requests. The 2 new tests fail (red).
- after: the first pass costs 10 statements, with 2 run identity reads
(per page: page, latest runs, context, link, wait keys). All 150 owed
requests are acknowledged, including the 5 whose provenance is
forbidden. 8 of 8 tests pass (green). All 7 files under
`chat-channels/github/`: 60 passed. `idle-completed-history.test.ts`: 75
passed.
- On master before #21705, the first pass was 362 statements, and every
company rebuild repeated it.
- `pnpm vitest run
src/services/chat-channels/github/response-run-state.test.ts`: 3 passed.
- `pnpm vitest run src/__tests__/chat-channels.integration.test.ts -t
"terminal|acknowledg|GitHub response|response comment|GitHub review"`:
192 passed.
- `pnpm typecheck` (server): passed.
- `pnpm verify` on #21705 selected 2,738 of 3,565 files (close to the
full suite). I relied on the focused runs and required CI, per
AGENTS.md.

## Risks

- The batched latest-run statement must return the same run as the
single read. The equivalence test covers native and legacy bindings,
ties, other agents and other companies.
- `check()` no longer reads the run state again right after an
acknowledgement. A request that is still owed keeps the deadline from
its own row and is read in full at its next check. The acknowledgement
still revalidates everything before it writes.
- A database error still rejects the whole page, as before. Only the 403
provenance decision is isolated per request.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL tests) and
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:15:08 -06:00

145 lines
8.8 KiB
TypeScript

import { and, eq, sql, type SQL } from "drizzle-orm";
import { chatMessageLinks, heartbeatRuns, type Db } from "@paperclipai/db";
import type { HttpError } from "../../../errors.js";
import { chatRunSourceMessageId, chatRunSourceMessageIds } from "../run-origin.js";
export const responseRunColumns = {
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
responsibleUserId: heartbeatRuns.responsibleUserId,
activeIdentityContextId: heartbeatRuns.activeIdentityContextId,
nativeIssueId: heartbeatRuns.nativeIssueId,
issueId: heartbeatRuns.issueId,
status: heartbeatRuns.status,
updatedAt: heartbeatRuns.updatedAt,
// Terminal execution can precede native presentation commit. Until its
// durable decision exists, missing output is undecided rather than failure.
nativePresentationPending: sql<boolean>`coalesce(
${heartbeatRuns.status} = 'succeeded'
and ${heartbeatRuns.runtimeMode} = 'native'
and (${heartbeatRuns.nativePhase} is distinct from 'committed'
or ${heartbeatRuns.resultJson}->'presentationDecision'->>'schema' is distinct from 'paperclip.run_presentation_decision.v1'
or coalesce(${heartbeatRuns.resultJson}->'presentationDecision'->>'commentAction', '') not in ('none', 'create', 'reuse')),
false
)`,
// Project only the server-committed no-publication decision, never the
// potentially large native result or model-authored summary.
passiveResponseWait: sql<boolean>`coalesce(
${heartbeatRuns.status} = 'succeeded'
and ${heartbeatRuns.runtimeMode} = 'native'
and ${heartbeatRuns.nativePhase} = 'committed'
and ${heartbeatRuns.resultJson}->>'finalizationPhase' = 'committed'
and ${heartbeatRuns.resultJson}->>'finalizationReasonCode' in
('external_chat_response_waiting', 'preserved_result_status_unproven')
and ${heartbeatRuns.resultJson}->'nativeResult'->>'schema' = 'paperclip.run_result.v1'
and ${heartbeatRuns.resultJson}->'nativeResult'->>'reportedWorkDisposition' = 'yielded'
and ${heartbeatRuns.resultJson}->'nativeResult'->'continuation'->>'kind' = 'response_wake'
and ${heartbeatRuns.resultJson}->'presentationDecision'->>'schema' = 'paperclip.run_presentation_decision.v1'
and ${heartbeatRuns.resultJson}->'presentationDecision'->>'chosenSource' = 'none'
and ${heartbeatRuns.resultJson}->'presentationDecision'->>'commentAction' = 'none'
and ${heartbeatRuns.resultJson}->'presentationDecision'->'reasonCodes' @> '["yielded_control_plane_wait"]'::jsonb,
false
)`,
contextSnapshot: sql<Record<string, unknown>>`jsonb_build_object(
'issueId', ${heartbeatRuns.contextSnapshot}->'issueId',
'taskId', ${heartbeatRuns.contextSnapshot}->'taskId',
'wakeCommentId', ${heartbeatRuns.contextSnapshot}->'wakeCommentId',
'wakeCommentIds', ${heartbeatRuns.contextSnapshot}->'wakeCommentIds'
)`,
};
export type GitHubResponseRunInput = {
companyId: string; endpointId: string; agentId: string; issueId: string | null;
deliveryId: string; sourceMessageId: string | null;
};
type RunRow = Omit<Awaited<ReturnType<typeof readRunRows>>[number], "ordinal">;
export type GitHubResponseRunState = { latestRun: Omit<RunRow, "wokeForRequest"> | undefined; ownsLatestRequest: boolean };
/** The state of one request of a batch, or the 403 that its single read throws. */
export type GitHubResponseRunRead = GitHubResponseRunState | { forbidden: HttpError };
/** A value of the run reads: a literal of one request in a single read, or a
* column of the request row in a batch read. */
type Value = string | SQL;
type Of = Record<"companyId" | "endpointId" | "agentId" | "issueId" | "deliveryId", Value>;
/** The assigned agent's latest run bound to the issue, where a run's issue is
* coalesce(native_issue_id, context_snapshot->>'issueId'). The two precedence
* branches are disjoint and each uses an issue index. */
const latestRunId = (of: Of) => sql`(select latest.id from (
(select r.id, r.created_at from ${heartbeatRuns} r
where r.company_id = ${of.companyId}::uuid and r.native_issue_id = ${of.issueId}::uuid and r.agent_id = ${of.agentId}::uuid
order by r.created_at desc, r.id desc limit 1)
union all
(select r.id, r.created_at from ${heartbeatRuns} r
where r.company_id = ${of.companyId}::uuid and r.native_issue_id is null
and r.context_snapshot->>'issueId' = ${of.issueId}::text and r.agent_id = ${of.agentId}::uuid
order by r.created_at desc, r.id desc limit 1)
) latest order by latest.created_at desc, latest.id desc limit 1)`;
/** Whether the run woke for one of the request's inbound comments. */
const wokeForRequest = (of: Of) => sql<boolean>`exists (select 1 from ${chatMessageLinks} l
where l.company_id = ${of.companyId}::uuid and l.endpoint_id = ${of.endpointId}::uuid
and l.delivery_id = ${of.deliveryId}::uuid and l.direction = 'inbound'
and (l.comment_id::text = ${heartbeatRuns.contextSnapshot}->>'wakeCommentId'
or coalesce(${heartbeatRuns.contextSnapshot}->'wakeCommentIds', '[]'::jsonb) ? l.comment_id::text))`;
const request = sql.identifier("github_response_request");
const column = (name: string) => sql`${request}.${sql.identifier(name)}`;
const columns: Array<[keyof Of, string]> = [["companyId", "company_id"], ["endpointId", "endpoint_id"],
["agentId", "agent_id"], ["issueId", "issue_id"], ["deliveryId", "delivery_id"]];
/** The latest run of each request, in one statement. For each request row the
* lateral subquery is the single read with the request columns in place of
* the literals. */
async function readRunRows(db: Db, inputs: Array<GitHubResponseRunInput & { issueId: string }>) {
const of = Object.fromEntries(columns.map(([key, name]) => [key, column(name)])) as Of;
return db.select({ ordinal: sql<number>`${column("ordinal")}`.mapWith(Number), ...responseRunColumns,
wokeForRequest: wokeForRequest(of) })
.from(sql`unnest(${sql.join(columns.map(([key]) => sql`${sql.param(inputs.map(input => input[key]))}::uuid[]`), sql`, `)})
with ordinality as ${request}(${sql.join([...columns.map(([, name]) => sql.identifier(name)), sql.identifier("ordinal")], sql`, `)})
cross join lateral (select ${latestRunId(of)} as id) latest_request_run`)
.innerJoin(heartbeatRuns, and(sql`${heartbeatRuns.id} = latest_request_run.id`,
sql`${heartbeatRuns.companyId} = ${column("company_id")}`));
}
function ownership(input: GitHubResponseRunInput, row: RunRow | undefined, originMessageId: string | null): GitHubResponseRunState {
if (!row) return { latestRun: undefined, ownsLatestRequest: false };
const { wokeForRequest: requestRunIsLatest, ...latestRun } = row;
const ownsLatestRequest = requestRunIsLatest ||
(!!input.sourceMessageId && originMessageId === input.sourceMessageId);
return { latestRun, ownsLatestRequest };
}
/** Shared request ownership and presentation state for delivery and recovery. */
export async function readGitHubResponseRun(db: Db, input: GitHubResponseRunInput): Promise<GitHubResponseRunState> {
const { issueId } = input;
if (!issueId) return { latestRun: undefined, ownsLatestRequest: false };
const of = { ...input, issueId };
const [row] = await db.select({ ...responseRunColumns, wokeForRequest: wokeForRequest(of) }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, input.companyId), sql`${heartbeatRuns.id} = ${latestRunId(of)}`));
const originMessageId = row?.activeIdentityContextId ? await chatRunSourceMessageId(db, row,
{ allowLegacyWakeComment: true, provider: "github" }) : null;
return ownership(input, row, originMessageId);
}
/** `readGitHubResponseRun` for a page of requests: one latest-run read for the
* whole page, then one batched provenance check of its distinct runs, which
* reuses the single check's reads. Each request gets, by position, the state a
* separate call returns or the 403 that call throws. */
export async function readGitHubResponseRuns(db: Db, inputs: GitHubResponseRunInput[]): Promise<GitHubResponseRunRead[]> {
const bound = inputs.flatMap((input, index) => input.issueId ? [{ index, input: { ...input, issueId: input.issueId } }] : []);
const found = new Map<number, RunRow>();
if (bound.length) {
for (const { ordinal, ...row } of await readRunRows(db, bound.map(entry => entry.input))) found.set(bound[ordinal - 1]!.index, row);
}
const runs = [...new Map([...found.values()].filter(row => row.activeIdentityContextId).map(row => [row.id, row])).values()];
const checked = await chatRunSourceMessageIds(db, runs, { allowLegacyWakeComment: true, provider: "github" });
const origins = new Map(runs.map((run, index) => [run.id, checked[index]!]));
return inputs.map((input, index) => {
const row = found.get(index);
const origin = row ? origins.get(row.id) : undefined;
if (origin && "forbidden" in origin) return { forbidden: origin.forbidden };
return ownership(input, row, origin?.messageId ?? null);
});
}