Files
PaperClipAI/scripts/benchmark-github-response-run-state.ts
T
DottaandClaude Opus 5.5 6b6c94a4ea perf(chat): read the GitHub response run by issue run indexes (#21577)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel keeps one response comment for each GitHub
request. Two background sweeps (acknowledgements and terminal responses)
and the acknowledgement step read the latest run of the assigned agent
for the request's issue with `readGitHubResponseRun`
> - That function sends two statements. Both match runs with
`coalesce(native_issue_id::text, context_snapshot->>'issueId') = <issue
id>`. No index has that expression, so each statement reads every run of
the agent and decompresses the large `context_snapshot` of each legacy
run
> - On a hosted test database, these two statements were the two most
expensive statements by database time. In a 701 s window, each ran 6.7
times a second, for 203 s and 137 s of database time (about 30 ms and 20
ms for each call). Together they kept about half of one database session
busy
> - This pull request reads the latest run with two index lookups
(native binding, and legacy context without a native binding), the same
way #20671 changed the terminal-response sweep. It decides request
ownership on that one run in the same statement
> - The benefit is that each call reads a few index entries in one
statement instead of the agent's whole run history in two statements.
The results do not change, and no migration is necessary

## Linked Issues or Issue Description

Refs #20671 (the same rewrite for the terminal-response discovery page;
its Risks section names this function as the remaining case).

**What existing behavior does this improve?**
`readGitHubResponseRun` in
`server/src/services/chat-channels/github/response-run-state.ts`. The
GitHub acknowledgement sweep calls it once for each candidate request.
The terminal-response sweep and the acknowledgement step call it through
`githubResponseExecution`.

**Subsystem affected**
GitHub chat channel: request acknowledgement and terminal-response
recovery.

**Current behavior**
The function sends two statements:

1. The latest run of the agent for the issue that woke for the request:
`heartbeat_runs` joined to the inbound `chat_message_links` row of the
delivery on `wakeCommentId` or `wakeCommentIds`, ordered by `created_at
desc`, limit 1.
2. The latest run of the agent for the issue, with the
`responseRunColumns` projection, ordered by `created_at desc, id desc`,
limit 1.

Both statements filter on `coalesce(native_issue_id::text,
context_snapshot->>'issueId') = $issueId`. The planner can only use
`heartbeat_runs_company_agent_started_idx` or
`heartbeat_runs_company_created_at_desc_idx`. It reads all runs of the
agent and evaluates the `coalesce` on each row, which decompresses the
context of each legacy run. On the hosted test database,
`heartbeat_runs` has about 31,000 rows and 1.96 GB, because
`context_snapshot` is large.

**Proposed behavior**
One statement:

- The latest run ID comes from two disjoint branches, each `order by
created_at desc, id desc limit 1`: runs with `native_issue_id =
$issueId` (through
`heartbeat_runs_company_native_issue_contract_id_uq`), and runs with
`native_issue_id is null and context_snapshot->>'issueId' = $issueId`
(through `heartbeat_runs_company_ctx_issue_created_idx`). Together the
branches are exactly the rows where the old `coalesce` equals the issue
ID.
- The statement reads that one run by primary key with the same
`responseRunColumns` projection.
- An `exists` subquery on the inbound `chat_message_links` row of the
delivery decides whether that run woke for the request. It uses the same
`wakeCommentId` / `wakeCommentIds` match as before, but on one run only.
- When the issue ID is null, the function returns no run without a
query. The old statements could not match a null issue ID either.

**Reason and benefit**
These were the two most expensive statements on the hosted test
database. On a local fixture with the hosted shape, the median call goes
from 111.6 ms to 1.03 ms (about 108 times faster), and the buffers for
one call go from about 73,000 to 98.

**Breaking changes**
None. The function signature, the returned run, and the ownership result
do not change.

## What Changed

- `readGitHubResponseRun` reads the latest run of the agent for the
issue through the two index-backed branches, and decides request
ownership on that run in the same statement. Two statements become one.
- Add `response-run-state.test.ts`. It runs the old two-statement
function (kept in the test as the oracle) and the new function on the
same inputs, and it requires equal results. It also requires a plan
bound: one call must read fewer than 150 `heartbeat_runs` rows while the
agent has 1,500 unrelated runs.
- Add `scripts/benchmark-github-response-run-state.ts`. It loads the
baseline module from a given commit, compares the complete output of
both versions for every input, measures both, and prints the plans.

## Verification

Why the old statements were slow, and the thesis: each statement filters
the agent's whole run history with an expression that no index has, and
detoasts the context of every legacy run. If each precedence branch uses
its own issue index and ownership is checked on the one latest run, the
cost no longer depends on the agent's run count, and the results stay
identical.

Red/green regression test:

- `cd server && npx vitest run
src/services/chat-channels/github/response-run-state.test.ts`
- Red (test file on the old `response-run-state.ts`): the plan bound
fails with `expected 3597 to be less than 150`. The equivalence test
passes, because the oracle is the old code.
- Green (this change): 2 of 2 pass.

The equivalence fixture has two companies, two agents each, 60 issues
each, and one or two GitHub requests for each issue. Runs cover these
cases: native binding, legacy context without a native binding, a native
binding to another issue while the legacy context names this issue,
another agent, another company, `wakeCommentId`, `wakeCommentIds` as an
array, `wakeCommentIds` as a JSON string, a link to the outbound message
of the delivery, a link to another company's comment, a link to the
sibling request of the same issue, a request with no inbound comment,
ties on `created_at`, and a null issue ID. It also calls with another
company's endpoint. The test requires more than 20 results in each
outcome (owned, latest run not owned, no run) and both binding kinds.

Local benchmark with the hosted shape:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-response-run-state.ts --baseline-ref=3c3e6c2d75
```

Fixture (synthetic): one company, 4 agents with about 7,750 runs each
(31,000 runs), 2,000 issues, 300 GitHub requests. Each run has about 60
KiB of incompressible context, and `heartbeat_runs` is 1,932 MB. Runs
are native or legacy at random, and about half link to the request
comment. Embedded PostgreSQL 18.1 on an Apple laptop. The hosted
database is Neon, so absolute times are local measurements only. The
benchmark compares the complete output for all 600 inputs (262 owned,
329 latest run not owned, 9 no run): equal. It then times 20 inputs with
runs (two agents), with 2 warmups and 7 alternating samples.

| | Statements per call | Median ms per call (range) | Buffers for one
call |
| --- | --- | --- | --- |
| Before | 2 | 111.6 (100.9–240.5) | 18,358 hit + 26,498 read, then
14,997 hit + 13,489 read |
| After | 1 | 1.03 (0.81–1.39) | 80 hit + 18 read |

Plans for one call. Before: statement 1 is a `Bitmap Heap Scan` on
`heartbeat_runs_company_agent_started_idx` that removes 7,674 rows by
filter (75.7 ms). Statement 2 is an `Index Scan` on
`heartbeat_runs_company_created_at_desc_idx` that removes 19,801 rows by
filter (60.2 ms). After: one `Index Scan` on `heartbeat_runs_pkey`. Its
`InitPlan` is a `Merge Append` of two `Limit` index scans on
`heartbeat_runs_company_native_issue_contract_id_uq` and
`heartbeat_runs_company_ctx_issue_created_idx`, and its `SubPlan` is one
`Index Scan` on `chat_message_links_delivery_idx` (0.128 ms).

Other checks:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed
(all GitHub chat-channel integration tests, including the
acknowledgement and terminal-response sweeps).
- `cd server && npx tsc --noEmit`: passed.
- `pnpm verify` exited 2 because a file under `scripts/` changed, which
requires the broad CI coverage. Required CI covers the rest.

Why the function runs 6.7 times a second. The acknowledgement sweep and
the terminal-response sweep rebuild their candidate set when a hint
arrives, and they call this function once for each candidate on each
rebuild. A run milestone for a run with no issue (for example, every
status change of a run without a task in `setRunStatus`, or a comment
merged into an active run in the wake queue) sends a company-wide hint
to both sweeps. So in a busy company, both sweeps rebuild about once a
second (their minimum pass interval), and each rebuild reads every
parked candidate again, although nothing about those candidates changed.
In a local reproduction with 8 parked candidates (4 for each sweep) and
the real coordinator and workers, 10 company-wide milestone hints a
second gave 8.75 calls a second; 1 hint a second gave 7.98 calls a
second; 10 hints a second for an unrelated issue gave 0 calls; and no
hints gave 0 calls. So the hosted rate is consistent with about 7 parked
GitHub requests in one company and a steady stream of company-wide run
milestones. This pull request makes each call cheap. A separate pull
request will address the repeated rebuilds.

## Risks

- Low risk. The change is one function. The run match is the same set of
runs: `native_issue_id::text = <uuid text>` is equal to `native_issue_id
= <uuid>` for the canonical UUID text that callers pass (the issue IDs
come from UUID columns), and a null `native_issue_id` makes the old
`coalesce` use the legacy context.
- Ownership: the old code compared the latest run that woke for the
request (ordered by `created_at` only) with the latest run (ordered by
`created_at, id`). That is true exactly when the latest run woke for the
request, except when two runs that both woke for the request tie on
`created_at` with the latest run. In that case the old result depended
on the physical row order. The new code always returns the deterministic
answer (the latest run woke for the request).
- The number of calls does not change in this pull request.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 00:59:25 -06:00

173 lines
12 KiB
TypeScript

import assert from "node:assert/strict";
import { execFileSync } from "node:child_process";
import { promises as fs } from "node:fs";
import path from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import { createDb, startEmbeddedPostgresTestDatabase, type Db } from "../packages/db/src/index.js";
import { readGitHubResponseRun } from "../server/src/services/chat-channels/github/response-run-state.js";
const baselineRef = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15);
assert.ok(baselineRef, "Pass an explicit pre-fix --baseline-ref=<commit>");
const number = (name: string, fallback: number) => Number(process.env[name] ?? fallback);
const fixture = {
agents: number("BENCH_AGENTS", 4),
runsPerAgent: number("BENCH_RUNS_PER_AGENT", 7_750),
issues: number("BENCH_ISSUES", 2_000),
requests: number("BENCH_REQUESTS", 300),
contextPadBytes: number("BENCH_CONTEXT_PAD_BYTES", 61_440),
};
const iterations = number("BENCH_ITERATIONS", 7);
const sampleInputs = number("BENCH_SAMPLE_INPUTS", 20);
const baselineRevision = execFileSync("git", ["rev-parse", "--verify", `${baselineRef}^{commit}`], { encoding: "utf8" }).trim();
const treatmentRevision = execFileSync("git", ["rev-parse", "HEAD"], { encoding: "utf8" }).trim();
const modulePath = "server/src/services/chat-channels/github/response-run-state.ts";
const root = fileURLToPath(new URL("../", import.meta.url));
const keepAlive = setInterval(() => {}, 1_000);
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: Db | undefined;
let baselineDir: string | undefined;
type Input = Parameters<typeof readGitHubResponseRun>[1];
async function seed(db: Db) {
const sql = db.$client;
const [{ company }] = await sql.unsafe(`insert into companies (name, issue_prefix) values ('Bench', 'BN') returning id as company`);
const agentRows = await sql.unsafe(`insert into agents (company_id, name) select $1, 'Agent ' || n from generate_series(1, ${fixture.agents}) n
returning id`, [company]);
const agentIds = agentRows.map(row => row.id as string);
const [{ application }] = await sql.unsafe(`insert into tool_applications (company_id, application_key, name, type)
values ($1, 'github', 'GitHub', 'chat') returning id as application`, [company]);
const [{ connection }] = await sql.unsafe(`insert into tool_connections (company_id, application_id, name, uid, transport, connection_purpose)
values ($1, $2, 'GitHub', 'github-bench', 'chat_sdk', 'channel') returning id as connection`, [company, application]);
const [{ endpoint }] = await sql.unsafe(`insert into chat_endpoints (company_id, connection_id, provider, public_id, assigned_agent_id, status)
values ($1, $2, 'github', 'bench', $3, 'active') returning id as endpoint`, [company, connection, agentIds[0]]);
await sql.unsafe(`create unlogged table bench_issue (n int primary key, id uuid)`);
await sql.unsafe(`with created as (insert into issues (company_id, title, status)
select $1, 'Issue ' || n, 'in_progress' from generate_series(1, ${fixture.issues}) n returning id, title)
insert into bench_issue select (regexp_replace(title, '\\D', '', 'g'))::int, id from created`, [company]);
await sql.unsafe(`insert into chat_conversations (company_id, endpoint_id, issue_id, external_conversation_id, external_label, external_thread_id)
select $1, $2, i.id, 'github:paperclipai/paperclip:' || i.n, 'PR #' || i.n, 'github:paperclipai/paperclip:' || i.n
from bench_issue i where i.n <= ${fixture.requests}`, [company, endpoint]);
await sql.unsafe(`insert into chat_deliveries (company_id, endpoint_id, provider_event_id, deduplication_key, event_kind,
normalized_event, conversation_id, state)
select $1, $2, 'event-' || i.n, 'dedupe-' || i.n, 'message', '{}', c.id, 'processed'
from bench_issue i join chat_conversations c on c.issue_id = i.id`, [company, endpoint]);
await sql.unsafe(`insert into issue_comments (company_id, issue_id, body)
select $1, i.id, 'Request ' || i.n from bench_issue i where i.n <= ${fixture.requests}`, [company]);
await sql.unsafe(`insert into chat_message_links (company_id, endpoint_id, conversation_id, delivery_id, comment_id, provider_message_id, direction)
select $1, $2, c.id, d.id, ic.id, 'IC_in_' || i.n, 'inbound'
from bench_issue i join chat_conversations c on c.issue_id = i.id join chat_deliveries d on d.conversation_id = c.id
join issue_comments ic on ic.issue_id = i.id`, [company, endpoint]);
await sql.unsafe(`create unlogged table bench_agent (n int primary key, id uuid)`);
for (const [index, id] of agentIds.entries()) await sql.unsafe(`insert into bench_agent values ($1, $2)`, [index, id]);
for (let batch = 0; batch < fixture.agents * fixture.runsPerAgent; batch += 1_000) {
await sql.unsafe(`insert into heartbeat_runs (company_id, agent_id, status, scope_kind, issue_id, native_issue_id, context_snapshot, created_at)
select $1, a.id, case when r % 9 = 0 then 'running' else 'succeeded' end,
case when abs(hashint4(r + 31)::bigint) % 2 = 0 then 'issue' else 'company' end,
case when abs(hashint4(r + 31)::bigint) % 2 = 0 then i.id end, case when abs(hashint4(r + 31)::bigint) % 2 = 0 then i.id end,
jsonb_build_object('issueId', i.id::text,
'wakeCommentId', (select ic.id::text from issue_comments ic where ic.issue_id = i.id and r % 3 = 0),
'wakeCommentIds', case when r % 5 = 0 then (select jsonb_agg(ic.id::text) from issue_comments ic where ic.issue_id = i.id) end,
'pad', (select string_agg(md5(random()::text || g::text || r::text), '') from generate_series(1, ${Math.ceil(fixture.contextPadBytes / 32)}) g)),
timestamptz '2026-01-01' + r * interval '1 minute'
from generate_series($2::int, $3::int) r
join bench_agent a on a.n = abs(hashint4(r + 17)::bigint) % ${fixture.agents}
join bench_issue i on i.n = 1 + abs(hashint4(r)::bigint) % ${fixture.issues}`,
[company, batch, Math.min(batch + 999, fixture.agents * fixture.runsPerAgent - 1)]);
}
const requests = await sql.unsafe(`select i.id as issue, d.id as delivery, ic.id as comment from bench_issue i
join chat_conversations c on c.issue_id = i.id join chat_deliveries d on d.conversation_id = c.id
join issue_comments ic on ic.issue_id = i.id order by i.n`);
await sql.unsafe("drop table bench_issue; drop table bench_agent");
const inputs: Input[] = requests.flatMap(row => agentIds.slice(0, 2).map(agentId => ({ companyId: company as string,
endpointId: endpoint as string, agentId, issueId: row.issue as string, deliveryId: row.delivery as string,
sourceMessageId: row.comment as string })));
return { company, inputs };
}
try {
const oldSource = execFileSync("git", ["show", `${baselineRevision}:${modulePath}`], { encoding: "utf8" });
const moduleUrl = new URL(`../${modulePath}`, import.meta.url);
baselineDir = await fs.mkdtemp(path.join(root, "server/scripts/.response-run-state-baseline-"));
const baselineFile = path.join(baselineDir, "response-run-state.ts");
await fs.writeFile(baselineFile, oldSource.replace(/from "(\.{1,2}\/[^\"]+)"/g,
(_match, specifier: string) => `from ${JSON.stringify(fileURLToPath(new URL(specifier, moduleUrl)))}`));
const baseline = await import(pathToFileURL(baselineFile).href) as typeof import("../server/src/services/chat-channels/github/response-run-state.js");
temporary = await startEmbeddedPostgresTestDatabase("github-response-run-state-benchmark-");
db = createDb(temporary.connectionString);
const seedStart = performance.now();
const { company, inputs } = await seed(db);
await db.$client.unsafe("ANALYZE");
const seedMs = performance.now() - seedStart;
const before = [], after = [];
for (const input of inputs) {
before.push(await baseline.readGitHubResponseRun(db, input));
after.push(await readGitHubResponseRun(db, input));
}
assert.deepEqual(after, before, "Every input must return the same latest run and ownership");
const outcomes = { owns: before.filter(result => result.ownsLatestRequest).length,
latestNotOwned: before.filter(result => result.latestRun && !result.ownsLatestRequest).length,
noRun: before.filter(result => !result.latestRun).length };
const withRuns = inputs.filter((_, index) => before[index]!.latestRun);
const sample = withRuns.filter((_, index) => index % Math.max(1, Math.floor(withRuns.length / sampleInputs)) === 0).slice(0, sampleInputs);
const sampledAgents = new Set(sample.map(input => input.agentId)).size;
const versions = { baseline: baseline.readGitHubResponseRun, treatment: readGitHubResponseRun };
async function timed(version: keyof typeof versions) {
const start = performance.now();
for (const input of sample) await versions[version](db!, input);
return (performance.now() - start) / sample.length;
}
const samples = { baseline: [] as number[], treatment: [] as number[] };
for (let n = 0; n < iterations + 2; n++) {
const order = n % 2 ? ["treatment", "baseline"] as const : ["baseline", "treatment"] as const;
for (const version of order) {
const elapsed = await timed(version);
if (n >= 2) samples[version].push(elapsed);
}
}
const stats = (values: number[]) => {
const sorted = [...values].sort((a, b) => a - b);
return { perCallMs: values.map(value => Math.round(value * 100) / 100), medianMs: sorted[Math.floor(sorted.length / 2)],
minMs: sorted[0], maxMs: sorted.at(-1) };
};
const captured: Record<keyof typeof versions, Array<{ sql: string; params: unknown[] }>> = { baseline: [], treatment: [] };
for (const version of Object.keys(versions) as Array<keyof typeof versions>) {
const proxy = new Proxy(db, { get(target, key) {
if (key !== "select") return Reflect.get(target, key);
return (...fields: Parameters<Db["select"]>) => {
const builder = target.select(...fields), from = builder.from.bind(builder);
builder.from = ((...args: Parameters<typeof from>) => {
const selection = from(...args), execute = selection.execute.bind(selection);
selection.execute = (async () => { captured[version].push(selection.toSQL()); return execute(); }) as typeof selection.execute;
return selection;
}) as typeof builder.from;
return builder;
};
} });
await versions[version](proxy as Db, sample[0]!);
}
async function plan(query: { sql: string; params: unknown[] }) {
return (await db!.$client.unsafe(`EXPLAIN (ANALYZE, BUFFERS) ${query.sql}`, query.params as never[]).values()).map(row => row[0]);
}
const counts = (await db.$client.unsafe(`select (select count(*) from heartbeat_runs)::int runs,
pg_size_pretty(pg_total_relation_size('heartbeat_runs')) runs_size,
(select count(*) from heartbeat_runs where agent_id = $1)::int runs_of_sampled_agent`, [sample[0]!.agentId]))[0];
console.log(JSON.stringify({ baselineRevision, treatmentRevision,
postgresVersion: (await db.$client.unsafe("SELECT version()"))[0]!.version,
fixture: { ...fixture, ...counts, seedMs: Math.round(seedMs), company, synthetic: true },
correctness: { completeOutputEqual: true, inputs: inputs.length, outcomes },
timings: { warmups: 2, samples: iterations, inputsPerSample: sample.length, sampledAgents, order: "alternating", unit: "ms per readGitHubResponseRun call",
baseline: stats(samples.baseline), treatment: stats(samples.treatment) },
statementsPerCall: { baseline: captured.baseline.length, treatment: captured.treatment.length },
plans: { baseline: await Promise.all(captured.baseline.map(plan)), treatment: await Promise.all(captured.treatment.map(plan)) },
}, null, 2));
} finally {
try {
await db?.$client.end({ timeout: 0 });
await temporary?.cleanup();
if (baselineDir) await fs.rm(baselineDir, { recursive: true, force: true });
} finally { clearInterval(keepAlive); }
}