mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
perf(chat): read the GitHub response run by issue run indexes (#21577)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel keeps one response comment for each GitHub request. Two background sweeps (acknowledgements and terminal responses) and the acknowledgement step read the latest run of the assigned agent for the request's issue with `readGitHubResponseRun` > - That function sends two statements. Both match runs with `coalesce(native_issue_id::text, context_snapshot->>'issueId') = <issue id>`. No index has that expression, so each statement reads every run of the agent and decompresses the large `context_snapshot` of each legacy run > - On a hosted test database, these two statements were the two most expensive statements by database time. In a 701 s window, each ran 6.7 times a second, for 203 s and 137 s of database time (about 30 ms and 20 ms for each call). Together they kept about half of one database session busy > - This pull request reads the latest run with two index lookups (native binding, and legacy context without a native binding), the same way #20671 changed the terminal-response sweep. It decides request ownership on that one run in the same statement > - The benefit is that each call reads a few index entries in one statement instead of the agent's whole run history in two statements. The results do not change, and no migration is necessary ## Linked Issues or Issue Description Refs #20671 (the same rewrite for the terminal-response discovery page; its Risks section names this function as the remaining case). **What existing behavior does this improve?** `readGitHubResponseRun` in `server/src/services/chat-channels/github/response-run-state.ts`. The GitHub acknowledgement sweep calls it once for each candidate request. The terminal-response sweep and the acknowledgement step call it through `githubResponseExecution`. **Subsystem affected** GitHub chat channel: request acknowledgement and terminal-response recovery. **Current behavior** The function sends two statements: 1. The latest run of the agent for the issue that woke for the request: `heartbeat_runs` joined to the inbound `chat_message_links` row of the delivery on `wakeCommentId` or `wakeCommentIds`, ordered by `created_at desc`, limit 1. 2. The latest run of the agent for the issue, with the `responseRunColumns` projection, ordered by `created_at desc, id desc`, limit 1. Both statements filter on `coalesce(native_issue_id::text, context_snapshot->>'issueId') = $issueId`. The planner can only use `heartbeat_runs_company_agent_started_idx` or `heartbeat_runs_company_created_at_desc_idx`. It reads all runs of the agent and evaluates the `coalesce` on each row, which decompresses the context of each legacy run. On the hosted test database, `heartbeat_runs` has about 31,000 rows and 1.96 GB, because `context_snapshot` is large. **Proposed behavior** One statement: - The latest run ID comes from two disjoint branches, each `order by created_at desc, id desc limit 1`: runs with `native_issue_id = $issueId` (through `heartbeat_runs_company_native_issue_contract_id_uq`), and runs with `native_issue_id is null and context_snapshot->>'issueId' = $issueId` (through `heartbeat_runs_company_ctx_issue_created_idx`). Together the branches are exactly the rows where the old `coalesce` equals the issue ID. - The statement reads that one run by primary key with the same `responseRunColumns` projection. - An `exists` subquery on the inbound `chat_message_links` row of the delivery decides whether that run woke for the request. It uses the same `wakeCommentId` / `wakeCommentIds` match as before, but on one run only. - When the issue ID is null, the function returns no run without a query. The old statements could not match a null issue ID either. **Reason and benefit** These were the two most expensive statements on the hosted test database. On a local fixture with the hosted shape, the median call goes from 111.6 ms to 1.03 ms (about 108 times faster), and the buffers for one call go from about 73,000 to 98. **Breaking changes** None. The function signature, the returned run, and the ownership result do not change. ## What Changed - `readGitHubResponseRun` reads the latest run of the agent for the issue through the two index-backed branches, and decides request ownership on that run in the same statement. Two statements become one. - Add `response-run-state.test.ts`. It runs the old two-statement function (kept in the test as the oracle) and the new function on the same inputs, and it requires equal results. It also requires a plan bound: one call must read fewer than 150 `heartbeat_runs` rows while the agent has 1,500 unrelated runs. - Add `scripts/benchmark-github-response-run-state.ts`. It loads the baseline module from a given commit, compares the complete output of both versions for every input, measures both, and prints the plans. ## Verification Why the old statements were slow, and the thesis: each statement filters the agent's whole run history with an expression that no index has, and detoasts the context of every legacy run. If each precedence branch uses its own issue index and ownership is checked on the one latest run, the cost no longer depends on the agent's run count, and the results stay identical. Red/green regression test: - `cd server && npx vitest run src/services/chat-channels/github/response-run-state.test.ts` - Red (test file on the old `response-run-state.ts`): the plan bound fails with `expected 3597 to be less than 150`. The equivalence test passes, because the oracle is the old code. - Green (this change): 2 of 2 pass. The equivalence fixture has two companies, two agents each, 60 issues each, and one or two GitHub requests for each issue. Runs cover these cases: native binding, legacy context without a native binding, a native binding to another issue while the legacy context names this issue, another agent, another company, `wakeCommentId`, `wakeCommentIds` as an array, `wakeCommentIds` as a JSON string, a link to the outbound message of the delivery, a link to another company's comment, a link to the sibling request of the same issue, a request with no inbound comment, ties on `created_at`, and a null issue ID. It also calls with another company's endpoint. The test requires more than 20 results in each outcome (owned, latest run not owned, no run) and both binding kinds. Local benchmark with the hosted shape: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-response-run-state.ts --baseline-ref=3c3e6c2d75 ``` Fixture (synthetic): one company, 4 agents with about 7,750 runs each (31,000 runs), 2,000 issues, 300 GitHub requests. Each run has about 60 KiB of incompressible context, and `heartbeat_runs` is 1,932 MB. Runs are native or legacy at random, and about half link to the request comment. Embedded PostgreSQL 18.1 on an Apple laptop. The hosted database is Neon, so absolute times are local measurements only. The benchmark compares the complete output for all 600 inputs (262 owned, 329 latest run not owned, 9 no run): equal. It then times 20 inputs with runs (two agents), with 2 warmups and 7 alternating samples. | | Statements per call | Median ms per call (range) | Buffers for one call | | --- | --- | --- | --- | | Before | 2 | 111.6 (100.9–240.5) | 18,358 hit + 26,498 read, then 14,997 hit + 13,489 read | | After | 1 | 1.03 (0.81–1.39) | 80 hit + 18 read | Plans for one call. Before: statement 1 is a `Bitmap Heap Scan` on `heartbeat_runs_company_agent_started_idx` that removes 7,674 rows by filter (75.7 ms). Statement 2 is an `Index Scan` on `heartbeat_runs_company_created_at_desc_idx` that removes 19,801 rows by filter (60.2 ms). After: one `Index Scan` on `heartbeat_runs_pkey`. Its `InitPlan` is a `Merge Append` of two `Limit` index scans on `heartbeat_runs_company_native_issue_contract_id_uq` and `heartbeat_runs_company_ctx_issue_created_idx`, and its `SubPlan` is one `Index Scan` on `chat_message_links_delivery_idx` (0.128 ms). Other checks: - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t ithub`: 104 passed (all GitHub chat-channel integration tests, including the acknowledgement and terminal-response sweeps). - `cd server && npx tsc --noEmit`: passed. - `pnpm verify` exited 2 because a file under `scripts/` changed, which requires the broad CI coverage. Required CI covers the rest. Why the function runs 6.7 times a second. The acknowledgement sweep and the terminal-response sweep rebuild their candidate set when a hint arrives, and they call this function once for each candidate on each rebuild. A run milestone for a run with no issue (for example, every status change of a run without a task in `setRunStatus`, or a comment merged into an active run in the wake queue) sends a company-wide hint to both sweeps. So in a busy company, both sweeps rebuild about once a second (their minimum pass interval), and each rebuild reads every parked candidate again, although nothing about those candidates changed. In a local reproduction with 8 parked candidates (4 for each sweep) and the real coordinator and workers, 10 company-wide milestone hints a second gave 8.75 calls a second; 1 hint a second gave 7.98 calls a second; 10 hints a second for an unrelated issue gave 0 calls; and no hints gave 0 calls. So the hosted rate is consistent with about 7 parked GitHub requests in one company and a steady stream of company-wide run milestones. This pull request makes each call cheap. A separate pull request will address the repeated rebuilds. ## Risks - Low risk. The change is one function. The run match is the same set of runs: `native_issue_id::text = <uuid text>` is equal to `native_issue_id = <uuid>` for the canonical UUID text that callers pass (the issue IDs come from UUID columns), and a null `native_issue_id` makes the old `coalesce` use the legacy context. - Ownership: the old code compared the latest run that woke for the request (ordered by `created_at` only) with the latest run (ordered by `created_at, id`). That is true exactly when the latest run woke for the request, except when two runs that both woke for the request tie on `created_at` with the latest run. In that case the old result depended on the physical row order. The new code always returns the deterministic answer (the latest run woke for the request). - The number of calls does not change in this pull request. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude Code with tool use (shell, file edits, and local PostgreSQL benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e977c3b922
commit
6b6c94a4ea
3 files changed
+421
-16
No files matched your search
@@ -0,0 +1,172 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { createDb, startEmbeddedPostgresTestDatabase, type Db } from "../packages/db/src/index.js";
|
||||
import { readGitHubResponseRun } from "../server/src/services/chat-channels/github/response-run-state.js";
|
||||
|
||||
const baselineRef = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15);
|
||||
assert.ok(baselineRef, "Pass an explicit pre-fix --baseline-ref=<commit>");
|
||||
const number = (name: string, fallback: number) => Number(process.env[name] ?? fallback);
|
||||
const fixture = {
|
||||
agents: number("BENCH_AGENTS", 4),
|
||||
runsPerAgent: number("BENCH_RUNS_PER_AGENT", 7_750),
|
||||
issues: number("BENCH_ISSUES", 2_000),
|
||||
requests: number("BENCH_REQUESTS", 300),
|
||||
contextPadBytes: number("BENCH_CONTEXT_PAD_BYTES", 61_440),
|
||||
};
|
||||
const iterations = number("BENCH_ITERATIONS", 7);
|
||||
const sampleInputs = number("BENCH_SAMPLE_INPUTS", 20);
|
||||
const baselineRevision = execFileSync("git", ["rev-parse", "--verify", `${baselineRef}^{commit}`], { encoding: "utf8" }).trim();
|
||||
const treatmentRevision = execFileSync("git", ["rev-parse", "HEAD"], { encoding: "utf8" }).trim();
|
||||
const modulePath = "server/src/services/chat-channels/github/response-run-state.ts";
|
||||
const root = fileURLToPath(new URL("../", import.meta.url));
|
||||
const keepAlive = setInterval(() => {}, 1_000);
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: Db | undefined;
|
||||
let baselineDir: string | undefined;
|
||||
type Input = Parameters<typeof readGitHubResponseRun>[1];
|
||||
|
||||
async function seed(db: Db) {
|
||||
const sql = db.$client;
|
||||
const [{ company }] = await sql.unsafe(`insert into companies (name, issue_prefix) values ('Bench', 'BN') returning id as company`);
|
||||
const agentRows = await sql.unsafe(`insert into agents (company_id, name) select $1, 'Agent ' || n from generate_series(1, ${fixture.agents}) n
|
||||
returning id`, [company]);
|
||||
const agentIds = agentRows.map(row => row.id as string);
|
||||
const [{ application }] = await sql.unsafe(`insert into tool_applications (company_id, application_key, name, type)
|
||||
values ($1, 'github', 'GitHub', 'chat') returning id as application`, [company]);
|
||||
const [{ connection }] = await sql.unsafe(`insert into tool_connections (company_id, application_id, name, uid, transport, connection_purpose)
|
||||
values ($1, $2, 'GitHub', 'github-bench', 'chat_sdk', 'channel') returning id as connection`, [company, application]);
|
||||
const [{ endpoint }] = await sql.unsafe(`insert into chat_endpoints (company_id, connection_id, provider, public_id, assigned_agent_id, status)
|
||||
values ($1, $2, 'github', 'bench', $3, 'active') returning id as endpoint`, [company, connection, agentIds[0]]);
|
||||
await sql.unsafe(`create unlogged table bench_issue (n int primary key, id uuid)`);
|
||||
await sql.unsafe(`with created as (insert into issues (company_id, title, status)
|
||||
select $1, 'Issue ' || n, 'in_progress' from generate_series(1, ${fixture.issues}) n returning id, title)
|
||||
insert into bench_issue select (regexp_replace(title, '\\D', '', 'g'))::int, id from created`, [company]);
|
||||
await sql.unsafe(`insert into chat_conversations (company_id, endpoint_id, issue_id, external_conversation_id, external_label, external_thread_id)
|
||||
select $1, $2, i.id, 'github:paperclipai/paperclip:' || i.n, 'PR #' || i.n, 'github:paperclipai/paperclip:' || i.n
|
||||
from bench_issue i where i.n <= ${fixture.requests}`, [company, endpoint]);
|
||||
await sql.unsafe(`insert into chat_deliveries (company_id, endpoint_id, provider_event_id, deduplication_key, event_kind,
|
||||
normalized_event, conversation_id, state)
|
||||
select $1, $2, 'event-' || i.n, 'dedupe-' || i.n, 'message', '{}', c.id, 'processed'
|
||||
from bench_issue i join chat_conversations c on c.issue_id = i.id`, [company, endpoint]);
|
||||
await sql.unsafe(`insert into issue_comments (company_id, issue_id, body)
|
||||
select $1, i.id, 'Request ' || i.n from bench_issue i where i.n <= ${fixture.requests}`, [company]);
|
||||
await sql.unsafe(`insert into chat_message_links (company_id, endpoint_id, conversation_id, delivery_id, comment_id, provider_message_id, direction)
|
||||
select $1, $2, c.id, d.id, ic.id, 'IC_in_' || i.n, 'inbound'
|
||||
from bench_issue i join chat_conversations c on c.issue_id = i.id join chat_deliveries d on d.conversation_id = c.id
|
||||
join issue_comments ic on ic.issue_id = i.id`, [company, endpoint]);
|
||||
await sql.unsafe(`create unlogged table bench_agent (n int primary key, id uuid)`);
|
||||
for (const [index, id] of agentIds.entries()) await sql.unsafe(`insert into bench_agent values ($1, $2)`, [index, id]);
|
||||
for (let batch = 0; batch < fixture.agents * fixture.runsPerAgent; batch += 1_000) {
|
||||
await sql.unsafe(`insert into heartbeat_runs (company_id, agent_id, status, scope_kind, issue_id, native_issue_id, context_snapshot, created_at)
|
||||
select $1, a.id, case when r % 9 = 0 then 'running' else 'succeeded' end,
|
||||
case when abs(hashint4(r + 31)::bigint) % 2 = 0 then 'issue' else 'company' end,
|
||||
case when abs(hashint4(r + 31)::bigint) % 2 = 0 then i.id end, case when abs(hashint4(r + 31)::bigint) % 2 = 0 then i.id end,
|
||||
jsonb_build_object('issueId', i.id::text,
|
||||
'wakeCommentId', (select ic.id::text from issue_comments ic where ic.issue_id = i.id and r % 3 = 0),
|
||||
'wakeCommentIds', case when r % 5 = 0 then (select jsonb_agg(ic.id::text) from issue_comments ic where ic.issue_id = i.id) end,
|
||||
'pad', (select string_agg(md5(random()::text || g::text || r::text), '') from generate_series(1, ${Math.ceil(fixture.contextPadBytes / 32)}) g)),
|
||||
timestamptz '2026-01-01' + r * interval '1 minute'
|
||||
from generate_series($2::int, $3::int) r
|
||||
join bench_agent a on a.n = abs(hashint4(r + 17)::bigint) % ${fixture.agents}
|
||||
join bench_issue i on i.n = 1 + abs(hashint4(r)::bigint) % ${fixture.issues}`,
|
||||
[company, batch, Math.min(batch + 999, fixture.agents * fixture.runsPerAgent - 1)]);
|
||||
}
|
||||
const requests = await sql.unsafe(`select i.id as issue, d.id as delivery, ic.id as comment from bench_issue i
|
||||
join chat_conversations c on c.issue_id = i.id join chat_deliveries d on d.conversation_id = c.id
|
||||
join issue_comments ic on ic.issue_id = i.id order by i.n`);
|
||||
await sql.unsafe("drop table bench_issue; drop table bench_agent");
|
||||
const inputs: Input[] = requests.flatMap(row => agentIds.slice(0, 2).map(agentId => ({ companyId: company as string,
|
||||
endpointId: endpoint as string, agentId, issueId: row.issue as string, deliveryId: row.delivery as string,
|
||||
sourceMessageId: row.comment as string })));
|
||||
return { company, inputs };
|
||||
}
|
||||
|
||||
try {
|
||||
const oldSource = execFileSync("git", ["show", `${baselineRevision}:${modulePath}`], { encoding: "utf8" });
|
||||
const moduleUrl = new URL(`../${modulePath}`, import.meta.url);
|
||||
baselineDir = await fs.mkdtemp(path.join(root, "server/scripts/.response-run-state-baseline-"));
|
||||
const baselineFile = path.join(baselineDir, "response-run-state.ts");
|
||||
await fs.writeFile(baselineFile, oldSource.replace(/from "(\.{1,2}\/[^\"]+)"/g,
|
||||
(_match, specifier: string) => `from ${JSON.stringify(fileURLToPath(new URL(specifier, moduleUrl)))}`));
|
||||
const baseline = await import(pathToFileURL(baselineFile).href) as typeof import("../server/src/services/chat-channels/github/response-run-state.js");
|
||||
temporary = await startEmbeddedPostgresTestDatabase("github-response-run-state-benchmark-");
|
||||
db = createDb(temporary.connectionString);
|
||||
const seedStart = performance.now();
|
||||
const { company, inputs } = await seed(db);
|
||||
await db.$client.unsafe("ANALYZE");
|
||||
const seedMs = performance.now() - seedStart;
|
||||
|
||||
const before = [], after = [];
|
||||
for (const input of inputs) {
|
||||
before.push(await baseline.readGitHubResponseRun(db, input));
|
||||
after.push(await readGitHubResponseRun(db, input));
|
||||
}
|
||||
assert.deepEqual(after, before, "Every input must return the same latest run and ownership");
|
||||
const outcomes = { owns: before.filter(result => result.ownsLatestRequest).length,
|
||||
latestNotOwned: before.filter(result => result.latestRun && !result.ownsLatestRequest).length,
|
||||
noRun: before.filter(result => !result.latestRun).length };
|
||||
|
||||
const withRuns = inputs.filter((_, index) => before[index]!.latestRun);
|
||||
const sample = withRuns.filter((_, index) => index % Math.max(1, Math.floor(withRuns.length / sampleInputs)) === 0).slice(0, sampleInputs);
|
||||
const sampledAgents = new Set(sample.map(input => input.agentId)).size;
|
||||
const versions = { baseline: baseline.readGitHubResponseRun, treatment: readGitHubResponseRun };
|
||||
async function timed(version: keyof typeof versions) {
|
||||
const start = performance.now();
|
||||
for (const input of sample) await versions[version](db!, input);
|
||||
return (performance.now() - start) / sample.length;
|
||||
}
|
||||
const samples = { baseline: [] as number[], treatment: [] as number[] };
|
||||
for (let n = 0; n < iterations + 2; n++) {
|
||||
const order = n % 2 ? ["treatment", "baseline"] as const : ["baseline", "treatment"] as const;
|
||||
for (const version of order) {
|
||||
const elapsed = await timed(version);
|
||||
if (n >= 2) samples[version].push(elapsed);
|
||||
}
|
||||
}
|
||||
const stats = (values: number[]) => {
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
return { perCallMs: values.map(value => Math.round(value * 100) / 100), medianMs: sorted[Math.floor(sorted.length / 2)],
|
||||
minMs: sorted[0], maxMs: sorted.at(-1) };
|
||||
};
|
||||
|
||||
const captured: Record<keyof typeof versions, Array<{ sql: string; params: unknown[] }>> = { baseline: [], treatment: [] };
|
||||
for (const version of Object.keys(versions) as Array<keyof typeof versions>) {
|
||||
const proxy = new Proxy(db, { get(target, key) {
|
||||
if (key !== "select") return Reflect.get(target, key);
|
||||
return (...fields: Parameters<Db["select"]>) => {
|
||||
const builder = target.select(...fields), from = builder.from.bind(builder);
|
||||
builder.from = ((...args: Parameters<typeof from>) => {
|
||||
const selection = from(...args), execute = selection.execute.bind(selection);
|
||||
selection.execute = (async () => { captured[version].push(selection.toSQL()); return execute(); }) as typeof selection.execute;
|
||||
return selection;
|
||||
}) as typeof builder.from;
|
||||
return builder;
|
||||
};
|
||||
} });
|
||||
await versions[version](proxy as Db, sample[0]!);
|
||||
}
|
||||
async function plan(query: { sql: string; params: unknown[] }) {
|
||||
return (await db!.$client.unsafe(`EXPLAIN (ANALYZE, BUFFERS) ${query.sql}`, query.params as never[]).values()).map(row => row[0]);
|
||||
}
|
||||
const counts = (await db.$client.unsafe(`select (select count(*) from heartbeat_runs)::int runs,
|
||||
pg_size_pretty(pg_total_relation_size('heartbeat_runs')) runs_size,
|
||||
(select count(*) from heartbeat_runs where agent_id = $1)::int runs_of_sampled_agent`, [sample[0]!.agentId]))[0];
|
||||
console.log(JSON.stringify({ baselineRevision, treatmentRevision,
|
||||
postgresVersion: (await db.$client.unsafe("SELECT version()"))[0]!.version,
|
||||
fixture: { ...fixture, ...counts, seedMs: Math.round(seedMs), company, synthetic: true },
|
||||
correctness: { completeOutputEqual: true, inputs: inputs.length, outcomes },
|
||||
timings: { warmups: 2, samples: iterations, inputsPerSample: sample.length, sampledAgents, order: "alternating", unit: "ms per readGitHubResponseRun call",
|
||||
baseline: stats(samples.baseline), treatment: stats(samples.treatment) },
|
||||
statementsPerCall: { baseline: captured.baseline.length, treatment: captured.treatment.length },
|
||||
plans: { baseline: await Promise.all(captured.baseline.map(plan)), treatment: await Promise.all(captured.treatment.map(plan)) },
|
||||
}, null, 2));
|
||||
} finally {
|
||||
try {
|
||||
await db?.$client.end({ timeout: 0 });
|
||||
await temporary?.cleanup();
|
||||
if (baselineDir) await fs.rm(baselineDir, { recursive: true, force: true });
|
||||
} finally { clearInterval(keepAlive); }
|
||||
}
|
||||
@@ -0,0 +1,224 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, desc, eq, or, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { chatMessageLinks, createDb, heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js";
|
||||
import { chatRunSourceMessageId } from "../run-origin.js";
|
||||
import { readGitHubResponseRun, responseRunColumns } from "./response-run-state.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const suite = support.supported ? describe : describe.skip;
|
||||
|
||||
type Input = Parameters<typeof readGitHubResponseRun>[1];
|
||||
|
||||
async function previousReadGitHubResponseRun(db: Db, input: Input) {
|
||||
const [requestRun] = await db.select({ runId: heartbeatRuns.id }).from(heartbeatRuns).innerJoin(chatMessageLinks, and(
|
||||
eq(chatMessageLinks.companyId, input.companyId), eq(chatMessageLinks.endpointId, input.endpointId),
|
||||
eq(chatMessageLinks.deliveryId, input.deliveryId), eq(chatMessageLinks.direction, "inbound"),
|
||||
or(sql`${chatMessageLinks.commentId}::text = ${heartbeatRuns.contextSnapshot}->>'wakeCommentId'`,
|
||||
sql`coalesce(${heartbeatRuns.contextSnapshot}->'wakeCommentIds', '[]'::jsonb) ? ${chatMessageLinks.commentId}::text`),
|
||||
)).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId),
|
||||
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${input.issueId}`))
|
||||
.orderBy(desc(heartbeatRuns.createdAt)).limit(1);
|
||||
const [latestRun] = await db.select(responseRunColumns).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId),
|
||||
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${input.issueId}`,
|
||||
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
||||
const originMessageId = latestRun?.activeIdentityContextId ? await chatRunSourceMessageId(db, latestRun,
|
||||
{ allowLegacyWakeComment: true, provider: "github" }) : null;
|
||||
const ownsLatestRequest = !!latestRun && (requestRun?.runId === latestRun.id ||
|
||||
(!!input.sourceMessageId && originMessageId === input.sourceMessageId));
|
||||
return { latestRun, ownsLatestRequest };
|
||||
}
|
||||
|
||||
function random(seed: number) {
|
||||
let state = seed >>> 0;
|
||||
return () => {
|
||||
state = (state + 0x6d2b79f5) >>> 0;
|
||||
let value = Math.imul(state ^ (state >>> 15), 1 | state);
|
||||
value = (value + Math.imul(value ^ (value >>> 7), 61 | value)) ^ value;
|
||||
return ((value ^ (value >>> 14)) >>> 0) / 4294967296;
|
||||
};
|
||||
}
|
||||
|
||||
type PlanNode = { "Relation Name"?: string; "Actual Loops"?: number; "Actual Rows"?: number;
|
||||
"Rows Removed by Filter"?: number; "Rows Removed by Index Recheck"?: number; Plans?: PlanNode[] };
|
||||
|
||||
function heartbeatRunRowsRead(node: PlanNode): number {
|
||||
const own = node["Relation Name"] === "heartbeat_runs" ? (node["Actual Loops"] ?? 0)
|
||||
* ((node["Actual Rows"] ?? 0) + (node["Rows Removed by Filter"] ?? 0) + (node["Rows Removed by Index Recheck"] ?? 0)) : 0;
|
||||
return own + (node.Plans ?? []).reduce((total, child) => total + heartbeatRunRowsRead(child), 0);
|
||||
}
|
||||
|
||||
type Tenant = { companyId: string; endpointId: string; agentIds: [string, string];
|
||||
requests: Array<{ issueId: string; deliveryId: string; inboundCommentId: string | null; outboundCommentId: string }> };
|
||||
|
||||
suite("GitHub response run state", () => {
|
||||
let db!: Db;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
const tenants: Tenant[] = [];
|
||||
const history = 1_500;
|
||||
|
||||
async function seedTenant(index: number): Promise<Tenant> {
|
||||
const client = db.$client;
|
||||
const [{ company }] = await client.unsafe(`insert into companies (name, issue_prefix) values ($1, $2) returning id as company`,
|
||||
[`Response ${index}`, `RS${index}`]);
|
||||
const agentRows = await client.unsafe(`insert into agents (company_id, name) values ($1, 'GitHub agent'), ($1, 'Other agent')
|
||||
returning id`, [company]);
|
||||
const agentIds = agentRows.map(row => row.id as string) as [string, string];
|
||||
const [{ application }] = await client.unsafe(`insert into tool_applications (company_id, application_key, name, type)
|
||||
values ($1, 'github', 'GitHub', 'chat') returning id as application`, [company]);
|
||||
const [{ connection }] = await client.unsafe(`insert into tool_connections (company_id, application_id, name, uid, transport, connection_purpose)
|
||||
values ($1, $2, 'GitHub', $3, 'chat_sdk', 'channel') returning id as connection`, [company, application, `github-${index}`]);
|
||||
const [{ endpoint }] = await client.unsafe(`insert into chat_endpoints (company_id, connection_id, provider, public_id, assigned_agent_id, status)
|
||||
values ($1, $2, 'github', $3, $4, 'active') returning id as endpoint`, [company, connection, `response-${index}`, agentIds[0]]);
|
||||
const requests: Tenant["requests"] = [];
|
||||
for (let issueNumber = 1; issueNumber <= 60; issueNumber++) {
|
||||
const [{ issue }] = await client.unsafe(`insert into issues (company_id, title, status) values ($1, $2, 'in_progress') returning id as issue`,
|
||||
[company, `Issue ${issueNumber}`]);
|
||||
const [{ conversation }] = await client.unsafe(`insert into chat_conversations (company_id, endpoint_id, issue_id,
|
||||
external_conversation_id, external_label, external_thread_id)
|
||||
values ($1, $2, $3, $4, $4, $4) returning id as conversation`, [company, endpoint, issue, `github:paperclipai/paperclip:${issueNumber}`]);
|
||||
for (let request = 0; request < 1 + (issueNumber % 2); request++) {
|
||||
const [{ delivery }] = await client.unsafe(`insert into chat_deliveries (company_id, endpoint_id, provider_event_id,
|
||||
deduplication_key, event_kind, normalized_event, conversation_id, state)
|
||||
values ($1, $2, $3, $3, 'message', '{}', $4, 'processed') returning id as delivery`,
|
||||
[company, endpoint, `event-${issueNumber}-${request}`, conversation]);
|
||||
const commentRows = await client.unsafe(`insert into issue_comments (company_id, issue_id, body)
|
||||
values ($1, $2, 'Inbound'), ($1, $2, 'Outbound') returning id`, [company, issue]);
|
||||
const inboundCommentId = issueNumber % 13 === 0 ? null : commentRows[0]!.id as string;
|
||||
const outboundCommentId = commentRows[1]!.id as string;
|
||||
await client.unsafe(`insert into chat_message_links (company_id, endpoint_id, conversation_id, delivery_id, comment_id,
|
||||
provider_message_id, direction)
|
||||
values ($1, $2, $3, $4, $5, $6, 'inbound'), ($1, $2, $3, $4, $7, $8, 'outbound')`,
|
||||
[company, endpoint, conversation, delivery, inboundCommentId, `in-${issueNumber}-${request}`,
|
||||
outboundCommentId, `out-${issueNumber}-${request}`]);
|
||||
requests.push({ issueId: issue, deliveryId: delivery, inboundCommentId, outboundCommentId });
|
||||
}
|
||||
}
|
||||
return { companyId: company, endpointId: endpoint, agentIds, requests };
|
||||
}
|
||||
|
||||
async function seedRuns(seed: number) {
|
||||
const next = random(seed);
|
||||
const pick = <T,>(values: readonly T[]): T => values[Math.floor(next() * values.length)]!;
|
||||
const base = Date.UTC(2026, 10, 1);
|
||||
const rows: Array<typeof heartbeatRuns.$inferInsert> = [];
|
||||
let timeline = 0;
|
||||
for (const tenant of tenants) {
|
||||
const other = tenants.find(candidate => candidate !== tenant)!;
|
||||
const issueIds = [...new Set(tenant.requests.map(request => request.issueId))];
|
||||
for (const issueId of issueIds) {
|
||||
timeline += 1;
|
||||
const requests = tenant.requests.filter(request => request.issueId === issueId);
|
||||
const slots = Array.from({ length: 1 + Math.floor(next() * 6) }, (_, slot) => slot);
|
||||
let linkedAt = new Set<number>();
|
||||
for (const slot of slots) {
|
||||
const tie = slot > 0 && next() < 0.3;
|
||||
const minute = tie ? slot - 1 : slot;
|
||||
const request = pick(requests);
|
||||
let wake = pick(["none", "comment", "comments", "scalar", "sibling", "outbound", "foreign"] as const);
|
||||
const links = wake === "comment" || wake === "comments" || wake === "scalar" || wake === "sibling";
|
||||
if (links && linkedAt.has(minute)) wake = "none";
|
||||
else if (links) linkedAt = new Set([...linkedAt, minute]);
|
||||
const sibling = requests.find(candidate => candidate !== request) ?? request;
|
||||
const foreignComment = pick(other.requests).inboundCommentId;
|
||||
const wakeContext = wake === "comment" ? { wakeCommentId: request.inboundCommentId }
|
||||
: wake === "comments" ? { wakeCommentIds: [randomUUID(), request.inboundCommentId] }
|
||||
: wake === "scalar" ? { wakeCommentIds: request.inboundCommentId }
|
||||
: wake === "outbound" ? { wakeCommentId: request.outboundCommentId }
|
||||
: wake === "foreign" ? { wakeCommentId: foreignComment }
|
||||
: wake === "sibling" ? { wakeCommentIds: [sibling.inboundCommentId] } : {};
|
||||
const binding = pick(["native", "native", "legacy", "legacy", "nativeElsewhere", "otherCompany", "otherAgent"] as const);
|
||||
const nativeIssueId = binding === "native" ? issueId : binding === "nativeElsewhere" ? pick(issueIds) : null;
|
||||
rows.push({
|
||||
id: randomUUID(),
|
||||
companyId: binding === "otherCompany" ? other.companyId : tenant.companyId,
|
||||
agentId: binding === "otherCompany" ? other.agentIds[0] : binding === "otherAgent" ? tenant.agentIds[1] : pick([tenant.agentIds[0], tenant.agentIds[0], tenant.agentIds[1]]),
|
||||
scopeKind: nativeIssueId ? "issue" : "company",
|
||||
issueId: nativeIssueId,
|
||||
nativeIssueId,
|
||||
status: pick(["queued", "running", "succeeded", "failed", "cancelled", "timed_out"]),
|
||||
createdAt: new Date(base + timeline * 3_600_000 + minute * 60_000),
|
||||
contextSnapshot: { issueId, ...wakeContext },
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
await db.insert(heartbeatRuns).values(rows);
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("paperclip-github-response-run-state-");
|
||||
db = createDb(temporary.connectionString);
|
||||
tenants.push(await seedTenant(0), await seedTenant(1));
|
||||
await seedRuns(20261114);
|
||||
await db.$client.unsafe(`insert into heartbeat_runs (company_id, agent_id, status, context_snapshot, created_at)
|
||||
select $1::uuid, $2::uuid, 'succeeded', jsonb_build_object('issueId', gen_random_uuid()::text,
|
||||
'prompt', (select string_agg(md5(random()::text || g::text || n::text), '') from generate_series(1, 64) g)),
|
||||
timestamptz '2026-10-01' + (n || ' minutes')::interval from generate_series(1, ${history}) n`,
|
||||
[tenants[0]!.companyId, tenants[0]!.agentIds[0]]);
|
||||
await db.execute(sql`analyze`);
|
||||
}, 120_000);
|
||||
|
||||
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
|
||||
|
||||
function inputs(): Input[] {
|
||||
const all: Input[] = [];
|
||||
for (const tenant of tenants) {
|
||||
const other = tenants.find(candidate => candidate !== tenant)!;
|
||||
for (const request of tenant.requests) {
|
||||
for (const agentId of [...tenant.agentIds, other.agentIds[0]]) {
|
||||
all.push({ companyId: tenant.companyId, endpointId: tenant.endpointId, agentId, issueId: request.issueId,
|
||||
deliveryId: request.deliveryId, sourceMessageId: request.inboundCommentId });
|
||||
}
|
||||
all.push({ companyId: tenant.companyId, endpointId: other.endpointId, agentId: tenant.agentIds[0], issueId: request.issueId,
|
||||
deliveryId: request.deliveryId, sourceMessageId: null });
|
||||
all.push({ companyId: tenant.companyId, endpointId: tenant.endpointId, agentId: tenant.agentIds[0], issueId: null,
|
||||
deliveryId: request.deliveryId, sourceMessageId: null });
|
||||
}
|
||||
}
|
||||
return all;
|
||||
}
|
||||
|
||||
it("returns the latest run and request ownership the previous queries returned, for every run binding and wake link", async () => {
|
||||
const before = [], after = [];
|
||||
for (const input of inputs()) {
|
||||
before.push(await previousReadGitHubResponseRun(db, input));
|
||||
after.push(await readGitHubResponseRun(db, input));
|
||||
}
|
||||
expect(after).toEqual(before);
|
||||
expect(before.filter(result => result.ownsLatestRequest).length).toBeGreaterThan(20);
|
||||
expect(before.filter(result => result.latestRun && !result.ownsLatestRequest).length).toBeGreaterThan(20);
|
||||
expect(before.filter(result => !result.latestRun).length).toBeGreaterThan(20);
|
||||
expect(new Set(before.flatMap(result => result.latestRun ? [result.latestRun.nativeIssueId ? "native" : "legacy"] : [])))
|
||||
.toEqual(new Set(["native", "legacy"]));
|
||||
}, 120_000);
|
||||
|
||||
it("reads a few runs of the issue, not the agent's whole run history", async () => {
|
||||
const tenant = tenants[0]!;
|
||||
const request = tenant.requests[0]!;
|
||||
const captured: Array<{ sql: string; params: unknown[] }> = [];
|
||||
const capture = new Proxy(db, { get(target, key) {
|
||||
if (key !== "select") return Reflect.get(target, key);
|
||||
return (...fields: Parameters<typeof db.select>) => {
|
||||
const builder = target.select(...fields), from = builder.from.bind(builder);
|
||||
builder.from = ((...args: Parameters<typeof from>) => {
|
||||
const selection = from(...args), execute = selection.execute.bind(selection);
|
||||
selection.execute = (async () => { captured.push(selection.toSQL()); return execute(); }) as typeof selection.execute;
|
||||
return selection;
|
||||
}) as typeof builder.from;
|
||||
return builder;
|
||||
};
|
||||
} });
|
||||
await readGitHubResponseRun(capture as Db, { companyId: tenant.companyId, endpointId: tenant.endpointId,
|
||||
agentId: tenant.agentIds[0], issueId: request.issueId, deliveryId: request.deliveryId, sourceMessageId: null });
|
||||
expect(captured.length).toBeGreaterThan(0);
|
||||
let rowsRead = 0;
|
||||
for (const query of captured) {
|
||||
const [row] = await db.$client.unsafe(`explain (analyze, format json) ${query.sql}`, query.params as never[]);
|
||||
rowsRead += heartbeatRunRowsRead((row!["QUERY PLAN"] as Array<{ Plan: PlanNode }>)[0]!.Plan);
|
||||
}
|
||||
expect(rowsRead).toBeLessThan(history / 10);
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -1,4 +1,4 @@
|
||||
import { and, desc, eq, or, sql } from "drizzle-orm";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { chatMessageLinks, heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import { chatRunSourceMessageId } from "../run-origin.js";
|
||||
|
||||
@@ -52,21 +52,30 @@ export async function readGitHubResponseRun(db: Db, input: {
|
||||
companyId: string; endpointId: string; agentId: string; issueId: string | null;
|
||||
deliveryId: string; sourceMessageId: string | null;
|
||||
}) {
|
||||
const [requestRun] = await db.select({ runId: heartbeatRuns.id }).from(heartbeatRuns).innerJoin(chatMessageLinks, and(
|
||||
eq(chatMessageLinks.companyId, input.companyId), eq(chatMessageLinks.endpointId, input.endpointId),
|
||||
eq(chatMessageLinks.deliveryId, input.deliveryId), eq(chatMessageLinks.direction, "inbound"),
|
||||
or(sql`${chatMessageLinks.commentId}::text = ${heartbeatRuns.contextSnapshot}->>'wakeCommentId'`,
|
||||
sql`coalesce(${heartbeatRuns.contextSnapshot}->'wakeCommentIds', '[]'::jsonb) ? ${chatMessageLinks.commentId}::text`),
|
||||
)).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId),
|
||||
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${input.issueId}`))
|
||||
.orderBy(desc(heartbeatRuns.createdAt)).limit(1);
|
||||
const [latestRun] = await db.select(responseRunColumns).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId),
|
||||
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${input.issueId}`,
|
||||
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
||||
const originMessageId = latestRun?.activeIdentityContextId ? await chatRunSourceMessageId(db, latestRun,
|
||||
const { issueId } = input;
|
||||
if (!issueId) return { latestRun: undefined, ownsLatestRequest: false };
|
||||
const latestRunId = sql`(select latest.id from (
|
||||
(select r.id, r.created_at from ${heartbeatRuns} r
|
||||
where r.company_id = ${input.companyId} and r.native_issue_id = ${issueId} and r.agent_id = ${input.agentId}
|
||||
order by r.created_at desc, r.id desc limit 1)
|
||||
union all
|
||||
(select r.id, r.created_at from ${heartbeatRuns} r
|
||||
where r.company_id = ${input.companyId} and r.native_issue_id is null
|
||||
and r.context_snapshot->>'issueId' = ${issueId} and r.agent_id = ${input.agentId}
|
||||
order by r.created_at desc, r.id desc limit 1)
|
||||
) latest order by latest.created_at desc, latest.id desc limit 1)`;
|
||||
const wokeForRequest = sql<boolean>`exists (select 1 from ${chatMessageLinks} l
|
||||
where l.company_id = ${input.companyId} and l.endpoint_id = ${input.endpointId}
|
||||
and l.delivery_id = ${input.deliveryId} and l.direction = 'inbound'
|
||||
and (l.comment_id::text = ${heartbeatRuns.contextSnapshot}->>'wakeCommentId'
|
||||
or coalesce(${heartbeatRuns.contextSnapshot}->'wakeCommentIds', '[]'::jsonb) ? l.comment_id::text))`;
|
||||
const [row] = await db.select({ ...responseRunColumns, wokeForRequest }).from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, input.companyId), sql`${heartbeatRuns.id} = ${latestRunId}`));
|
||||
if (!row) return { latestRun: undefined, ownsLatestRequest: false };
|
||||
const { wokeForRequest: requestRunIsLatest, ...latestRun } = row;
|
||||
const originMessageId = latestRun.activeIdentityContextId ? await chatRunSourceMessageId(db, latestRun,
|
||||
{ allowLegacyWakeComment: true, provider: "github" }) : null;
|
||||
const ownsLatestRequest = !!latestRun && (requestRun?.runId === latestRun.id ||
|
||||
(!!input.sourceMessageId && originMessageId === input.sourceMessageId));
|
||||
const ownsLatestRequest = requestRunIsLatest ||
|
||||
(!!input.sourceMessageId && originMessageId === input.sourceMessageId);
|
||||
return { latestRun, ownsLatestRequest };
|
||||
}
|
||||
Reference in new issue
Block a user