mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 22:13:59 +01:00
perf(github): reuse the inherited review wait query template (#18965)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External review waits check current GitHub review and publication bindings. > - The GitHub service rebuilds the same seven-table Drizzle query on each check. > - A sampled production window attributed 6.3% of SQL serialization CPU to this query shape. > - This pull request prepares that query once per exact database or transaction object. > - Each call still reads current provenance, review, connection and publication state. > - The local service benchmark reduces median client CPU per read by 24.35%. ## Linked Issues or Issue Description **What existing behavior does this improve?** Repeated `readInheritedGitHubReviewWait` calls from external review wait reads and attention retention. **Subsystem affected** Server GitHub chat review waits. **Current behavior** Every call constructs and serializes the same joined SELECT. The database driver already supports prepared statements. That setting does not reuse the JavaScript query template. **Proposed behavior** Compile the existing selector once for each supplied database, transaction or savepoint object. Bind company, issue, agent and comment IDs on each execution. Keep every current database read and check. **Reason and benefit** Remove repeated query construction and serialization CPU. Do not cache rows or authorization decisions. Exact open-PR searches found no duplicate query-template change. ## What Changed - Add an exact-handle `WeakMap` for the GitHub review selector and execute it with fresh placeholders. - Preserve the SQL projection, joins, JSON types, latest-review order and all current authority checks. - Extend real PostgreSQL tests for new bindings, warm-cache revocation, latest heads and transaction/savepoint rollback. - Capture prepared executions in the existing test helper. - Add an actual-service benchmark and a report with every timing sample and setup failure. ## Verification - `github-review-wait.test.ts`: 16 tests passed on real PostgreSQL. - `external-chat-wait.integration.test.ts -t 'reads small current external review waits without saved prompts'`: four affected cases passed; 269 unrelated cases skipped. - Pinned baseline: `e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Final source SHA256: `fe647e7fe2aed82609e8f1d66a0109e7a10ee28313ae2ef218ba3ab39b5cd105`. - Exclusive-window benchmark: 500 warmups per version, nine alternating pairs of 100 complete service reads on disposable PostgreSQL. Median client CPU: 1.00336 → 0.75908 ms/read (24.35% less). Median wall time: 2.87089 → 2.38731 ms/read (16.84% less). CPU excludes the PostgreSQL process. - Every generated SQL statement, parameter, query count and complete result matched. Warm joined-selector compilations per ten service reads: 10 → 0. - See `doc/performance/2026-11-04-prepared-github-review-wait.md` for reproduction, all samples and measurement limits. - Setup record: loopback required execution outside the sandbox. The fresh embedded-Postgres dependency lacked macOS dylib aliases; aliases to existing identical bytes restored the fixture. The SDK import needed a current workspace SDK/shared build. A broad external-chat suite run was interrupted and replaced by the affected test filter. Those attempts are not counted as passing suites. - Greptile: 5/5 on current head `d2e6ae8814c3a56c380bd5d60c47e6f98642f085`, with no actionable issues or inline comments. - Current-head CI is green at `d2e6ae8814c3a56c380bd5d60c47e6f98642f085`: 52 successful checks, three intentional skips and the successful Snyk status. This includes typecheck, build, the full test matrix, end-to-end shards and the aggregate gate. - Combined local validation at `4ab58ff70b` includes both measured production templates and final test fixes. Full recursive `pnpm -r typecheck` passed in 39.864 seconds. Three real PostgreSQL files passed 35/35 tests (five chat cases, 16 GitHub review cases and 14 shared prepared-query cases). The combined build passed earlier with the same production template bytes. - The initial combined typecheck failed on a separate chat test typing error and build setup. The later passing run resolves that attempt. The original full local `pnpm test:run` was deliberately interrupted after six chat shards passed and partial general tests ran; it is not a passing full local suite. Each PR has independent current-head CI for broad coverage. The initial loopback sandbox restriction was resolved by an escalated run of the same source. - Local benchmark/test dependencies were locked at the baseline; the rebase adds the upstream lockfile refresh and UI paste fix only. ## Risks - A prepared query captures its session. The exact supplied handle is the cache key so transactions and savepoints do not borrow the root session. - Placeholder mistakes could cross bindings. Tests alternate independent company/issue/agent/comment fixtures on the same handle. - The benchmark has one synthetic eligible fixture with warm sequential reads. It does not predict production endpoint latency or whole-server CPU savings. - No schema, index, database driver setting or public API changes. ## Model Used OpenAI `gpt-6.1-sol` through Codex. The context-window size is not exposed by the session. Used reasoning, tool calls, source editing and code execution. Independent Codex review found no correctness issues. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
76e43852ac
commit
2cbfc35a07
5 files changed
+311
-27
No files matched your search
@@ -80,8 +80,41 @@ describe("inherited GitHub review wait projection", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it("binds each call afresh while reusing the compiled query", async () => {
|
||||
const first = await seedGitHubReviewWait(db), second = await seedGitHubReviewWait(db);
|
||||
const queries: CapturedReviewWaitQuery[] = [];
|
||||
const captured = captureReviewWaitQueries(db, queries);
|
||||
for (const f of [first, second, first]) {
|
||||
expect(await readInheritedGitHubReviewWait(captured, f.run)).toMatchObject({ reviewId: f.reviewId });
|
||||
}
|
||||
const joined = queries.filter(query => query.sql.includes('join "chat_github_reviews"'));
|
||||
expect(joined).toHaveLength(3);
|
||||
expect(joined[0]).toEqual(joined[2]);
|
||||
expect(joined[0]!.sql).toBe(joined[1]!.sql);
|
||||
expect(joined[0]!.params).not.toEqual(joined[1]!.params);
|
||||
});
|
||||
|
||||
it("keeps transaction and savepoint reads on their own sessions", async () => {
|
||||
const f = await seedGitHubReviewWait(db);
|
||||
expect((await readInheritedGitHubReviewWait(db, f.run))?.kind).toBe("published");
|
||||
await expect(db.transaction(async tx => {
|
||||
const scoped = tx as unknown as Db;
|
||||
await tx.update(toolConnections).set({ enabled: false }).where(eq(toolConnections.id, f.connectionId));
|
||||
expect(await readInheritedGitHubReviewWait(scoped, f.run)).toEqual({ kind: "revoked" });
|
||||
await expect(tx.transaction(async savepoint => {
|
||||
await savepoint.update(toolConnections).set({ enabled: true }).where(eq(toolConnections.id, f.connectionId));
|
||||
expect((await readInheritedGitHubReviewWait(savepoint as unknown as Db, f.run))?.kind).toBe("published");
|
||||
throw new Error("rollback savepoint");
|
||||
})).rejects.toThrow("rollback savepoint");
|
||||
expect(await readInheritedGitHubReviewWait(scoped, f.run)).toEqual({ kind: "revoked" });
|
||||
throw new Error("rollback transaction");
|
||||
})).rejects.toThrow("rollback transaction");
|
||||
expect((await readInheritedGitHubReviewWait(db, f.run))?.kind).toBe("published");
|
||||
});
|
||||
|
||||
it("does not reuse an older completed review when a newer head is queued", async () => {
|
||||
const f = await seedGitHubReviewWait(db);
|
||||
expect((await readInheritedGitHubReviewWait(db, f.run))?.kind).toBe("published");
|
||||
const [review] = await db.select().from(chatGitHubReviews).where(eq(chatGitHubReviews.id, f.reviewId));
|
||||
await db.insert(chatGitHubReviews).values({ ...review!, id: randomUUID(), deliveryId: randomUUID(),
|
||||
headSha: "2".repeat(40), state: "queued", createdAt: new Date(review!.createdAt.getTime() + 1_000) });
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { and, eq, fillPlaceholders } from "drizzle-orm";
|
||||
import { agents, chatActions, chatConversations, chatEndpointResources, chatEndpoints,
|
||||
chatGitHubReviews, chatMessageLinks, companies, heartbeatRuns, issueComments, issues,
|
||||
runIdentityContexts, toolApplications, toolConnections, type Db } from "@paperclipai/db";
|
||||
@@ -61,6 +61,16 @@ export function captureReviewWaitQueries(db: Db, queries: CapturedReviewWaitQuer
|
||||
const builder = target.select(...fields), from = builder.from.bind(builder);
|
||||
builder.from = ((...args: Parameters<typeof from>) => {
|
||||
const selection = from(...args), execute = selection.execute.bind(selection);
|
||||
const prepare = selection.prepare.bind(selection);
|
||||
selection.prepare = ((...args: Parameters<typeof prepare>) => {
|
||||
const prepared = prepare(...args), executePrepared = prepared.execute.bind(prepared);
|
||||
prepared.execute = async (values = {}) => {
|
||||
const query = prepared.getQuery();
|
||||
queries.push({ sql: query.sql, params: fillPlaceholders(query.params, values) });
|
||||
return executePrepared(values);
|
||||
};
|
||||
return prepared;
|
||||
}) as typeof selection.prepare;
|
||||
selection.execute = async () => { queries.push(selection.toSQL()); return execute(); };
|
||||
return selection;
|
||||
}) as typeof builder.from;
|
||||
|
||||
@@ -33,6 +33,47 @@ export async function hasPublishedGitHubReview(db: Db, binding: {
|
||||
type WaitRun = Pick<typeof heartbeatRuns.$inferSelect,
|
||||
"id" | "companyId" | "agentId" | "responsibleUserId" | "activeIdentityContextId" | "nativeIssueId" | "issueId" | "contextSnapshot">;
|
||||
|
||||
const inheritedReviewQueries = new WeakMap<Db, ReturnType<typeof prepareInheritedReviewQuery>>();
|
||||
|
||||
function prepareInheritedReviewQuery(db: Db) {
|
||||
return db.select({
|
||||
id: chatGitHubReviews.id, endpointId: chatGitHubReviews.endpointId,
|
||||
state: chatGitHubReviews.state, repository: chatGitHubReviews.repository,
|
||||
repositoryId: chatGitHubReviews.repositoryId, headSha: chatGitHubReviews.headSha,
|
||||
reviewedCommit: sql<unknown>`${chatGitHubReviews.assessment}->'reviewedCommit'`, reviewRunId: chatGitHubReviews.runId,
|
||||
connectionEnabled: toolConnections.enabled, connectionStatus: toolConnections.status,
|
||||
reviewAgentId: heartbeatRuns.agentId, reviewCompanyId: heartbeatRuns.companyId,
|
||||
resourceRepository: chatEndpointResources.providerResourceId,
|
||||
resourceRepositoryId: sql<string | null>`${chatEndpointResources.metadata}->>'providerRepositoryId'`,
|
||||
pullNumber: chatGitHubReviews.pullNumber, thread: chatConversations.externalThreadId,
|
||||
}).from(chatMessageLinks)
|
||||
.innerJoin(chatConversations, and(eq(chatConversations.id, chatMessageLinks.conversationId),
|
||||
eq(chatConversations.companyId, sql.placeholder("companyId")), eq(chatConversations.issueId, sql.placeholder("issueId"))))
|
||||
.innerJoin(chatEndpoints, and(eq(chatEndpoints.id, chatConversations.endpointId),
|
||||
eq(chatEndpoints.companyId, sql.placeholder("companyId")), eq(chatEndpoints.provider, "github"),
|
||||
eq(chatEndpoints.assignedAgentId, sql.placeholder("agentId"))))
|
||||
.leftJoin(toolConnections, and(eq(toolConnections.id, chatEndpoints.connectionId),
|
||||
eq(toolConnections.companyId, sql.placeholder("companyId"))))
|
||||
.innerJoin(chatEndpointResources, and(eq(chatEndpointResources.id, chatConversations.resourceId),
|
||||
eq(chatEndpointResources.companyId, sql.placeholder("companyId")), eq(chatEndpointResources.endpointId, chatEndpoints.id)))
|
||||
.innerJoin(chatGitHubReviews, and(eq(chatGitHubReviews.endpointId, chatEndpoints.id),
|
||||
eq(chatGitHubReviews.companyId, sql.placeholder("companyId")), eq(chatGitHubReviews.issueId, sql.placeholder("issueId"))))
|
||||
.leftJoin(heartbeatRuns, eq(heartbeatRuns.id, chatGitHubReviews.runId))
|
||||
.where(and(eq(chatMessageLinks.companyId, sql.placeholder("companyId")), eq(chatMessageLinks.commentId, sql.placeholder("commentId")),
|
||||
eq(chatMessageLinks.direction, "inbound")))
|
||||
.orderBy(desc(chatGitHubReviews.createdAt), desc(chatGitHubReviews.id)).limit(1)
|
||||
.prepare("inherited_github_review_wait");
|
||||
}
|
||||
|
||||
function inheritedReviewQuery(db: Db) {
|
||||
let query = inheritedReviewQueries.get(db);
|
||||
if (!query) {
|
||||
query = prepareInheritedReviewQuery(db);
|
||||
inheritedReviewQueries.set(db, query);
|
||||
}
|
||||
return query;
|
||||
}
|
||||
|
||||
/** Passive review waits inherit the same accepted requester lineage as GitHub
|
||||
* tools. This is not a grant to publish, run a monitor, or reuse attachments. */
|
||||
export async function readInheritedGitHubReviewWait(db: Db, run: WaitRun) {
|
||||
@@ -45,32 +86,9 @@ export async function readInheritedGitHubReviewWait(db: Db, run: WaitRun) {
|
||||
throw error;
|
||||
}
|
||||
if (!commentId) return null;
|
||||
const [review] = await db.select({
|
||||
id: chatGitHubReviews.id, endpointId: chatGitHubReviews.endpointId,
|
||||
state: chatGitHubReviews.state, repository: chatGitHubReviews.repository,
|
||||
repositoryId: chatGitHubReviews.repositoryId, headSha: chatGitHubReviews.headSha,
|
||||
reviewedCommit: sql<unknown>`${chatGitHubReviews.assessment}->'reviewedCommit'`, reviewRunId: chatGitHubReviews.runId,
|
||||
connectionEnabled: toolConnections.enabled, connectionStatus: toolConnections.status,
|
||||
reviewAgentId: heartbeatRuns.agentId, reviewCompanyId: heartbeatRuns.companyId,
|
||||
resourceRepository: chatEndpointResources.providerResourceId,
|
||||
resourceRepositoryId: sql<string | null>`${chatEndpointResources.metadata}->>'providerRepositoryId'`,
|
||||
pullNumber: chatGitHubReviews.pullNumber, thread: chatConversations.externalThreadId,
|
||||
}).from(chatMessageLinks)
|
||||
.innerJoin(chatConversations, and(eq(chatConversations.id, chatMessageLinks.conversationId),
|
||||
eq(chatConversations.companyId, run.companyId), eq(chatConversations.issueId, run.nativeIssueId)))
|
||||
.innerJoin(chatEndpoints, and(eq(chatEndpoints.id, chatConversations.endpointId),
|
||||
eq(chatEndpoints.companyId, run.companyId), eq(chatEndpoints.provider, "github"),
|
||||
eq(chatEndpoints.assignedAgentId, run.agentId)))
|
||||
.leftJoin(toolConnections, and(eq(toolConnections.id, chatEndpoints.connectionId),
|
||||
eq(toolConnections.companyId, run.companyId)))
|
||||
.innerJoin(chatEndpointResources, and(eq(chatEndpointResources.id, chatConversations.resourceId),
|
||||
eq(chatEndpointResources.companyId, run.companyId), eq(chatEndpointResources.endpointId, chatEndpoints.id)))
|
||||
.innerJoin(chatGitHubReviews, and(eq(chatGitHubReviews.endpointId, chatEndpoints.id),
|
||||
eq(chatGitHubReviews.companyId, run.companyId), eq(chatGitHubReviews.issueId, run.nativeIssueId)))
|
||||
.leftJoin(heartbeatRuns, eq(heartbeatRuns.id, chatGitHubReviews.runId))
|
||||
.where(and(eq(chatMessageLinks.companyId, run.companyId), eq(chatMessageLinks.commentId, commentId),
|
||||
eq(chatMessageLinks.direction, "inbound")))
|
||||
.orderBy(desc(chatGitHubReviews.createdAt), desc(chatGitHubReviews.id)).limit(1);
|
||||
const [review] = await inheritedReviewQuery(db).execute({
|
||||
companyId: run.companyId, issueId: run.nativeIssueId, agentId: run.agentId, commentId,
|
||||
});
|
||||
// A newer unreviewed head cannot inherit the prior head's publication.
|
||||
if (!review) return null;
|
||||
if (!review.connectionEnabled || review.connectionStatus !== "active" || review.state !== "completed" ||
|
||||
|
||||
Reference in new issue
Block a user