mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 03:29:18 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel has two sweeps for request response
comments: the acknowledgement sweep (`acknowledgement-work.ts`) and the
terminal-response sweep (`terminal-response-work.ts`). Each reads pages
of up to 100 requests.
> - For each loaded request, both sweeps call `readGitHubResponseRun`
one row at a time. That is one latest-run statement, plus the run
identity context read and the message link read of
`chatRunSourceMessageId`. A page of 100 requests costs about 300
statements.
> - On paperclip.staging, those two provenance reads ran at about 13/s
each. #21705 stops the idle backlog from being reloaded. This PR makes
the reads that remain cost a constant number of statements per page.
> - The per-row read also had a failure mode. A 403 from one run's
provenance check aborted the acknowledgement sweep's whole page, so no
request of that company was acknowledged until the 403 went away.
> - This pull request adds a batched `readGitHubResponseRuns` and uses
it once per page in both sweeps.
> - The benefit is a constant number of statements per page, and a
provenance failure that affects only its own request.
## Linked Issues or Issue Description
Refs #21705 (the same sweeps; that PR stops reloading requests that
cannot progress).
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
On paperclip.staging (2026-11-15, about 01:45–02:25Z),
`provenanceReads.context` (`select … from run_identity_contexts where id
= $1 and run_id = $2 and company_id = $3 and status = $4 limit $5`) ran
at 13.37/s, and `provenanceReads.link` (`select
chat_conversations.issue_id, chat_endpoints.provider from
chat_message_links join … where chat_message_links.company_id = $3 and
comment_id = $4 …`) ran at 13.20/s. Both GitHub response sweeps reach
them through `readGitHubResponseRun` → `chatRunSourceMessageId(db, run,
{ provider: "github" })`, once for each loaded row. `load()` costs one
page read plus three statements per row. `check()` read the run state
twice for each due request (before and after the acknowledgement).
In the acknowledgement sweep, a 403 from `chatRunSourceMessageId` (for
example, a run whose identity context is no longer accepted) threw out
of `entry()` and aborted the whole page. The terminal sweep already kept
such a row due, so the acknowledgement owner could revoke it.
**Expected behavior**
A page of requests costs a constant number of statements. A failure for
one run affects only the requests of that run.
**Steps to reproduce**
1. Create 120 GitHub response requests on 30 tasks. Give each task an
ended run with an accepted identity context.
2. Register the terminal-response sweep. On master, the first pass costs
one page read plus three statements per row.
3. Mark the requests `received`, revoke one run's identity context, and
register the acknowledgement sweep. On master, the load fails with a
403, and none of the 150 requests is acknowledged.
**Paperclip version or commit**
master at ea8c734cb9 (includes #21705).
## What Changed
- `server/src/services/chat-channels/github/response-run-state.ts`:
- The latest-run subquery and the "woke for this request" check are
built once, from literals (single read) or from the columns of a request
row (batch read). The single read keeps the same query shape.
- New `readGitHubResponseRuns(db, inputs)`. It reads the latest run of
every request of a page in one statement: `unnest(…) with ordinality`
and a lateral latest-run subquery, then a join to `heartbeat_runs` by id
and company. It then calls the existing `chatRunSourceMessageIds(db,
runs, { allowLegacyWakeComment: true, provider: "github" })` once for
the page's distinct runs. That helper reuses `provenanceReads`, so the
scope, status filters and joins are the same as the single check.
Results map back by position. A run whose provenance check is forbidden
gives `{ forbidden }` for its requests only.
- `server/src/services/chat-channels/github/response-execution.ts`:
`githubResponseExecution` takes an optional pre-read state. It throws a
`{ forbidden }` state as the same 403 that the single read throws.
- `server/src/services/chat-channels/github/terminal-response-work.ts`:
`load()` reads each page's run state with one `readGitHubResponseRuns`
call. `check()` reads the run state once: after the acknowledgement it
reads the row only, and keeps a request that is still owed for the next
check, which reads its run state again first.
- `server/src/services/chat-channels/github/acknowledgement-work.ts`:
the same batching. A forbidden provenance check keeps that request due,
so the acknowledgement owner revokes it (`authorization_changed`), and
the rest of the page continues. After an acknowledgement, a still-owed
request keeps the deadline from its own row (retry time, fast-response
expiry or lease).
- Tests:
- `terminal-response-work.test.ts`: `reads a page of waiting responses
in a constant number of statements` and `acknowledges every owed request
of a page when one run's requester provenance changed`.
- `response-run-state.test.ts`: `reads a page of requests in one
latest-run statement with the state each single read returns` compares
the batch with the single read for every seeded binding and wake link.
- `response-sweep-scope.test.ts`: `rebuilds parked requests once per
interval for company-wide hints` counted run-state *statements* as a
proxy for requests reloaded. With batching, one statement reads a whole
page, so the count dropped from 6 to 2 and the test failed in CI. The
behavior it guards must stay: a company-wide rebuild reloads every
eligible request once per interval, and repeated hints inside the
interval reload none. The test now counts the requests that each
run-state read names (the length of a batched read's request arrays, or
1 for a single read). The expectations stay 6 / `[6, 0, 0, 0, 0]` / 6,
so the test still proves the same rebuild behavior.
No GitHub comment behavior changes.
## Verification
- `pnpm vitest run
src/services/chat-channels/github/terminal-response-work.test.ts`
(server), staging-shaped fixtures (120 waiting requests on 30 tasks;
codes null / `acknowledgement_unconfirmed` /
`acknowledgement_superseded`):
- before (#21705 only): the first pass costs 364 statements, with 120
run identity reads. With one revoked context, the acknowledgement sweep
acknowledges 0 of 150 owed requests. The 2 new tests fail (red).
- after: the first pass costs 10 statements, with 2 run identity reads
(per page: page, latest runs, context, link, wait keys). All 150 owed
requests are acknowledged, including the 5 whose provenance is
forbidden. 8 of 8 tests pass (green). All 7 files under
`chat-channels/github/`: 60 passed. `idle-completed-history.test.ts`: 75
passed.
- On master before #21705, the first pass was 362 statements, and every
company rebuild repeated it.
- `pnpm vitest run
src/services/chat-channels/github/response-run-state.test.ts`: 3 passed.
- `pnpm vitest run src/__tests__/chat-channels.integration.test.ts -t
"terminal|acknowledg|GitHub response|response comment|GitHub review"`:
192 passed.
- `pnpm typecheck` (server): passed.
- `pnpm verify` on #21705 selected 2,738 of 3,565 files (close to the
full suite). I relied on the focused runs and required CI, per
AGENTS.md.
## Risks
- The batched latest-run statement must return the same run as the
single read. The equivalence test covers native and legacy bindings,
ties, other agents and other companies.
- `check()` no longer reads the run state again right after an
acknowledgement. A request that is still owed keeps the deadline from
its own row and is read in full at its next check. The acknowledgement
still revalidates everything before it writes.
- A database error still rejects the whole page, as before. Only the 403
provenance decision is isolated per request.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL tests) and
extended reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
131 lines
8.8 KiB
TypeScript
131 lines
8.8 KiB
TypeScript
import { readGitHubResponseRuns, type GitHubResponseRunRead } from "./response-run-state.js";
|
|
import { githubRunReplyState } from "../../chat-run-publications.js";
|
|
import { and, asc, eq, gt, isNotNull, or, sql } from "drizzle-orm";
|
|
import { chatActions, chatConversations, chatEndpointLeases, chatEndpointResources,
|
|
chatPublications, fastResponseRequests, chatEndpoints, chatMessageLinks, type Db } from "@paperclipai/db";
|
|
import type { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js";
|
|
import { DELIVERY_QUEUES } from "../../delivery-work-notifications.js";
|
|
import { registerScopedDeadlineWork } from "../../scoped-deadline-work.js";
|
|
import { githubResponseReconstructionIntervalMs, type GitHubResponseSweepScope as Scope } from "./response-sweep-scope.js";
|
|
type Entry = { id: string; companyId: string; endpointId: string; deliveryId: string;
|
|
issueId: string | null; conversationId: string | null; repositoryId: string | null; number: number | null; dueAt: number };
|
|
type FastResponse = { status: string; expiresAt: string; publicationState: string | null };
|
|
|
|
export function registerGitHubAcknowledgementWork(
|
|
db: Db,
|
|
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
|
|
acknowledge: (deliveryId: string, canContinue: () => boolean) => Promise<unknown>,
|
|
canRun: () => boolean,
|
|
) {
|
|
const repositoryId = sql<string | null>`${chatEndpointResources.metadata}->>'providerRepositoryId'`;
|
|
const number = sql<string | null>`substring(${chatConversations.externalThreadId}
|
|
from '^github:[^:]+:(?:issue:)?([1-9][0-9]*)(?::rc:[1-9][0-9]*)?$')`;
|
|
const selection = {
|
|
id: chatActions.id, companyId: chatActions.companyId, endpointId: chatActions.endpointId,
|
|
deliveryId: chatActions.deliveryId, conversationId: chatActions.conversationId, issueId: chatConversations.issueId,
|
|
agentId: chatEndpoints.assignedAgentId,
|
|
sourceMessageId: sql<string | null>`(select m.comment_id from ${chatMessageLinks} m
|
|
where m.company_id = ${chatActions.companyId} and m.endpoint_id = ${chatActions.endpointId}
|
|
and m.delivery_id = ${chatActions.deliveryId} and m.direction = 'inbound' limit 1)`,
|
|
repositoryId, number, status: chatActions.status, result: chatActions.result,
|
|
leaseExpiresAt: sql<string | null>`(select max(l.expires_at) from ${chatEndpointLeases} l
|
|
where l.company_id = ${chatActions.companyId} and l.endpoint_id = ${chatActions.endpointId}
|
|
and l.lease_key = 'github-publication:' || (${repositoryId}) || ':' || ${number})`,
|
|
fast: sql<FastResponse | null>`(select jsonb_build_object('status', f.status, 'expiresAt', f.expires_at,
|
|
'publicationState', p.state) from ${fastResponseRequests} f left join ${chatPublications} p
|
|
on p.company_id = f.company_id and p.comment_id = f.comment_id
|
|
and p.endpoint_id = f.endpoint_id and p.conversation_id = f.conversation_id
|
|
where f.company_id = ${chatActions.companyId} and f.endpoint_id = ${chatActions.endpointId}
|
|
and f.delivery_id = ${chatActions.deliveryId} limit 1)`,
|
|
};
|
|
async function page(scope?: Scope, cursor?: string, actionId?: string) {
|
|
return db.select(selection).from(chatActions)
|
|
.innerJoin(chatEndpoints, and(eq(chatEndpoints.id, chatActions.endpointId), eq(chatEndpoints.companyId, chatActions.companyId)))
|
|
.leftJoin(chatConversations, and(eq(chatConversations.id, chatActions.conversationId), eq(chatConversations.companyId, chatActions.companyId)))
|
|
.leftJoin(chatEndpointResources, and(eq(chatEndpointResources.id, chatConversations.resourceId), eq(chatEndpointResources.companyId, chatActions.companyId)))
|
|
.where(and(eq(chatActions.kind, "github_response_comment"), isNotNull(chatActions.deliveryId),
|
|
sql`coalesce(${chatActions.result}->>'final', 'false') <> 'true'`,
|
|
or(eq(chatActions.status, "received"), and(eq(chatActions.status, "failed"), sql`${chatActions.result}->>'retryable' = 'true'`)),
|
|
scope ? eq(chatActions.companyId, scope.companyId) : undefined,
|
|
scope?.issueId ? eq(chatConversations.issueId, scope.issueId) : undefined,
|
|
scope?.conversationId ? eq(chatActions.conversationId, scope.conversationId) : undefined,
|
|
scope?.deliveryId ? eq(chatActions.deliveryId, scope.deliveryId) : undefined,
|
|
scope?.endpointId ? eq(chatActions.endpointId, scope.endpointId) : undefined,
|
|
scope?.repositoryId ? eq(repositoryId, scope.repositoryId) : undefined,
|
|
scope?.number ? eq(number, String(scope.number)) : undefined,
|
|
cursor ? gt(chatActions.id, cursor) : undefined,
|
|
actionId ? eq(chatActions.id, actionId) : undefined,
|
|
)).orderBy(asc(chatActions.id)).limit(100);
|
|
}
|
|
type Row = Awaited<ReturnType<typeof page>>[number];
|
|
async function entry(row: Row, read: GitHubResponseRunRead): Promise<Entry | null> {
|
|
// A request whose requester provenance changed stays due, so the
|
|
// acknowledgement owner revokes it. It must not stop the rest of the page.
|
|
if (!("forbidden" in read)) {
|
|
const { latestRun, ownsLatestRequest } = read;
|
|
// Execution success alone does not decide whether a final response exists.
|
|
// Presentation/publication commits invalidate this parked request.
|
|
if (row.issueId && ownsLatestRequest && latestRun?.nativePresentationPending &&
|
|
await githubRunReplyState(db, { companyId: row.companyId, endpointId: row.endpointId,
|
|
issueId: row.issueId, runId: latestRun.id }) === "none") return null;
|
|
}
|
|
return schedule(row);
|
|
}
|
|
/** The entries of a page, by position, from one batched run-state read. */
|
|
async function entries(rows: Row[]) {
|
|
const reads = await readGitHubResponseRuns(db, rows.map(row => ({ companyId: row.companyId, endpointId: row.endpointId,
|
|
agentId: row.agentId, issueId: row.issueId, deliveryId: row.deliveryId!, sourceMessageId: row.sourceMessageId })));
|
|
const values: Array<Entry | null> = [];
|
|
for (const [index, row] of rows.entries()) values.push(await entry(row, reads[index]!));
|
|
return values;
|
|
}
|
|
async function current(actionId: string) {
|
|
const [row] = await page(undefined, undefined, actionId);
|
|
return row ? (await entries([row]))[0] ?? null : null;
|
|
}
|
|
/** When the request is due, from its own row. */
|
|
function schedule(row: Row): Entry | null {
|
|
let dueAt = row.status === "failed" ? Date.parse(String(row.result?.retryAt)) : 0;
|
|
if (!Number.isFinite(dueAt)) throw new Error("GitHub acknowledgement has an invalid retry deadline");
|
|
const fast = row.fast;
|
|
if (fast?.publicationState === "streaming") return null;
|
|
if (fast && !["published", "delivery_unknown"].includes(fast.publicationState ?? "") &&
|
|
(["pending", "running"].includes(fast.status) || ["pending", "retry"].includes(fast.publicationState ?? ""))) {
|
|
dueAt = Math.max(dueAt, Date.parse(fast.expiresAt));
|
|
}
|
|
if (row.leaseExpiresAt) dueAt = Math.max(dueAt, Date.parse(row.leaseExpiresAt));
|
|
return { id: row.id, companyId: row.companyId, endpointId: row.endpointId, deliveryId: row.deliveryId!,
|
|
issueId: row.issueId, conversationId: row.conversationId, repositoryId: row.repositoryId, number: row.number ? Number(row.number) : null, dueAt };
|
|
}
|
|
return registerScopedDeadlineWork(coordinator, {
|
|
queue: DELIVERY_QUEUES.chatGitHubAcknowledgements,
|
|
reconstructionIntervalMs: githubResponseReconstructionIntervalMs,
|
|
decodeScope: (hint: string): Scope | undefined => hint === "*" ? undefined : JSON.parse(hint),
|
|
matches: (value: Entry, scope?: Scope) => !scope || (value.companyId === scope.companyId &&
|
|
(!scope.issueId || value.issueId === scope.issueId) && (!scope.deliveryId || value.deliveryId === scope.deliveryId) &&
|
|
(!scope.conversationId || value.conversationId === scope.conversationId) &&
|
|
(!scope.endpointId || value.endpointId === scope.endpointId) && (!scope.repositoryId || value.repositoryId === scope.repositoryId) &&
|
|
(!scope.number || value.number === scope.number)),
|
|
load: async (scope?: Scope, signal?: AbortSignal) => {
|
|
const loaded: Entry[] = [];
|
|
let cursor: string | undefined;
|
|
while (canRun() && !signal?.aborted) {
|
|
const rows = await page(scope, cursor);
|
|
for (const value of await entries(rows)) if (value) loaded.push(value);
|
|
if (rows.length < 100) return loaded;
|
|
cursor = rows[rows.length - 1].id;
|
|
}
|
|
throw signal?.reason ?? new Error("GitHub acknowledgement recovery stopped");
|
|
},
|
|
check: async (value: Entry, signal?: AbortSignal) => {
|
|
const due = await current(value.id);
|
|
if (!due || due.dueAt > Date.now() || !canRun() || signal?.aborted) return due;
|
|
await acknowledge(due.deliveryId, () => canRun() && !signal?.aborted);
|
|
// A request that is still owed keeps its saved deadline. The next check
|
|
// reads its run state again before it acknowledges.
|
|
const [remaining] = await page(undefined, undefined, value.id);
|
|
return remaining ? schedule(remaining) : null;
|
|
},
|
|
}, canRun);
|
|
}
|