From ea8c734cb9968cb190138237084324105120224c Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 20:40:39 -0600 Subject: [PATCH] fix(chat): park GitHub terminal responses that wait for a newer run or task change (#21705) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The GitHub chat channel posts one App-owned response comment per accepted request. The terminal-response sweep (`terminal-response-work.ts`) finds processed, non-final responses whose task's latest run has ended, and asks the acknowledgement owner to post the final text. > - On paperclip.staging, 1,557 processed `github_response_comment` rows are not final. None changed in the last 5 minutes, and the oldest dates from 2026-10-28. The sweep reads all of them again on each company-wide rebuild (every 30 s per company), with one page read plus three statements per row. Two of those statements run at about 13/s each in `pg_stat_statements`. > - The sweep cannot make progress on these rows. Their task's latest run ended without owning the request (a later request started it), or the pause is already shown. Nothing that the decision reads changes until a new run starts, the task status changes, or the row changes. > - This pull request records that wait on the row and skips it on broad rebuilds until one of those inputs changes. > - The benefit is that an idle backlog costs one page read per rebuild instead of about three statements per row, and no request loses a path to settle. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On paperclip.staging (2026-11-15, about 01:45–02:25Z), two single-row statements ran at nearly the same rate: - `select id, run_id, company_id, status, responsible_user_id, message_id, parent_context_id from run_identity_contexts where id = $1 and run_id = $2 and company_id = $3 and status = $4 limit $5` at 13.37/s. - `select chat_conversations.issue_id, chat_endpoints.provider from chat_message_links join chat_conversations … join chat_endpoints … where chat_message_links.company_id = $3 and comment_id = $4 …` at 13.20/s. They are `provenanceReads.context` and `provenanceReads.link` in `run-origin.ts`. The terminal-response sweep reaches them through `readGitHubResponseRun` for each row it loads. The backlog of `github_response_comment` actions on staging: - `processed`, `delivery_id` set, `result.final` not `true`: 1,557 rows (the terminal sweep's set). None updated in the last 5 minutes. The oldest `updated_at` is 2026-10-28. - `result.code` of those rows: null 1,387, `acknowledgement_unconfirmed` 150, `acknowledgement_superseded` 20. - The acknowledgement set (`received`, or `failed` and retryable) is empty. - Other rows: processed and final 6,645, failed and not retryable 265, cancelled 116. **Root cause** `entry()` returns `null` ("nothing to post now") when `githubResponseExecution` has no body. That happens when the task's latest run has ended but does not own the request and the task is not blocked. It also happens when the task is blocked and the row already shows this pause (`pausedVersion` matches). The page query still selects these rows, because the latest run has ended (or the task is blocked) and the row is not final. Nothing records the decision, so each company-wide rebuild reads every row again: `readGitHubResponseRun` (one statement) plus `chatRunSourceMessageId` (the context and link reads) for each row. The codes do not matter. `acknowledgement_unconfirmed` is a retry code that a later successful acknowledgement keeps, and `acknowledgement_superseded` marks a request whose shared PR comment has a newer version. All three groups wait for the same thing: a newer run that owns the request, or a task status change. **Expected behavior** A response that cannot progress costs nothing until the run, the task or the response changes. A response that can progress settles as before. **Steps to reproduce** 1. Create a GitHub endpoint and a task with several processed, non-final response comments (codes null, `acknowledgement_unconfirmed`, `acknowledgement_superseded`). 2. Give the task a newer ended run whose identity context names a later request, so the run does not own the older requests. 3. Register the terminal-response sweep and send a company-wide hint every 30 s. 4. On master, each rebuild reads every row again with about three statements per row and writes nothing. **Paperclip version or commit** master at 958eda6f16. ## What Changed - `server/src/services/chat-channels/github/terminal-response-work.ts`: - The page query reads the task's latest run through one lateral subquery. The subquery returns the run status (for the existing filter) and a run version (id, status, `updated_at`). The run-index union and its filters are unchanged. - A wait key joins the run version, the task status and the row's `updated_at`. - When `entry()` decides that a row has nothing to post, the sweep records `result.terminalWaitKey` in one batched update per page. The update writes only if the row has not changed since the read, and it leaves `updated_at` alone, because the key includes it. - A broad load (every company, or one company with no narrower scope) skips rows whose stored key equals the current key. Targeted loads (task, delivery, conversation, endpoint, repository or number) and `check()` still read every matching row. - Only waits that do not depend on the owning run's reply are recorded: the latest run does not own the request, or the pause is already shown. A request that waits for its owning run's reply publication or native presentation is never parked, because a publication commit changes none of the key's inputs and a restart can lose its lease-release hint. Those requests stay eligible for every broad load, as on master. - `check()` records the key too when its revalidation finds nothing to post. - `server/src/services/chat-channels/github/terminal-response-work.test.ts` (new, real PostgreSQL), with staging-shaped fixtures: 30 tasks with 4 waiting responses each (codes null 89%, `acknowledgement_unconfirmed` 10%, `acknowledgement_superseded` 1%), and a newer ended run per task that answered a later request: - `stops re-reading processed, non-final responses that cannot progress on every company rebuild` - `still settles a waiting response once a run that owns it ends` - `still pauses waiting responses once their task is blocked` - `still settles a response after a restart lost the hint of the reply publication it waited for` (an owning run's formal review is `processing`; it commits with no hint; after a coordinator restart the broad load acknowledges the request) - `server/src/__tests__/chat-channels.integration.test.ts`: `finds owed terminal GitHub work beyond a full page of unrelated request history` now expects the 101 retained requests to carry a wait key in addition to their code. No GitHub comment is edited, posted or deleted by this change. User-visible behavior is the same: a waiting request keeps its current comment until a run that owns it ends or its task is blocked, as before. ## Verification - `pnpm vitest run src/services/chat-channels/github/terminal-response-work.test.ts` (server): - master: 2 of 3 fail (red). The first pass costs 362 statements for 120 rows (2 page reads + 3 per row). Four company rebuilds cost 1,448 statements, with 480 run identity reads, and write nothing. - this branch: 4 of 4 pass (green). The first pass costs 364 statements (the same reads plus 2 batched wait-key writes). Four company rebuilds cost 4 statements (one page read each) and 0 run identity reads. An owning run and a blocked task still settle their requests. - The restart test fails on the first revision of this PR (no acknowledgement) and passes now. - `pnpm vitest run src/services/chat-channels/github/terminal-response-work.test.ts src/__tests__/chat-channels.integration.test.ts -t "terminal|acknowledg|GitHub response|…"`: 193 passed. - `pnpm typecheck` (server): passed. - `pnpm verify` selected 2,738 of 3,565 test files ("retained conservatively"), close to the full suite. I stopped it and relied on the focused runs above plus required CI, per AGENTS.md. Before I stopped it, two unrelated files failed locally: `agent-commentary-skills.test.ts` (a local skills file is missing, ENOENT) and one test in `agent-lifecycle/lifecycle.test.ts` (it passes in isolation). ## Risks - The wait key must change whenever a skipped row could progress. Its inputs are the latest run (id, status, `updated_at`), the task status and the row version. Run writers set `updated_at`, and the publication lease sends a targeted hint that bypasses the key. A missed input delays a final comment until the run, the task or the row next changes. It does not lose the request. - The first rebuild after deploy still reads the whole backlog once, and writes one key update per page. The next PR batches those reads. - `result` gains one internal key, `terminalWaitKey`. Writers that copy `result` forward also set `updated_at`, which invalidates a copied key. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL tests) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Related: #21674, #21675, #21676 (same class: a read loop that makes no progress). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../chat-channels.integration.test.ts | 3 +- .../github/terminal-response-work.test.ts | 206 ++++++++++++++++++ .../github/terminal-response-work.ts | 79 +++++-- 3 files changed, 269 insertions(+), 19 deletions(-) create mode 100644 server/src/services/chat-channels/github/terminal-response-work.test.ts diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index a9aad9d63fa..5c6e071c913 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -4510,9 +4510,10 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { await expect.poll(() => [...f.responseComments.values()][0]?.body, { timeout: 5000 }).toContain("stopped before completing"); await expect.poll(() => coordinator.nextWakeAt(), { timeout: 5000 }).toBeNull(); expect(f.responseWrites).toHaveLength(baseline + 1); + // The retained requests only record that they wait for a newer run. expect(await db.select().from(chatActions).where(inArray(chatActions.id, history.map(row => row.actionId)))) .toEqual(expect.arrayContaining(history.map(row => expect.objectContaining({ id: row.actionId, - status: "processed", result: { code: "fast_response_superseded" } })))); + status: "processed", result: { code: "fast_response_superseded", terminalWaitKey: expect.any(String) } })))); expect(errors).toEqual([]); } finally { await coordinator.stop(); diff --git a/server/src/services/chat-channels/github/terminal-response-work.test.ts b/server/src/services/chat-channels/github/terminal-response-work.test.ts new file mode 100644 index 00000000000..cd28076b6f2 --- /dev/null +++ b/server/src/services/chat-channels/github/terminal-response-work.test.ts @@ -0,0 +1,206 @@ +import { randomUUID } from "node:crypto"; +import { setTimeout as delay } from "node:timers/promises"; +import { and, eq, inArray, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, chatActions, companies, createDb, heartbeatRuns, issues, runIdentityContexts, type Db } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js"; +import { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../delivery-work-notifications.js"; +import { idleWorkSnapshot } from "../../task-admission.js"; +import { registerGitHubTerminalResponseWork } from "./terminal-response-work.js"; + +const support = await getEmbeddedPostgresTestSupport(); + +// Staging (2026-11-15) held 1,557 processed, non-final response comments whose +// result codes were null (89%), acknowledgement_unconfirmed (10%) and +// acknowledgement_superseded (1%). Each task's latest run had ended and had +// been started by a later GitHub request, so it did not own these requests. +const ISSUES = 30; +const STALE_PER_ISSUE = 4; +const STALE = ISSUES * STALE_PER_ISSUE; +const code = (index: number) => index % 50 === 1 ? "acknowledgement_superseded" + : index % 10 === 0 ? "acknowledgement_unconfirmed" : null; + +describe.skipIf(!support.supported)("GitHub terminal response sweep backlog", () => { + let temporary: Awaited>; + let db: Db; + let coordinator: ReturnType | undefined; + const errors: unknown[] = []; + const acknowledged: string[] = []; + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("github-terminal-response-work-"); + db = createDb(temporary.connectionString); + }, 60_000); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; acknowledged.length = 0; + vi.useRealTimers(); + expect(errors.splice(0)).toEqual([]); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + + type Request = { issueId: string; deliveryId: string; actionId: string; commentId: string }; + async function seed() { + const client = db.$client; + const companyId = randomUUID(), agentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "GitHub backlog", issuePrefix: `G${companyId.slice(0, 7)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Reviewer" }); + const [{ application }] = await client.unsafe(`insert into tool_applications (company_id, application_key, name, type) + values ($1, 'github', 'GitHub', 'chat') returning id as application`, [companyId]); + const [{ connection }] = await client.unsafe(`insert into tool_connections (company_id, application_id, name, uid, transport, connection_purpose) + values ($1, $2, 'GitHub', $3, 'chat_sdk', 'channel') returning id as connection`, [companyId, application, `github-${companyId}`]); + const [{ endpoint }] = await client.unsafe(`insert into chat_endpoints (company_id, connection_id, provider, public_id, assigned_agent_id, status) + values ($1, $2, 'github', $3, $4, 'active') returning id as endpoint`, [companyId, connection, `backlog-${companyId}`, agentId]); + const stale: Request[] = []; + const runs: Array<{ issueId: string; runId: string }> = []; + const base = Date.UTC(2026, 9, 28); + for (let issueNumber = 1; issueNumber <= ISSUES; issueNumber++) { + const [{ issue }] = await client.unsafe(`insert into issues (company_id, title, status, assignee_agent_id) + values ($1, $2, 'in_progress', $3) returning id as issue`, [companyId, `Pull ${issueNumber}`, agentId]); + const [{ conversation }] = await client.unsafe(`insert into chat_conversations (company_id, endpoint_id, issue_id, + external_conversation_id, external_label, external_thread_id) + values ($1, $2, $3, $4, $4, $4) returning id as conversation`, [companyId, endpoint, issue, `github:paperclipai/paperclip:${issueNumber}`]); + const requests: Request[] = []; + for (let request = 0; request <= STALE_PER_ISSUE; request++) { + const [{ delivery }] = await client.unsafe(`insert into chat_deliveries (company_id, endpoint_id, provider_event_id, + deduplication_key, event_kind, normalized_event, conversation_id, state) + values ($1, $2, $3, $3, 'message', '{}', $4, 'processed') returning id as delivery`, + [companyId, endpoint, `event-${companyId}-${issueNumber}-${request}`, conversation]); + const [{ comment }] = await client.unsafe(`insert into issue_comments (company_id, issue_id, body) + values ($1, $2, 'Please review') returning id as comment`, [companyId, issue]); + await client.unsafe(`insert into chat_message_links (company_id, endpoint_id, conversation_id, delivery_id, comment_id, + provider_message_id, direction) values ($1, $2, $3, $4, $5, $6, 'inbound')`, + [companyId, endpoint, conversation, delivery, comment, `in-${companyId}-${issueNumber}-${request}`]); + const index = stale.length; + const latest = request === STALE_PER_ISSUE; + const [action] = await db.insert(chatActions).values({ companyId, endpointId: endpoint, conversationId: conversation, + deliveryId: delivery, kind: "github_response_comment", providerActionId: `github_response:${delivery}`, + payload: { agentId }, status: "processed", + result: { id: String(1_000 + index), url: `https://github.com/paperclipai/paperclip/pull/${issueNumber}#c${index}`, + final: latest, versionAt: new Date(base).toISOString(), ...(latest ? {} : { code: code(index) }) }, + updatedAt: new Date(base + index * 60_000) }).returning(); + requests.push({ issueId: issue, deliveryId: delivery, actionId: action!.id, commentId: comment }); + } + stale.push(...requests.slice(0, STALE_PER_ISSUE)); + // The latest run answered the newest request, through its identity context. + const runId = randomUUID(), contextId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue, issueId: issue, + scopeKind: "issue", status: "succeeded", responsibleUserId: "requester", activeIdentityContextId: contextId, + contextSnapshot: { issueId: issue }, createdAt: new Date(base + issueNumber * 3_600_000) }); + await db.insert(runIdentityContexts).values({ id: contextId, companyId, runId, revision: 1, responsibleUserId: "requester", + messageId: requests[STALE_PER_ISSUE]!.commentId, cause: "chat_request", correlationId: runId, status: "accepted" }); + runs.push({ issueId: issue, runId }); + } + await db.execute(sql`analyze`); + return { companyId, agentId, endpointId: endpoint as string, stale, runs }; + } + + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, + onError: error => errors.push(error) }); + const worker = registerGitHubTerminalResponseWork(db, coordinator, async deliveryId => { + acknowledged.push(deliveryId); + // Stand in for the real acknowledgement, which records a final response. + await db.update(chatActions).set({ result: sql`${chatActions.result} || '{"final": true}'::jsonb`, updatedAt: new Date() }) + .where(and(eq(chatActions.deliveryId, deliveryId), eq(chatActions.kind, "github_response_comment"))); + }, () => true); + await worker.ready; + } + async function advance(ms: number) { + const until = Date.now() + ms; + while (Date.now() < until) { + await vi.advanceTimersByTimeAsync(Math.min(1_000, until - Date.now())); await delay(1); + while (idleWorkSnapshot().active > 0) await delay(5); + } + } + /** Company-wide hints, as run lifecycle changes elsewhere in the company send them. */ + async function companyHint(companyId: string) { + await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubTerminalResponses, JSON.stringify({ companyId })); + }); + } + async function observe(work: () => Promise) { + const statements: string[] = [], prior = db.$client.options.debug; + db.$client.options.debug = (_connection, query) => { statements.push(query); }; + try { await work(); } finally { db.$client.options.debug = prior; } + const sweep = statements.filter(query => /"chat_actions"|heartbeat_runs|run_identity_contexts|chat_message_links/.test(query)); + return { + statements: sweep.length, + provenance: sweep.filter(query => /from "run_identity_contexts"/.test(query)).length, + }; + } + const finals = async (actionIds: string[]) => (await db.select({ result: chatActions.result }).from(chatActions) + .where(inArray(chatActions.id, actionIds))).filter(row => row.result?.final === true).length; + + it("stops re-reading processed, non-final responses that cannot progress on every company rebuild", async () => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const f = await seed(); + const first = await observe(async () => { await start(); await advance(5_000); }); + expect(acknowledged).toEqual([]); + const rebuilds = await observe(async () => { + for (let pass = 0; pass < 4; pass++) { await companyHint(f.companyId); await advance(31_000); } + }); + console.info(`terminal response sweep: first pass ${first.statements} statements (${first.provenance} run identity reads); ` + + `4 company rebuilds ${rebuilds.statements} statements (${rebuilds.provenance} run identity reads) for ${STALE} rows`); + // Nothing these rows wait for changed, so a rebuild reads none of them. + expect(rebuilds.provenance).toBe(0); + expect(rebuilds.statements).toBeLessThanOrEqual(4 * 2); + expect(acknowledged).toEqual([]); + // No GitHub comment changed and no row was finalized. + expect(await finals(f.stale.map(request => request.actionId))).toBe(0); + }, 300_000); + + it("still settles a waiting response once a run that owns it ends", async () => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const f = await seed(); + await start(); await advance(5_000); + const target = f.stale[5]!; + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, nativeIssueId: target.issueId, + issueId: target.issueId, scopeKind: "issue", status: "failed", + contextSnapshot: { issueId: target.issueId, wakeCommentId: target.commentId }, createdAt: new Date() }); + await companyHint(f.companyId); + await advance(31_000); + expect(acknowledged).toEqual([target.deliveryId]); + const rebuilds = await observe(async () => { await companyHint(f.companyId); await advance(31_000); }); + expect(rebuilds.provenance).toBe(0); + expect(acknowledged).toEqual([target.deliveryId]); + }, 300_000); + + it("still settles a response after a restart lost the hint of the reply publication it waited for", async () => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const f = await seed(); + const target = f.stale[9]!; + const [response] = await db.select().from(chatActions).where(eq(chatActions.id, target.actionId)); + const runId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId, nativeIssueId: target.issueId, + issueId: target.issueId, scopeKind: "issue", status: "failed", + contextSnapshot: { issueId: target.issueId, wakeCommentId: target.commentId }, createdAt: new Date() }); + // The owning run's formal review is still being published. + const [publication] = await db.insert(chatActions).values({ companyId: f.companyId, endpointId: f.endpointId, + conversationId: response!.conversationId, deliveryId: target.deliveryId, kind: "github_review_publication", + providerActionId: `github_review:${runId}`, status: "processing", + payload: { operation: "formal_review", session: { companyId: f.companyId, issueId: target.issueId, runId, agentId: f.agentId } } }) + .returning(); + await start(); await advance(5_000); + expect(acknowledged).toEqual([]); + // The publication commits, and the server stops before its lease release + // sends the targeted hint. The next start runs only a broad load. + await db.update(chatActions).set({ status: "processed", updatedAt: new Date(), + result: { id: "77", url: "https://github.com/paperclipai/paperclip/pull/1#pullrequestreview-77" } }) + .where(eq(chatActions.id, publication!.id)); + await coordinator!.stop(); coordinator = undefined; + await start(); await advance(5_000); + expect(acknowledged).toEqual([target.deliveryId]); + }, 300_000); + + it("still pauses waiting responses once their task is blocked", async () => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const f = await seed(); + await start(); await advance(5_000); + const blocked = f.runs[2]!.issueId; + await db.update(issues).set({ status: "blocked", updatedAt: new Date() }).where(eq(issues.id, blocked)); + await companyHint(f.companyId); + await advance(31_000); + expect(new Set(acknowledged)).toEqual(new Set(f.stale.filter(request => request.issueId === blocked).map(request => request.deliveryId))); + }, 300_000); +}); diff --git a/server/src/services/chat-channels/github/terminal-response-work.ts b/server/src/services/chat-channels/github/terminal-response-work.ts index fd60d7538e5..49c8447c25a 100644 --- a/server/src/services/chat-channels/github/terminal-response-work.ts +++ b/server/src/services/chat-channels/github/terminal-response-work.ts @@ -10,6 +10,10 @@ import { HttpError } from "../../../errors.js"; type Entry = { id: string; companyId: string; endpointId: string; deliveryId: string; issueId: string; conversationId: string; repositoryId: string | null; number: number | null; dueAt: number }; +/** A load of a whole company (or every company) rather than a targeted hint. */ +const broad = (scope?: Scope) => !scope || (!scope.issueId && !scope.deliveryId && !scope.conversationId && + !scope.endpointId && !scope.repositoryId && !scope.number); + export function registerGitHubTerminalResponseWork( db: Db, coordinator: ReturnType, @@ -19,28 +23,37 @@ export function registerGitHubTerminalResponseWork( const repositoryId = sql`${chatEndpointResources.metadata}->>'providerRepositoryId'`; const number = sql`substring(${chatConversations.externalThreadId} from '^github:[^:]+:(?:issue:)?([1-9][0-9]*)(?::rc:[1-9][0-9]*)?$')`; - // Status of the assigned agent's latest run bound to the issue, where a run's - // issue is coalesce(native_issue_id, context_snapshot->>'issueId'). The two + // The assigned agent's latest run bound to the issue, where a run's issue is + // coalesce(native_issue_id, context_snapshot->>'issueId'). The two // precedence branches are disjoint and each uses an issue index, so a // candidate does not walk (and detoast) the agent's whole run history. - const latestIssueRunStatus = sql`select latest.status from ( - (select r.status, r.created_at, r.id from ${heartbeatRuns} r + const latestIssueRun = db.select({ + status: sql`latest.status`.as("latest_run_status"), + version: sql`latest.id::text || ':' || latest.status || ':' || extract(epoch from latest.updated_at)::text` + .as("latest_run_version"), + }).from(sql`((select r.status, r.created_at, r.id, r.updated_at from ${heartbeatRuns} r where r.company_id = ${chatActions.companyId} and r.native_issue_id = ${chatConversations.issueId} and r.agent_id = ${chatEndpoints.assignedAgentId} order by r.created_at desc, r.id desc limit 1) union all - (select r.status, r.created_at, r.id from ${heartbeatRuns} r + (select r.status, r.created_at, r.id, r.updated_at from ${heartbeatRuns} r where r.company_id = ${chatActions.companyId} and r.native_issue_id is null and r.context_snapshot->>'issueId' = ${chatConversations.issueId}::text and r.agent_id = ${chatEndpoints.assignedAgentId} - order by r.created_at desc, r.id desc limit 1) - ) latest order by latest.created_at desc, latest.id desc limit 1`; + order by r.created_at desc, r.id desc limit 1)) latest`) + .orderBy(sql`latest.created_at desc, latest.id desc`).limit(1).as("latest_issue_run"); + // A request whose latest run does not own it (or whose pause is already + // shown) can progress only once the task's latest run, the task's status or + // the request's own row changes. Such a request records this key, and a broad + // rebuild skips it until one of them changes. A targeted load still reads it. + const actionVersion = sql`extract(epoch from ${chatActions.updatedAt})::text`; + const waitKey = sql`coalesce(${latestIssueRun.version}, 'none') || '|' || ${issues.status} || '|' || ${actionVersion}`; async function page(scope?: Scope, cursor?: string, actionId?: string) { return db.select({ id: chatActions.id, companyId: chatActions.companyId, endpointId: chatActions.endpointId, deliveryId: chatActions.deliveryId, conversationId: chatConversations.id, issueId: issues.id, agentId: chatEndpoints.assignedAgentId, issueStatus: issues.status, - deliveryCreatedAt: chatDeliveries.createdAt, repositoryId, number, result: chatActions.result, + deliveryCreatedAt: chatDeliveries.createdAt, repositoryId, number, result: chatActions.result, actionVersion, waitKey, sourceMessageId: sql`(select l.comment_id from ${chatMessageLinks} l where l.company_id = ${chatActions.companyId} and l.endpoint_id = ${chatActions.endpointId} and l.delivery_id = ${chatActions.deliveryId} and l.direction = 'inbound' limit 1)`, @@ -53,10 +66,12 @@ export function registerGitHubTerminalResponseWork( .innerJoin(chatDeliveries, and(eq(chatDeliveries.id, chatActions.deliveryId), eq(chatDeliveries.companyId, chatActions.companyId))) .innerJoin(issues, and(eq(issues.id, chatConversations.issueId), eq(issues.companyId, chatActions.companyId))) .leftJoin(chatEndpointResources, and(eq(chatEndpointResources.id, chatConversations.resourceId), eq(chatEndpointResources.companyId, chatActions.companyId))) + .leftJoinLateral(latestIssueRun, sql`true`) .where(and(eq(chatActions.kind, "github_response_comment"), eq(chatActions.status, "processed"), isNotNull(chatActions.deliveryId), isNotNull(chatEndpoints.assignedAgentId), sql`coalesce(${chatActions.result}->>'final', 'false') <> 'true'`, - sql`(${issues.status} = 'blocked' or (${latestIssueRunStatus}) in ('succeeded', 'failed', 'interrupted', 'timed_out', 'cancelled'))`, + sql`(${issues.status} = 'blocked' or ${latestIssueRun.status} in ('succeeded', 'failed', 'interrupted', 'timed_out', 'cancelled'))`, + broad(scope) && !actionId ? sql`${chatActions.result}->>'terminalWaitKey' is distinct from ${waitKey}` : undefined, scope ? eq(chatActions.companyId, scope.companyId) : undefined, scope?.issueId ? eq(issues.id, scope.issueId) : undefined, scope?.conversationId ? eq(chatActions.conversationId, scope.conversationId) : undefined, @@ -67,14 +82,20 @@ export function registerGitHubTerminalResponseWork( cursor ? gt(chatActions.id, cursor) : undefined, actionId ? eq(chatActions.id, actionId) : undefined, )).orderBy(asc(chatActions.id)).limit(100); } - async function entry(row: Awaited>[number]): Promise { + type Row = Awaited>[number]; + /** The request's entry when it is owed now, else whether its wait may be + * recorded. A wait on the owning run's reply or presentation is never + * recorded: a publication that commits it changes no wait-key input, and a + * restart can lose the hint that its lease release sends. */ + async function entry(row: Row): Promise { const pending: Entry = { id: row.id, companyId: row.companyId, endpointId: row.endpointId, deliveryId: row.deliveryId!, issueId: row.issueId, conversationId: row.conversationId, repositoryId: row.repositoryId, number: row.number ? Number(row.number) : null, dueAt: row.leaseExpiresAt ? Date.parse(row.leaseExpiresAt) : 0 }; try { const execution = await githubResponseExecution(db, { ...row, agentId: row.agentId!, deliveryId: row.deliveryId! }); if (execution.body === undefined || execution.reply === "unsettled" || - (execution.paused && row.result?.pausedVersion === execution.version)) return null; + (execution.paused && row.result?.pausedVersion === execution.version)) + return { parkable: !execution.presentationPending && execution.reply === "none" }; } catch (error) { if (!(error instanceof HttpError) || error.status !== 403) throw error; // Keep this request eligible for the acknowledgement owner's authority @@ -82,6 +103,27 @@ export function registerGitHubTerminalResponseWork( } return pending; } + /** Record the wait of requests that cannot progress yet. A request whose row + * changed after the read gets no key and is read again. The write leaves + * updated_at alone, because the key includes it. */ + async function park(rows: Row[]) { + const waiting = rows.filter(row => row.result?.terminalWaitKey !== row.waitKey); + if (!waiting.length) return; + await db.execute(sql`update ${chatActions} + set result = coalesce(${chatActions.result}, '{}'::jsonb) || jsonb_build_object('terminalWaitKey', waiting.key) + from unnest(${sql.param(waiting.map(row => row.id))}::uuid[], ${sql.param(waiting.map(row => row.actionVersion))}::text[], + ${sql.param(waiting.map(row => row.waitKey))}::text[]) as waiting(id, version, key) + where ${chatActions.id} = waiting.id and ${actionVersion} = waiting.version + and ${chatActions.status} = 'processed' and coalesce(${chatActions.result}->>'final', 'false') <> 'true'`); + } + async function current(actionId: string) { + const [row] = await page(undefined, undefined, actionId); + if (!row) return null; + const value = await entry(row); + if (!("parkable" in value)) return value; + if (value.parkable) await park([row]); + return null; + } return registerScopedDeadlineWork(coordinator, { queue: DELIVERY_QUEUES.chatGitHubTerminalResponses, @@ -97,22 +139,23 @@ export function registerGitHubTerminalResponseWork( let cursor: string | undefined; while (canRun() && !signal?.aborted) { const rows = await page(scope, cursor); + const waiting: Row[] = []; for (const row of rows) { if (!canRun() || signal?.aborted) throw signal?.reason ?? new Error("GitHub terminal-response recovery stopped"); - const value = await entry(row); if (value) entries.push(value); + const value = await entry(row); + if (!("parkable" in value)) entries.push(value); else if (value.parkable) waiting.push(row); } + await park(waiting); if (rows.length < 100) return entries; cursor = rows[rows.length - 1].id; } throw signal?.reason ?? new Error("GitHub terminal-response recovery stopped"); }, check: async (value: Entry, signal?: AbortSignal) => { - const [row] = await page(undefined, undefined, value.id); - const current = row ? await entry(row) : null; - if (!current || current.dueAt > Date.now() || !canRun() || signal?.aborted) return current; - await acknowledge(current.deliveryId, () => canRun() && !signal?.aborted); - const [remaining] = await page(undefined, undefined, value.id); - return remaining ? entry(remaining) : null; + const due = await current(value.id); + if (!due || due.dueAt > Date.now() || !canRun() || signal?.aborted) return due; + await acknowledge(due.deliveryId, () => canRun() && !signal?.aborted); + return current(value.id); }, }, canRun); }