fix(server): wake GitHub acknowledgements from committed changes (#17656)

<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/206/changes/paperclip/206
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip coordinates agent work and presents its progress in chat.
> - GitHub deliveries save acknowledgement actions and provider
receipts.
> - Periodic maintenance currently searches for unfinished
acknowledgements even when idle.
> - Committed delivery, lease, response and presentation changes
identify when work can advance.
> - The existing owner should wake on those changes and recover only at
saved deadlines.
> - Completed history should become quiet without losing unfinished
provider effects.

## Linked Issues or Issue Description

Refs #17653, which is merged. This existing GitHub acknowledgement slice
targets master.

**Problem:** A saved GitHub acknowledgement can wait for a publication
lease, fast-response result or native presentation decision. Polling
empty or unchanged work keeps the tenant active. Removing maintenance
without covering these producers can strand the acknowledgement.

**Expected behavior:** Committed changes wake the scoped owner. It
reconstructs existing actions on startup, retries at saved deadlines,
parks unchanged presentation waits and becomes quiet when settled.

## What Changed

- Register GitHub acknowledgement work with the shared scoped deadline
coordinator. Reuse existing action, delivery, lease and fast-response
rows.
- Add transaction-aware response-availability notifications at supported
writes. Keep GitHub protocol details in its platform owner and shared
notifications limited to IDs.
- Share the bounded latest-request/run projection between scheduling and
direct response authorization. An absent native presentation decision
parks the action until its real writer commits.
- Retain publication leases, source/principal authority, version
fencing, response markers and final receipt precedence. Check admission
before each new provider request; let in-flight work settle. Stop does
not consume the provider retry budget.
- Remove only recurring acknowledgement recovery and keep its manual
path. Add no work table or scheduler service.
- Regenerate the three original indexes as
`0400_absent_phantom_reporter.sql`: pending acknowledgement IDs,
fast-response delivery lookup and publication comment/destination
lookup. Each uses concurrent/idempotent creation and the existing
top-level migration runner. Preserve all landed SQL/journal entries and
the five newest snapshots.

## Verification

Published head: `09ca00b9f029838434ec042cda691345d67b493e`, on landed
continuation parent `063cfd43813cabf18417605fb9eefb9aacf95c8d`.

- **87 distinct current cases pass:** 18 real PostgreSQL acknowledgement
lifecycle/presentation cases, all 26 fast-response service cases, all 30
heartbeat completion cases and all 13 startup cases. Database build,
server typechecking/build, module boundaries and three fresh migrations
through 0400 pass. The current Runner bundle rebuild also passes after
upstream provider-event changes.
- The lifecycle matrix covers empty/history quiet, lease release,
restart, drain/standby, fast-response failure/expiry/publication/unknown
outcome, streaming quarantine, outer rollback/commit, saved provider
retry, stop between requests and receipt state. The real presentation
writer proves quiet while pending, rollback, committed reactivation and
final quiet.
- The earlier **305-case** matrix (236 full GitHub workflow cases, 26
fast-response service cases, 30 completion cases and 13 startup cases)
passes at `337f272853d4f8a51b5b9300900a46eb4e13f698`. It retains its
prepared-parent scope. Repeated cases are not added to current 87.
- A populated 0399 database upgrades to 0400 with all 398 prior
migration receipts unchanged, one new receipt and no additional receipt
on replay. All three indexes are valid/ready on fresh and upgraded
databases. Their SQL definitions match the original draft after
whitespace normalization; migration history and retained snapshot bytes
are preserved.
- Normal integration `9b3607e159153e00ccdfe538d52b591bd38d6499`
preserves the full 28-path owned binary patch and all 20 unrelated
master paths, including native retry, provider-event and evaluation
workflow changes. Both normal merge parents and the exact tree are
checked. No force push or rebase occurs.
- [The owner
audit](https://github.com/paperclipai/paperclip/blob/09ca00b9f029838434ec042cda691345d67b493e/doc/plans/2026-10-25-github-chat-maintenance-sleep-audit.md#acceptance-on-the-actual-landed-continuation-parent)
retains the original missing-notification, JSON precedence, receipt
lifecycle, external-database fixture and migration-registry failures.
Earlier 69/228/broad counts keep their original scope. Skips and failed
revisions are not relabelled as current passes.
- The final audit-only edit leaves all owned source/test/migration
hashes unchanged. **Fresh exact-head CI and Greptile 5/5 with no
actionable findings remain required.** No full local repository, live
GitHub tenant or deployed sleep result is claimed.

## Risks

- Process-local transaction hints cover supported application writers
and shared pools. Independent SQL writers and other replicas do not gain
a new wake guarantee.
- Provider effects remain guarded by the existing publication lease and
current source/principal authority. Unknown outcomes retain their
existing recovery semantics.
- Presentation waits depend on their supported milestone/transaction
producers. A missing producer can defer recovery until another supported
change or restart.
- Index creation can take time on large histories and runs concurrently
outside a transaction. Existing migration recovery handles replay; prior
SQL is unchanged.
- Worker quiet is distinct from host/database suspension. Cloud wake
coverage and sleep admission remain unchanged; later maintenance duties
are separate slices.

## Model Used

OpenAI Codex, using reasoning, repository inspection, code execution and
real PostgreSQL tests. The exact model identifier and context-window
size are not exposed in this session.

## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-28 16:06:42 -05:00
1 parent 063cfd4381
commit ba5bba86dc
28 files changed
+1253 -111

No files matched your search

@@ -94,6 +94,10 @@ describe("createChatReconciliationCoordinator", () => {
expect(publicationMaintenance).not.toContain("await enqueueFailedChatRetryPublications(");
expect(publicationMaintenance).not.toContain("reconcileTerminalConfirmationActions(");
expect(source).toContain("chatChannels.registerInteractionContinuationWork(deliveryWork,");
expect(source).toContain("chatChannels.registerGitHubAcknowledgementWork(deliveryWork,");
const github = readFileSync(new URL("../services/chat-channels/github/maintenance.ts", import.meta.url), "utf8");
expect(github).not.toContain("githubResponseCommentService(db, fetchImpl).processPending()");
expect(github).toContain("githubResponseCommentService(db, fetchImpl).processTerminalResponses()");
expect(source).toContain("chatChannels.registerTerminalConfirmationWork(deliveryWork,");
expect(source).toContain("chatFailedRetryNoticeWork(chatChannels).register(deliveryWork,");
});
@@ -1,3 +1,4 @@
import { recordRunPresentationDecision } from "../services/heartbeat/run-completion.js";
import { environmentService } from "../services/environments.js";
import { createAdmissionTransactionScope, createWakeAdmissionWriter } from "../modules/wake-queue/adapters/postgres.js";
import { chatFailedRetryNoticeWork } from "../services/chat-failed-retry-notice-work.js";
@@ -43,7 +44,7 @@ import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-wor
import { aiConnectionService } from "../services/ai-connections.js";
import { fastResponseService } from "../services/fast-responses.js";
import { fastResponseReceipt } from "../services/fast-response-provider.js";
import { fastResponseRequests } from "@paperclipai/db";
import { companyFastResponses, fastResponseRequests } from "@paperclipai/db";
import { toolAccessService } from "../services/tool-access.js";
import { toolActionRequests, toolInvocations, toolPolicies } from "@paperclipai/db";
import { GitHubPublicationLeaseLost, withGitHubPublicationLease } from "../services/chat-github-publication-lease.js";
@@ -4035,6 +4036,240 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
expect(attempts).toBe(2);
expect(f.responseWrites).toHaveLength(writes);
});
it.each(["release", "restart", "drain", "standby"] as const)("wakes GitHub acknowledgements from a real publication lease %s", async mode => {
const f = await reviewBotFixture();
const [resource] = await db.select().from(chatEndpointResources).where(eq(chatEndpointResources.endpointId, f.endpoint.id));
const errors: unknown[] = [];
let canRun = true;
let coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun,
canReconcile: () => mode !== "standby" || canRun, onError: error => errors.push(error) });
const register = () => f.service.registerGitHubAcknowledgementWork(coordinator, () => canRun);
let worker = register();
let release!: () => void, entered!: () => void;
const hold = new Promise<void>(resolve => { release = resolve; });
const claimed = new Promise<void>(resolve => { entered = resolve; });
let lease: Promise<unknown> | undefined;
try {
await worker.ready;
expect(errors).toEqual([]);
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
expect(coordinator.nextWakeAt()).toBeNull();
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
await vi.advanceTimersByTimeAsync(120_000);
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
reads.mockRestore(); writes.mockRestore();
vi.setSystemTime(Date.now() - 120_000);
lease = withGitHubPublicationLease(db, { companyId: f.companyId, endpointId: f.endpoint.id,
repositoryId: String(resource.metadata.providerRepositoryId), number: 418 }, f.providerFetch, async () => { entered(); await hold; });
await claimed;
if (mode === "restart") await coordinator.stop();
if (mode === "drain" || mode === "standby") canRun = false;
const thread = makeThread({ channelId: "paperclipai/paperclip", id: "github:paperclipai/paperclip:issue:418" }).thread;
await deliverMessage({ callbacks: f.callbacks, endpointId: f.endpoint.id, provider: "github", thread,
trigger: "mention", message: makeMessage({ id: "41801", text: "Fix this", userId: "42", userName: "octocat", mentioned: true }) });
const [action] = await db.select().from(chatActions).where(and(eq(chatActions.endpointId, f.endpoint.id), eq(chatActions.kind, "github_response_comment")));
expect(action.status).toBe("received"); expect(f.responseWrites).toEqual([]);
release(); await lease;
if (mode === "drain" || mode === "standby") {
await vi.advanceTimersByTimeAsync(1000);
expect(f.responseWrites).toEqual([]);
canRun = true; worker.wake();
}
if (mode === "restart") {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
worker = register(); await worker.ready;
}
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(10);
expect((await db.select().from(chatActions).where(eq(chatActions.id, action.id)))[0].status).toBe("processed");
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
expect(f.responseWrites).toHaveLength(1);
expect(f.responseWrites[0].body).toContain("Working on this");
const settledReads = vi.spyOn(db, "select"), settledWrites = vi.spyOn(db, "transaction");
await vi.advanceTimersByTimeAsync(120_000);
expect(settledReads).not.toHaveBeenCalled(); expect(settledWrites).not.toHaveBeenCalled();
settledReads.mockRestore(); settledWrites.mockRestore();
expect(errors).toEqual([]);
} finally {
release(); await lease; await coordinator.stop(); vi.useRealTimers();
}
});
it.each(["failure", "expiry", "published", "delivery_unknown", "streaming quarantine"] as const)("releases a GitHub acknowledgement after fast response %s without maintenance polling", async outcome => {
await instanceSettingsService(db).updateExperimental({ enableFastResponses: true });
const f = await reviewBotFixture();
const binding = await aiConnectionService(db).save(f.companyId, "owner-user", { provider: "openrouter", method: "api_key", name: "Fast GitHub", ownership: "shared", apiKey: "fixture-key", agentIds: [], allAgents: true }, "fixture-key");
const fast = fastResponseService(db, { authorizeExternal: f.service.authorizeFastResponse,
provider: async () => ({ ...(outcome === "failure" ? { errorCode: "invalid_output", noProviderWork: true } : { text: "I’ll check it." }), receipt: fastResponseReceipt({ usage: { inputTokens: 80, outputTokens: 8 } }) }) });
await fast.configure(f.companyId, "owner-user", { enabled: true, ...binding, model: "openai/gpt-oss-120b", allowSponsored: true });
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
try {
await f.service.registerGitHubAcknowledgementWork(coordinator, () => true).ready;
const thread = makeThread({ channelId: "paperclipai/paperclip", id: "github:paperclipai/paperclip:issue:418" }).thread;
await deliverMessage({ callbacks: f.callbacks, endpointId: f.endpoint.id, provider: "github", thread,
trigger: "mention", message: makeMessage({ id: "41801", text: "Fix this", userId: "42", userName: "octocat", mentioned: true }) });
const [job] = await db.select().from(fastResponseRequests).where(eq(fastResponseRequests.companyId, f.companyId));
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(10);
expect(coordinator.nextWakeAt()).toBe(job.expiresAt.getTime());
}, { timeout: 5000 });
expect(f.responseWrites).toEqual([]);
if (outcome === "expiry") await vi.advanceTimersByTimeAsync(job.expiresAt.getTime() - Date.now());
else {
await fast.process(job);
if (outcome === "streaming quarantine") {
await db.update(chatPublications).set({ state: "streaming", updatedAt: new Date(0) })
.where(eq(chatPublications.idempotencyKey, `fast-response:${job.id}`));
await vi.advanceTimersByTimeAsync(Math.max(0, job.expiresAt.getTime() - Date.now()));
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(10); expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0); });
expect(f.responseWrites).toEqual([]);
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
await vi.advanceTimersByTimeAsync(120_000);
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
reads.mockRestore(); writes.mockRestore();
await f.service.processPendingPublications();
} else if (outcome === "delivery_unknown") {
await db.transaction(async tx => {
await notifyChatPublicationWork(tx, { companyId: job.companyId, issueId: job.issueId });
await tx.update(chatPublications).set({ state: "delivery_unknown" }).where(eq(chatPublications.idempotencyKey, `fast-response:${job.id}`));
});
} else if (outcome === "published") await f.service.processPendingPublications();
}
let processedAt: number | undefined;
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(10);
const [action] = await db.select().from(chatActions).where(and(eq(chatActions.endpointId, f.endpoint.id), eq(chatActions.kind, "github_response_comment")));
expect(action.status).toBe("processed");
processedAt ??= Date.now();
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
if (outcome !== "expiry" && outcome !== "streaming quarantine") expect(processedAt).toBeLessThan(job.expiresAt.getTime());
expect(f.responseWrites).toHaveLength(["failure", "expiry"].includes(outcome) ? 1 : 0);
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
await vi.advanceTimersByTimeAsync(120_000);
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
reads.mockRestore(); writes.mockRestore();
expect(errors).toEqual([]);
} finally { await coordinator.stop(); vi.useRealTimers(); }
});
it("wakes a GitHub acknowledgement only after the outer fast-response transaction commits", async () => {
await instanceSettingsService(db).updateExperimental({ enableFastResponses: true });
const f = await reviewBotFixture();
const binding = await aiConnectionService(db).save(f.companyId, "owner-user", { provider: "openrouter", method: "api_key", name: "Fast GitHub", ownership: "shared", apiKey: "fixture-key", agentIds: [], allAgents: true }, "fixture-key");
const fast = fastResponseService(db, { authorizeExternal: f.service.authorizeFastResponse });
await fast.configure(f.companyId, "owner-user", { enabled: true, ...binding, model: "openai/gpt-oss-120b", allowSponsored: true });
const thread = makeThread({ channelId: "paperclipai/paperclip", id: "github:paperclipai/paperclip:issue:418" }).thread;
await deliverMessage({ callbacks: f.callbacks, endpointId: f.endpoint.id, provider: "github", thread,
trigger: "mention", message: makeMessage({ id: "41801", text: "Fix this", userId: "42", userName: "octocat", mentioned: true }) });
const [job] = await db.update(fastResponseRequests).set({ expiresAt: new Date(Date.now() + 60_000) })
.where(eq(fastResponseRequests.companyId, f.companyId)).returning();
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
try {
await f.service.registerGitHubAcknowledgementWork(coordinator, () => true).ready;
expect(coordinator.nextWakeAt()).toBe(job.expiresAt.getTime());
const skip = async (tx: Parameters<Parameters<Db["transaction"]>[0]>[0]) => {
await tx.update(companyFastResponses).set({ enabled: false }).where(eq(companyFastResponses.companyId, f.companyId));
expect(await fastResponseService(tx as unknown as Db).process(job)).toMatchObject({ status: "unavailable" });
await vi.advanceTimersByTimeAsync(100);
expect(f.responseWrites).toEqual([]);
expect((await db.select().from(fastResponseRequests).where(eq(fastResponseRequests.id, job.id)))[0].status).toBe("pending");
};
await expect(db.transaction(async tx => { await skip(tx); throw new Error("rollback fast response skip"); })).rejects.toThrow("rollback fast response skip");
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(10); expect(idleWorkSnapshot().active).toBe(0); });
expect(f.responseWrites).toEqual([]);
expect(coordinator.nextWakeAt()).toBe(job.expiresAt.getTime());
await db.transaction(skip);
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(10);
expect(f.responseWrites).toHaveLength(1);
expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0);
}, { timeout: 10_000 });
expect(Date.now()).toBeLessThan(job.expiresAt.getTime());
expect(errors).toEqual([]);
} finally { await coordinator.stop(); vi.useRealTimers(); }
});
it.each(["drain", "standby", "shutdown"] as const)("stops GitHub acknowledgement sends at %s between provider requests", async mode => {
const f = await reviewBotFixture();
f.setSupplementalProviderFetch(async (input, init) => String(input).includes("/issues/418/comments") && init?.method === "POST" ? new Response("unavailable", { status: 503 }) : undefined);
const thread = makeThread({ channelId: "paperclipai/paperclip", id: "github:paperclipai/paperclip:issue:418" }).thread;
await deliverMessage({ callbacks: f.callbacks, endpointId: f.endpoint.id, provider: "github", thread,
trigger: "mention", message: makeMessage({ id: "41801", text: "Fix this", userId: "42", userName: "octocat", mentioned: true }) });
const [action] = await db.select().from(chatActions).where(and(eq(chatActions.endpointId, f.endpoint.id), eq(chatActions.kind, "github_response_comment")));
expect(action.status).toBe("failed"); expect(action.result?.attempts).toBe(1);
await db.update(chatActions).set({ result: { ...action.result, retryAt: new Date(0).toISOString() } }).where(eq(chatActions.id, action.id));
let release!: () => void, entered!: () => void;
const held = new Promise<void>(resolve => { release = resolve; });
const started = new Promise<void>(resolve => { entered = resolve; });
f.setSupplementalProviderFetch(async input => {
if (String(input).includes("/issues/418/comments?")) { entered(); await held; }
return undefined;
});
let canRun = true;
const errors: unknown[] = [];
let coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun,
canReconcile: () => mode !== "standby" || canRun, onError: error => errors.push(error) });
let worker = f.service.registerGitHubAcknowledgementWork(coordinator, () => canRun);
let stopping: Promise<void> | undefined;
try {
await started;
if (mode === "shutdown") stopping = coordinator.stop(); else canRun = false;
release(); await worker.ready; await stopping;
expect(f.responseWrites).toEqual([]);
expect((await db.select().from(chatActions).where(eq(chatActions.id, action.id)))[0].result?.attempts).toBe(1);
canRun = true;
if (mode === "shutdown") {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
worker = f.service.registerGitHubAcknowledgementWork(coordinator, () => true); await worker.ready;
} else worker.wake();
await vi.waitFor(() => { expect(f.responseWrites).toHaveLength(1); expect(coordinator.nextWakeAt()).toBeNull(); }, { timeout: 10_000 });
expect(errors).toEqual([]);
} finally { release(); await stopping; await coordinator.stop(); }
});
it("retries a failed GitHub acknowledgement at its saved deadline", async () => {
const f = await reviewBotFixture();
let unavailable = true, attempts = 0;
f.setSupplementalProviderFetch(async (input, init) => {
if (String(input).includes("/issues/418/comments") && init?.method === "POST") { attempts++; if (unavailable) return new Response("unavailable", { status: 503 }); }
return undefined;
});
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => { throw error; } });
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
try {
await f.service.registerGitHubAcknowledgementWork(coordinator, () => true).ready;
const thread = makeThread({ channelId: "paperclipai/paperclip", id: "github:paperclipai/paperclip:issue:418" }).thread;
await deliverMessage({ callbacks: f.callbacks, endpointId: f.endpoint.id, provider: "github", thread,
trigger: "mention", message: makeMessage({ id: "41801", text: "Fix this", userId: "42", userName: "octocat", mentioned: true }) });
const [action] = await db.select().from(chatActions).where(and(eq(chatActions.endpointId, f.endpoint.id), eq(chatActions.kind, "github_response_comment")));
expect(action.status).toBe("failed"); expect(attempts).toBe(1);
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(10); expect(coordinator.nextWakeAt()).toBe(Date.parse(String(action.result?.retryAt))); });
unavailable = false;
await vi.advanceTimersByTimeAsync(Math.max(0, coordinator.nextWakeAt()! - Date.now()));
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(10); expect(f.responseWrites).toHaveLength(1); expect(coordinator.nextWakeAt()).toBeNull(); }, { timeout: 10_000 });
expect(attempts).toBe(2);
} finally { await coordinator.stop(); vi.useRealTimers(); }
});
it.each(["received", "failed", "cancelled"] as const)("preserves the %s GitHub receipt lifecycle during acknowledgement recovery", async status => {
const f = await toolReplyFixture();
const [action] = await db.select().from(chatActions).where(and(eq(chatActions.endpointId, f.endpoint.id), eq(chatActions.kind, "github_response_comment")));
expect(action.result?.id).toBeTruthy();
await db.update(chatActions).set({ status }).where(eq(chatActions.id, action.id));
const writes = f.responseWrites.length;
await githubResponseCommentService(db, f.providerFetch).tryAcknowledge(action.deliveryId!);
const [settled] = await db.select().from(chatActions).where(eq(chatActions.id, action.id));
expect(settled.status).toBe(status === "cancelled" ? "cancelled" : "processed");
expect(settled.result).toEqual(action.result);
expect(f.responseWrites).toHaveLength(writes);
});
it("does not post a GitHub working comment when the scheduler declines the request", async () => {
const f = await reviewBotFixture();
f.wakeup.mockImplementation(async (agentId, opts) => {
@@ -4167,6 +4402,54 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
expect([...f.responseComments.values()][0].body).toContain("paperclip-response-state:final");
expect(f.responseWrites.map(write => write.method)).toEqual(["POST", "PATCH"]);
});
it("parks a GitHub acknowledgement until native presentation commits", async () => {
const f = await toolReplyFixture();
const [action] = await db.select().from(chatActions).where(and(
eq(chatActions.endpointId, f.endpoint.id), eq(chatActions.kind, "github_response_comment")));
await db.update(chatActions).set({ status: "received", result: null }).where(eq(chatActions.id, action.id));
await db.update(heartbeatRuns).set({ status: "succeeded", nativePhase: "committed", resultJson: {} })
.where(eq(heartbeatRuns.id, f.run.id));
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
onError: error => errors.push(error) });
const decision = { schema: "paperclip.run_presentation_decision.v1" as const,
commentAction: "none" as const, chosenSource: "none" as const, reasonCodes: ["no_final_response"] };
const commit = (database: typeof db) => recordRunPresentationDecision(database, {
companyId: f.companyId, runId: f.run.id, issueId: f.conversation.issueId!, decision,
});
try {
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
await f.service.registerGitHubAcknowledgementWork(coordinator, () => true).ready;
const quiet = async () => {
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(10);
expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0); });
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
try {
await vi.advanceTimersByTimeAsync(120_000);
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
} finally { reads.mockRestore(); writes.mockRestore(); }
};
await quiet();
expect(f.responseWrites).toHaveLength(1);
await expect(db.transaction(async tx => {
await commit(tx as unknown as typeof db);
expect((await readHeartbeatRunsById(f.run.id))[0].resultJson?.presentationDecision).toBeUndefined();
throw new Error("rollback presentation");
})).rejects.toThrow("rollback presentation");
await quiet();
expect(f.responseWrites).toHaveLength(1);
await commit(db);
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(10);
expect((await db.select().from(chatActions).where(eq(chatActions.id, action.id)))[0])
.toMatchObject({ status: "processed", result: { final: true } });
}, { timeout: 10_000 });
expect(f.responseWrites.map(write => write.method)).toEqual(["POST", "PATCH"]);
expect(f.responseWrites[1].body).toContain("without publishing a final response");
await quiet();
expect(errors).toEqual([]);
} finally { await coordinator.stop(); vi.useRealTimers(); }
});
it("does not let an unsettled GitHub fallback block a later board message", async () => {
const f = await toolReplyFixture();
await db.insert(chatActions).values({ companyId: f.companyId, endpointId: f.endpoint.id, conversationId: f.conversation.id,
+6 -3
View File
@@ -37,7 +37,7 @@ import { instanceSettingsService } from "../services/instance-settings.js";
import { settleConversationTurn } from "../services/agent-conversations.js";
import { buildLowTrustSourceTrust, LOW_TRUST_QUARANTINED_BODY } from "../services/source-trust.js";
import { deliverBudgetEnforcement } from "../services/budgets.js";
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>,
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined,
db: ReturnType<typeof createDb>,
home: string;
beforeAll(async () => {
@@ -46,12 +46,15 @@ beforeAll(async () => {
);
vi.stubEnv("PAPERCLIP_HOME", home);
vi.stubEnv("PAPERCLIP_INSTANCE_ID", "fast-response-fixture");
database = await startEmbeddedPostgresTestDatabase(
const external = process.env.PAPERCLIP_TEST_DATABASE_URL;
if (external && !new URL(external).pathname.endsWith("_test")) throw new Error("Fast-response tests require a database ending in _test");
database = external ? undefined : await startEmbeddedPostgresTestDatabase(
"paperclip-fast-response-db-",
);
db = createDb(database.connectionString);
db = createDb(external ?? database!.connectionString);
}, 90000);
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await database?.cleanup();
vi.unstubAllEnvs();
if (home) await rm(home, { recursive: true, force: true });
+1
View File
@@ -1289,6 +1289,7 @@ export async function createApp(
chatChannels.registerTeamsConsentExpiryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTeamsAttemptRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerInteractionContinuationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerGitHubAcknowledgementWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTerminalConfirmationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerCommandWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
nativeChatProgressWork(db).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
+2
View File
@@ -11,6 +11,7 @@ import { credentialFingerprint } from "./chat-channels/credential-fingerprint.js
import { createChatSdkMessageReceiver } from "./chat-channels/runtime/sdk-messages.js";
import { admitDiscordSdkMessage } from "./chat-channels/discord/messages.js";
import { resolveInboundConversationBinding } from "./chat-channels/inbound/conversation-binding.js";
import { githubResponseCommentService } from "./chat-github-response-comments.js";
import { createGitHubMaintenance } from "./chat-channels/github/maintenance.js";
import { createSpekoChannelService, authorizeSpekoPublicVoiceTask, replaceSpekoChannelRuntime, SPEKO_CONVERSATION_IS_LINEAR } from "./chat-channels/speko/runtime.js";
import { prepareDiscordRuntimeCommand } from "./chat-channels/discord/runtime-commands.js";
@@ -3503,6 +3504,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
const { getPublicationBatchStatus, publishComment, publishBoardMessage } = createChatPublicationRequests(db, { endpointRecord, stageTeamsBoardFileIntent, processPendingSlackBoardMessages, processPendingPublications, prepareTeamsBoardSend });
return {
registerInteractionContinuationWork,
registerGitHubAcknowledgementWork: githubResponseCommentService(db, fetchImpl).registerAcknowledgementWork,
registerTerminalConfirmationWork,
loadFailedRetryNoticeScopes,
reconcileFailedRetryNotices,
@@ -0,0 +1,112 @@
import { readGitHubResponseRun } from "./response-run-state.js";
import { githubRunReplyState } from "../../chat-run-publications.js";
import { and, asc, eq, gt, isNotNull, or, sql } from "drizzle-orm";
import { chatActions, chatConversations, chatEndpointLeases, chatEndpointResources,
chatPublications, fastResponseRequests, chatEndpoints, chatMessageLinks, type Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../../delivery-work-notifications.js";
import type { ChatResponseScope } from "../../chat-response-availability.js";
import { registerScopedDeadlineWork } from "../../scoped-deadline-work.js";
type Scope = ChatResponseScope & { endpointId?: string; repositoryId?: string; number?: number };
type Entry = { id: string; companyId: string; endpointId: string; deliveryId: string;
issueId: string | null; conversationId: string | null; repositoryId: string | null; number: number | null; dueAt: number };
type FastResponse = { status: string; expiresAt: string; publicationState: string | null };
export function registerGitHubAcknowledgementWork(
db: Db,
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
acknowledge: (deliveryId: string, canContinue: () => boolean) => Promise<unknown>,
canRun: () => boolean,
) {
const repositoryId = sql<string | null>`${chatEndpointResources.metadata}->>'providerRepositoryId'`;
const number = sql<string | null>`substring(${chatConversations.externalThreadId}
from '^github:[^:]+:(?:issue:)?([1-9][0-9]*)(?::rc:[1-9][0-9]*)?$')`;
const selection = {
id: chatActions.id, companyId: chatActions.companyId, endpointId: chatActions.endpointId,
deliveryId: chatActions.deliveryId, conversationId: chatActions.conversationId, issueId: chatConversations.issueId,
agentId: chatEndpoints.assignedAgentId,
sourceMessageId: sql<string | null>`(select m.comment_id from ${chatMessageLinks} m
where m.company_id = ${chatActions.companyId} and m.endpoint_id = ${chatActions.endpointId}
and m.delivery_id = ${chatActions.deliveryId} and m.direction = 'inbound' limit 1)`,
repositoryId, number, status: chatActions.status, result: chatActions.result,
leaseExpiresAt: sql<string | null>`(select max(l.expires_at) from ${chatEndpointLeases} l
where l.company_id = ${chatActions.companyId} and l.endpoint_id = ${chatActions.endpointId}
and l.lease_key = 'github-publication:' || (${repositoryId}) || ':' || ${number})`,
fast: sql<FastResponse | null>`(select jsonb_build_object('status', f.status, 'expiresAt', f.expires_at,
'publicationState', p.state) from ${fastResponseRequests} f left join ${chatPublications} p
on p.company_id = f.company_id and p.comment_id = f.comment_id
and p.endpoint_id = f.endpoint_id and p.conversation_id = f.conversation_id
where f.company_id = ${chatActions.companyId} and f.endpoint_id = ${chatActions.endpointId}
and f.delivery_id = ${chatActions.deliveryId} limit 1)`,
};
async function page(scope?: Scope, cursor?: string, actionId?: string) {
return db.select(selection).from(chatActions)
.innerJoin(chatEndpoints, and(eq(chatEndpoints.id, chatActions.endpointId), eq(chatEndpoints.companyId, chatActions.companyId)))
.leftJoin(chatConversations, and(eq(chatConversations.id, chatActions.conversationId), eq(chatConversations.companyId, chatActions.companyId)))
.leftJoin(chatEndpointResources, and(eq(chatEndpointResources.id, chatConversations.resourceId), eq(chatEndpointResources.companyId, chatActions.companyId)))
.where(and(eq(chatActions.kind, "github_response_comment"), isNotNull(chatActions.deliveryId),
sql`coalesce(${chatActions.result}->>'final', 'false') <> 'true'`,
or(eq(chatActions.status, "received"), and(eq(chatActions.status, "failed"), sql`${chatActions.result}->>'retryable' = 'true'`)),
scope ? eq(chatActions.companyId, scope.companyId) : undefined,
scope?.issueId ? eq(chatConversations.issueId, scope.issueId) : undefined,
scope?.conversationId ? eq(chatActions.conversationId, scope.conversationId) : undefined,
scope?.deliveryId ? eq(chatActions.deliveryId, scope.deliveryId) : undefined,
scope?.endpointId ? eq(chatActions.endpointId, scope.endpointId) : undefined,
scope?.repositoryId ? eq(repositoryId, scope.repositoryId) : undefined,
scope?.number ? eq(number, String(scope.number)) : undefined,
cursor ? gt(chatActions.id, cursor) : undefined,
actionId ? eq(chatActions.id, actionId) : undefined,
)).orderBy(asc(chatActions.id)).limit(100);
}
async function entry(row: Awaited<ReturnType<typeof page>>[number]): Promise<Entry | null> {
const { latestRun, ownsLatestRequest } = await readGitHubResponseRun(db, {
companyId: row.companyId, endpointId: row.endpointId, agentId: row.agentId,
issueId: row.issueId, deliveryId: row.deliveryId!, sourceMessageId: row.sourceMessageId,
});
// Execution success alone does not decide whether a final response exists.
// Presentation/publication commits invalidate this parked request.
if (row.issueId && ownsLatestRequest && latestRun?.nativePresentationPending &&
await githubRunReplyState(db, { companyId: row.companyId, endpointId: row.endpointId,
issueId: row.issueId, runId: latestRun.id }) === "none") return null;
let dueAt = row.status === "failed" ? Date.parse(String(row.result?.retryAt)) : 0;
if (!Number.isFinite(dueAt)) throw new Error("GitHub acknowledgement has an invalid retry deadline");
const fast = row.fast;
if (fast?.publicationState === "streaming") return null;
if (fast && !["published", "delivery_unknown"].includes(fast.publicationState ?? "") &&
(["pending", "running"].includes(fast.status) || ["pending", "retry"].includes(fast.publicationState ?? ""))) {
dueAt = Math.max(dueAt, Date.parse(fast.expiresAt));
}
if (row.leaseExpiresAt) dueAt = Math.max(dueAt, Date.parse(row.leaseExpiresAt));
return { id: row.id, companyId: row.companyId, endpointId: row.endpointId, deliveryId: row.deliveryId!,
issueId: row.issueId, conversationId: row.conversationId, repositoryId: row.repositoryId, number: row.number ? Number(row.number) : null, dueAt };
}
return registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.chatGitHubAcknowledgements,
decodeScope: (hint: string): Scope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (value: Entry, scope?: Scope) => !scope || (value.companyId === scope.companyId &&
(!scope.issueId || value.issueId === scope.issueId) && (!scope.deliveryId || value.deliveryId === scope.deliveryId) &&
(!scope.conversationId || value.conversationId === scope.conversationId) &&
(!scope.endpointId || value.endpointId === scope.endpointId) && (!scope.repositoryId || value.repositoryId === scope.repositoryId) &&
(!scope.number || value.number === scope.number)),
load: async (scope?: Scope, signal?: AbortSignal) => {
const entries: Entry[] = [];
let cursor: string | undefined;
while (canRun() && !signal?.aborted) {
const rows = await page(scope, cursor);
for (const row of rows) { const value = await entry(row); if (value) entries.push(value); }
if (rows.length < 100) return entries;
cursor = rows[rows.length - 1].id;
}
throw signal?.reason ?? new Error("GitHub acknowledgement recovery stopped");
},
check: async (value: Entry, signal?: AbortSignal) => {
const [row] = await page(undefined, undefined, value.id);
const current = row ? await entry(row) : null;
if (!current || current.dueAt > Date.now() || !canRun() || signal?.aborted) return current;
await acknowledge(current.deliveryId, () => canRun() && !signal?.aborted);
const [remaining] = await page(undefined, undefined, value.id);
return remaining ? await entry(remaining) : null;
},
}, canRun);
}
@@ -13,7 +13,7 @@ export function createGitHubMaintenance(db: Db, fetchImpl: typeof globalThis.fet
options.scheduleMessageProcessing(async () => {
try {
// Provider I/O must not block Slack, Teams, or ordinary message receipts.
await githubResponseCommentService(db, fetchImpl).processPending();
await githubResponseCommentService(db, fetchImpl).processTerminalResponses();
await githubReviewCheckService(db, fetchImpl).processPending();
} finally {
githubMaintenancePending = false;
@@ -0,0 +1,71 @@
import { and, desc, eq, or, sql } from "drizzle-orm";
import { chatMessageLinks, heartbeatRuns, type Db } from "@paperclipai/db";
import { chatRunSourceMessageId } from "../run-origin.js";
export const responseRunColumns = {
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
responsibleUserId: heartbeatRuns.responsibleUserId,
activeIdentityContextId: heartbeatRuns.activeIdentityContextId,
nativeIssueId: heartbeatRuns.nativeIssueId,
issueId: heartbeatRuns.issueId,
status: heartbeatRuns.status,
updatedAt: heartbeatRuns.updatedAt,
// Terminal execution can precede native presentation commit. Until its
// durable decision exists, missing output is undecided rather than failure.
nativePresentationPending: sql<boolean>`coalesce(
${heartbeatRuns.status} = 'succeeded'
and ${heartbeatRuns.runtimeMode} = 'native'
and (${heartbeatRuns.nativePhase} is distinct from 'committed'
or ${heartbeatRuns.resultJson}->'presentationDecision'->>'schema' is distinct from 'paperclip.run_presentation_decision.v1'
or coalesce(${heartbeatRuns.resultJson}->'presentationDecision'->>'commentAction', '') not in ('none', 'create', 'reuse')),
false
)`,
// Project only the server-committed no-publication decision, never the
// potentially large native result or model-authored summary.
passiveResponseWait: sql<boolean>`coalesce(
${heartbeatRuns.status} = 'succeeded'
and ${heartbeatRuns.runtimeMode} = 'native'
and ${heartbeatRuns.nativePhase} = 'committed'
and ${heartbeatRuns.resultJson}->>'finalizationPhase' = 'committed'
and ${heartbeatRuns.resultJson}->>'finalizationReasonCode' = 'external_chat_response_waiting'
and ${heartbeatRuns.resultJson}->'nativeResult'->>'schema' = 'paperclip.run_result.v1'
and ${heartbeatRuns.resultJson}->'nativeResult'->>'reportedWorkDisposition' = 'yielded'
and ${heartbeatRuns.resultJson}->'nativeResult'->'continuation'->>'kind' = 'response_wake'
and ${heartbeatRuns.resultJson}->'presentationDecision'->>'schema' = 'paperclip.run_presentation_decision.v1'
and ${heartbeatRuns.resultJson}->'presentationDecision'->>'chosenSource' = 'none'
and ${heartbeatRuns.resultJson}->'presentationDecision'->>'commentAction' = 'none'
and ${heartbeatRuns.resultJson}->'presentationDecision'->'reasonCodes' @> '["yielded_control_plane_wait"]'::jsonb,
false
)`,
contextSnapshot: sql<Record<string, unknown>>`jsonb_build_object(
'issueId', ${heartbeatRuns.contextSnapshot}->'issueId',
'taskId', ${heartbeatRuns.contextSnapshot}->'taskId',
'wakeCommentId', ${heartbeatRuns.contextSnapshot}->'wakeCommentId',
'wakeCommentIds', ${heartbeatRuns.contextSnapshot}->'wakeCommentIds'
)`,
};
/** Shared request ownership and presentation state for delivery and recovery. */
export async function readGitHubResponseRun(db: Db, input: {
companyId: string; endpointId: string; agentId: string; issueId: string | null;
deliveryId: string; sourceMessageId: string | null;
}) {
const [requestRun] = await db.select({ runId: heartbeatRuns.id }).from(heartbeatRuns).innerJoin(chatMessageLinks, and(
eq(chatMessageLinks.companyId, input.companyId), eq(chatMessageLinks.endpointId, input.endpointId),
eq(chatMessageLinks.deliveryId, input.deliveryId), eq(chatMessageLinks.direction, "inbound"),
or(sql`${chatMessageLinks.commentId}::text = ${heartbeatRuns.contextSnapshot}->>'wakeCommentId'`,
sql`coalesce(${heartbeatRuns.contextSnapshot}->'wakeCommentIds', '[]'::jsonb) ? ${chatMessageLinks.commentId}::text`),
)).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId),
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${input.issueId}`))
.orderBy(desc(heartbeatRuns.createdAt)).limit(1);
const [latestRun] = await db.select(responseRunColumns).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId),
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${input.issueId}`,
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
const originMessageId = latestRun?.activeIdentityContextId ? await chatRunSourceMessageId(db, latestRun,
{ allowLegacyWakeComment: true, provider: "github" }) : null;
const ownsLatestRequest = !!latestRun && (requestRun?.runId === latestRun.id ||
(!!input.sourceMessageId && originMessageId === input.sourceMessageId));
return { latestRun, ownsLatestRequest };
}
@@ -1,3 +1,4 @@
import { notifyChatResponseAvailability } from "../../chat-response-availability.js";
import { notifyChatEndpointConfiguration } from "../configuration-work-notifications.js";
import { notifyChatConnectionHealth } from "../connection-health-notifications.js";
import { isOutboundAttachmentHydrationError, isOutboundAttachmentValidationError } from "../outbound-attachments.js";
@@ -121,6 +122,7 @@ export function createChatPublicationFailures(db: Db, options: PublicationFailur
? "cancelled"
: "failed";
await notifyChatPublicationWork(tx, state === "failed" || state === "cancelled" ? publication : undefined);
if (state === "retry" || state === "delivery_unknown") await notifyChatResponseAvailability(tx, publication);
const [ownedPublication] = await tx
.update(chatPublications)
.set({
@@ -1,3 +1,4 @@
import { notifyChatResponseAvailability } from "../../chat-response-availability.js";
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
import { unresolvedSlackPublicationCondition } from "../../connectors/slack-publication.js";
@@ -242,9 +243,11 @@ export function createChatPublicationQueue(db: Db, options: PublicationQueueOpti
),
),
);
const candidates = await tx.select({ id: chatPublications.id }).from(chatPublications).where(condition).for("update", { skipLocked: true });
const candidates = await tx.select({ id: chatPublications.id, companyId: chatPublications.companyId, issueId: chatPublications.issueId })
.from(chatPublications).where(condition).for("update", { skipLocked: true });
if (!candidates.length) return [];
await notifyChatPublicationWork(tx);
for (const candidate of candidates) await notifyChatResponseAvailability(tx, candidate);
return tx
.update(chatPublications)
.set({
@@ -1,3 +1,4 @@
import { notifyChatResponseAvailability } from "../../chat-response-availability.js";
import type { DbOrTransaction } from "../runtime-types.js";
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
@@ -116,6 +117,7 @@ export function createChatPublicationTaskControl(db: Db) {
// finishes after /new or /close. A racing insert that commits after this
// transaction is rejected by the completed-conversation send guard.
await notifyChatPublicationWork(tx);
await notifyChatResponseAvailability(tx, publication);
await tx
.update(chatPublications)
.set({
@@ -1,3 +1,4 @@
import { notifyChatResponseAvailability } from "../../chat-response-availability.js";
import { notifyVoicePublication } from "../../voice/voice-reply-notifications.js";
import { createGitHubPublicationPolicy } from "../github/publication-policy.js";
@@ -155,6 +156,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
: Math.min(60_000, 2 ** Math.max(0, attempts) * 1_000);
const result = await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx, terminal ? publication : undefined);
if (!terminal) await notifyChatResponseAvailability(tx, publication);
const [settled] = await tx
.update(chatPublications)
.set({
@@ -475,6 +477,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
await publicationLease.assertOwned();
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatResponseAvailability(tx, publication);
return tx
.update(chatPublications)
.set({
@@ -116,6 +116,7 @@ export async function withGitHubPublicationLease<T>(
await renewal;
await db.transaction(async tx => {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubPublications);
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubAcknowledgements, JSON.stringify(scope));
await tx.delete(chatEndpointLeases).where(and(eq(chatEndpointLeases.id, claimed.id), eq(chatEndpointLeases.token, token)));
});
}
@@ -1,3 +1,4 @@
import { readGitHubResponseRun, responseRunColumns } from "./chat-channels/github/response-run-state.js";
import { and, asc, desc, eq, inArray, or, sql } from "drizzle-orm";
import {
chatActions, chatConversations, chatDeliveries, chatEndpoints,
@@ -5,6 +6,9 @@ import {
fastResponseRequests, chatPublications,
type Db,
} from "@paperclipai/db";
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
import { registerGitHubAcknowledgementWork } from "./chat-channels/github/acknowledgement-work.js";
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
import { conflict, forbidden, HttpError } from "../errors.js";
import { githubChatPrincipalAccess } from "./chat-github-access.js";
import { githubAutomaticAdmission } from "./chat-github-events.js";
@@ -25,51 +29,8 @@ type Api = { request<T>(path: string, options?: { method?: string; body?: unknow
type Lease = { commit<T>(write: (tx: Transaction) => Promise<T>): Promise<T> };
type Source = GitHubResponseSource;
type Comment = { id: number; body?: string; html_url: string; user?: { login?: string } };
const responseRunColumns = {
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
responsibleUserId: heartbeatRuns.responsibleUserId,
activeIdentityContextId: heartbeatRuns.activeIdentityContextId,
nativeIssueId: heartbeatRuns.nativeIssueId,
issueId: heartbeatRuns.issueId,
status: heartbeatRuns.status,
updatedAt: heartbeatRuns.updatedAt,
// Terminal execution can precede native presentation commit. Until its
// durable decision exists, missing output is undecided rather than failure.
nativePresentationPending: sql<boolean>`coalesce(
${heartbeatRuns.status} = 'succeeded'
and ${heartbeatRuns.runtimeMode} = 'native'
and (${heartbeatRuns.nativePhase} is distinct from 'committed'
or ${heartbeatRuns.resultJson}->'presentationDecision'->>'schema' is distinct from 'paperclip.run_presentation_decision.v1'
or coalesce(${heartbeatRuns.resultJson}->'presentationDecision'->>'commentAction', '') not in ('none', 'create', 'reuse')),
false
)`,
// Project only the server-committed no-publication decision, never the
// potentially large native result or model-authored summary.
passiveResponseWait: sql<boolean>`coalesce(
${heartbeatRuns.status} = 'succeeded'
and ${heartbeatRuns.runtimeMode} = 'native'
and ${heartbeatRuns.nativePhase} = 'committed'
and ${heartbeatRuns.resultJson}->>'finalizationPhase' = 'committed'
and ${heartbeatRuns.resultJson}->>'finalizationReasonCode' = 'external_chat_response_waiting'
and ${heartbeatRuns.resultJson}->'nativeResult'->>'schema' = 'paperclip.run_result.v1'
and ${heartbeatRuns.resultJson}->'nativeResult'->>'reportedWorkDisposition' = 'yielded'
and ${heartbeatRuns.resultJson}->'nativeResult'->'continuation'->>'kind' = 'response_wake'
and ${heartbeatRuns.resultJson}->'presentationDecision'->>'schema' = 'paperclip.run_presentation_decision.v1'
and ${heartbeatRuns.resultJson}->'presentationDecision'->>'chosenSource' = 'none'
and ${heartbeatRuns.resultJson}->'presentationDecision'->>'commentAction' = 'none'
and ${heartbeatRuns.resultJson}->'presentationDecision'->'reasonCodes' @> '["yielded_control_plane_wait"]'::jsonb,
false
)`,
contextSnapshot: sql<Record<string, unknown>>`jsonb_build_object(
'issueId', ${heartbeatRuns.contextSnapshot}->'issueId',
'taskId', ${heartbeatRuns.contextSnapshot}->'taskId',
'wakeCommentId', ${heartbeatRuns.contextSnapshot}->'wakeCommentId',
'wakeCommentIds', ${heartbeatRuns.contextSnapshot}->'wakeCommentIds'
)`,
};
const generation = (endpoint: Source["endpoint"]) => (endpoint.setup as { runtimeGeneration?: number }).runtimeGeneration ?? 0;
const stoppedAcknowledgement = Symbol("stopped GitHub acknowledgement");
const actionKey = (deliveryId: string) => `github_response:${deliveryId}`;
const responseMarker = (source: Source) => isMainPullResponse(source) ? pullResponseMarker(source) : requestResponseMarker(source);
@@ -91,18 +52,23 @@ async function findResponseComment(source: Source, api: Api, legacyMarkers: stri
* Reconnect epochs fence publication attempts, not the durable comment identity.
* Legacy receipts are also checked against the App-owned provider comment. */
async function stageResponse(db: Db, source: Source) {
const [inserted] = await db.insert(chatActions).values({
companyId: source.endpoint.companyId, endpointId: source.endpoint.id,
conversationId: source.conversation.id, deliveryId: source.delivery.id,
principalId: source.delivery.principalId, kind: "github_response_comment",
providerActionId: actionKey(source.delivery.id),
payload: { botExternalId: source.endpoint.botExternalId, agentId: source.endpoint.assignedAgentId },
status: "received",
}).onConflictDoNothing().returning();
const response = inserted ?? (await db.select().from(chatActions).where(and(
const lookup = () => db.select().from(chatActions).where(and(
eq(chatActions.companyId, source.endpoint.companyId), eq(chatActions.endpointId, source.endpoint.id),
eq(chatActions.providerActionId, actionKey(source.delivery.id)),
)))[0];
)).then(rows => rows[0]);
const existing = await lookup();
const response = existing ?? await db.transaction(async tx => {
await notifyChatResponseAvailability(tx, { companyId: source.endpoint.companyId, deliveryId: source.delivery.id });
const [inserted] = await tx.insert(chatActions).values({
companyId: source.endpoint.companyId, endpointId: source.endpoint.id,
conversationId: source.conversation.id, deliveryId: source.delivery.id,
principalId: source.delivery.principalId, kind: "github_response_comment",
providerActionId: actionKey(source.delivery.id),
payload: { botExternalId: source.endpoint.botExternalId, agentId: source.endpoint.assignedAgentId },
status: "received",
}).onConflictDoNothing().returning();
return inserted;
}) ?? await lookup();
if (!response || response.conversationId !== source.conversation.id ||
(response.payload.botExternalId !== undefined && response.payload.botExternalId !== source.endpoint.botExternalId) ||
response.payload.agentId !== source.endpoint.assignedAgentId)
@@ -135,6 +101,10 @@ export async function writeGitHubResponseComment(
(input.versionAt && Date.parse(String(response.result.versionAt)) > input.versionAt.getTime()))) {
if (!input.acknowledgement && response.result.final === true && !input.final)
throw conflict("This GitHub response is already final");
if (["received", "failed"].includes(response.status)) await lease.commit(async tx => {
await tx.update(chatActions).set({ status: "processed", updatedAt: new Date() })
.where(and(eq(chatActions.id, response.id), eq(chatActions.status, response.status)));
});
return { id: String(response.result.id), url: String(response.result.url) };
}
const marker = responseMarker(source);
@@ -213,13 +183,18 @@ export function githubResponseCommentService(db: Db, fetchImpl = fetch) {
throw forbidden("Invalid GitHub request repository");
return { ...row, repositoryId, repository: match[1]!, number: Number(match[2]), replyId: match[3] ?? null };
}
async function acknowledge(deliveryId: string, finalBody?: string, assertRun?: () => Promise<void>) {
async function acknowledge(deliveryId: string, finalBody?: string, assertRun?: () => Promise<void>, canContinue = () => true) {
if (!canContinue()) return;
const source = await context(deliveryId);
// Persist before competing for the publication lane. A busy lane leaves
// durable acknowledgement work for maintenance instead of losing it.
await stageResponse(db, source);
if (!canContinue()) return;
return withGitHubPublicationLease(db, { companyId: source.endpoint.companyId, endpointId: source.endpoint.id,
repositoryId: source.repositoryId, number: source.number }, fetchImpl, async lease => {
repositoryId: source.repositoryId, number: source.number }, (input, init) => {
if (!canContinue()) throw stoppedAcknowledgement;
return fetchImpl(input, init);
}, async lease => {
const assertCurrent = async () => {
await assertRun?.();
const current = await context(deliveryId);
@@ -234,22 +209,11 @@ export function githubResponseCommentService(db: Db, fetchImpl = fetch) {
let preserveFinal = false;
if (finalBody === undefined) {
const existing = await stageResponse(db, source);
const [requestRun] = await db.select({ runId: heartbeatRuns.id }).from(heartbeatRuns).innerJoin(chatMessageLinks, and(
eq(chatMessageLinks.companyId, source.endpoint.companyId), eq(chatMessageLinks.endpointId, source.endpoint.id),
eq(chatMessageLinks.deliveryId, deliveryId), eq(chatMessageLinks.direction, "inbound"),
or(sql`${chatMessageLinks.commentId}::text = ${heartbeatRuns.contextSnapshot}->>'wakeCommentId'`,
sql`coalesce(${heartbeatRuns.contextSnapshot}->'wakeCommentIds', '[]'::jsonb) ? ${chatMessageLinks.commentId}::text`),
)).where(and(eq(heartbeatRuns.companyId, source.endpoint.companyId), eq(heartbeatRuns.agentId, source.endpoint.assignedAgentId),
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${source.conversation.issueId}`))
.orderBy(desc(heartbeatRuns.createdAt)).limit(1);
const [latestRun] = await db.select(responseRunColumns).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, source.endpoint.companyId), eq(heartbeatRuns.agentId, source.endpoint.assignedAgentId),
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${source.conversation.issueId}`,
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
const originMessageId = latestRun?.activeIdentityContextId ? await chatRunSourceMessageId(db, latestRun,
{ allowLegacyWakeComment: true, provider: "github" }) : null;
const ownsLatestRequest = !!latestRun && (requestRun?.runId === latestRun.id ||
(!!source.sourceMessageId && originMessageId === source.sourceMessageId));
const { latestRun, ownsLatestRequest } = await readGitHubResponseRun(db, {
companyId: source.endpoint.companyId, endpointId: source.endpoint.id,
agentId: source.endpoint.assignedAgentId, issueId: source.conversation.issueId,
deliveryId, sourceMessageId: source.sourceMessageId,
});
requestRunning = ownsLatestRequest && latestRun?.status === "running";
const terminal = latestRun && ["succeeded", "failed", "interrupted", "timed_out", "cancelled"].includes(latestRun.status);
const run = terminal && (ownsLatestRequest || source.issueStatus === "blocked") ? latestRun : null;
@@ -289,7 +253,10 @@ export function githubResponseCommentService(db: Db, fetchImpl = fetch) {
(await stagePullResponse(db, source)).result?.id === existing.result.id)) {
// Rotate no-op maintenance checks so a page of newer queued asks
// cannot hide other conversations that need terminal reconciliation.
await lease.commit(async tx => { await tx.update(chatActions).set({ updatedAt: new Date() }).where(eq(chatActions.id, existing.id)); });
await lease.commit(async tx => { await tx.update(chatActions).set({
status: ["received", "failed"].includes(existing.status) ? "processed" : existing.status,
updatedAt: new Date(),
}).where(and(eq(chatActions.id, existing.id), eq(chatActions.status, existing.status))); });
return { id: String(existing.result.id), url: String(existing.result.url) };
}
if (finalBody === undefined) {
@@ -439,6 +406,9 @@ export function githubResponseCommentService(db: Db, fetchImpl = fetch) {
or(eq(chatActions.status, "received"), and(eq(chatActions.status, "failed"), sql`${chatActions.result}->>'retryable' = 'true'`,
sql`(${chatActions.result}->>'retryAt')::timestamptz <= now()`)))).orderBy(asc(chatActions.createdAt)).limit(limit);
for (const row of rows) if (row.deliveryId) await tryAcknowledge(row.deliveryId);
return rows.length + await processTerminalResponses(limit);
}
async function processTerminalResponses(limit = 10) {
// Recovery can stop execution before a milestone publication is staged.
// Revisit acknowledged requests with terminal execution, even after a
// restart; successful acknowledgement is not successful task completion.
@@ -463,21 +433,28 @@ export function githubResponseCommentService(db: Db, fetchImpl = fetch) {
or coalesce(r.context_snapshot->'wakeCommentIds', '[]'::jsonb) ? l.comment_id::text))))`,
)).orderBy(asc(chatActions.updatedAt)).limit(limit);
for (const row of stopped) if (row.deliveryId) await tryAcknowledge(row.deliveryId);
return rows.length + stopped.length;
return stopped.length;
}
async function tryAcknowledge(deliveryId: string) {
try { return await acknowledge(deliveryId); }
async function tryAcknowledge(deliveryId: string, canContinue = () => true) {
try { return await acknowledge(deliveryId, undefined, undefined, canContinue); }
catch (error) {
if (error === stoppedAcknowledgement || !canContinue()) return;
const [row] = await db.select().from(chatActions).where(and(eq(chatActions.kind, "github_response_comment"), eq(chatActions.deliveryId, deliveryId)));
if (!row || row.result?.final === true) return;
const attempts = Number(row.result?.attempts ?? 0) + 1;
const denied = error instanceof Error && "status" in error && error.status === 403;
await db.update(chatActions).set({ status: denied ? "cancelled" : "failed", result: {
...row.result, attempts, retryable: !denied && attempts < 8,
retryAt: new Date(Date.now() + Math.min(300000, 1000 * 2 ** attempts)).toISOString(),
code: denied ? "authorization_changed" : "acknowledgement_unconfirmed",
}, updatedAt: new Date() }).where(and(eq(chatActions.id, row.id), sql`coalesce(${chatActions.result}->>'final', 'false') <> 'true'`));
await db.transaction(async tx => {
await notifyChatResponseAvailability(tx, { companyId: row.companyId, deliveryId });
await tx.update(chatActions).set({ status: denied ? "cancelled" : "failed", result: {
...row.result, attempts, retryable: !denied && attempts < 8,
retryAt: new Date(Date.now() + Math.min(300000, 1000 * 2 ** attempts)).toISOString(),
code: denied ? "authorization_changed" : "acknowledgement_unconfirmed",
}, updatedAt: new Date() }).where(and(eq(chatActions.id, row.id), sql`coalesce(${chatActions.result}->>'final', 'false') <> 'true'`));
});
}
}
return { tryAcknowledge, processPending, failRun, finishRun, hasWorkingResponseForRun };
return { tryAcknowledge, processPending, processTerminalResponses, failRun, finishRun, hasWorkingResponseForRun,
registerAcknowledgementWork: (coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) =>
registerGitHubAcknowledgementWork(db, coordinator, tryAcknowledge, canRun),
};
}
@@ -0,0 +1,7 @@
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export type ChatResponseScope = { companyId: string; issueId?: string; deliveryId?: string; conversationId?: string };
export async function notifyChatResponseAvailability(tx: object, scope: ChatResponseScope) {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubAcknowledgements, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, deliveryId: scope.deliveryId, conversationId: scope.conversationId }));
}
@@ -1,3 +1,4 @@
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
import { notifyChatConfirmationWork } from "./chat-confirmation-notifications.js";
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
import { notifyChatNativeProgress } from "./chat-native-progress-notifications.js";
@@ -15,4 +16,6 @@ export async function notifyChatRunMilestones(tx: object, scope: ChatRunMileston
// Admission and current chat authority also govern saved queue notices.
await notifyChatInboundWakeNotices(tx, scope);
await notifyChatConfirmationWork(tx, scope);
// Native presentation and run ownership can unblock a parked acknowledgement.
await notifyChatResponseAvailability(tx, scope);
}
@@ -1,3 +1,4 @@
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyChatConnectionConfiguration } from "./chat-channels/configuration-work-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
@@ -17,6 +18,7 @@ export async function notifyChatPublicationWork(
// release its provider response slot. Callers supply the affected issue only
// for those changes; transport claim/lease bookkeeping has no projection debt.
if (milestoneScope?.issueId) {
await notifyChatResponseAvailability(tx, { companyId: milestoneScope.companyId, issueId: milestoneScope.issueId });
await notifyChatRunMilestones(tx, { companyId: milestoneScope.companyId, issueId: milestoneScope.issueId });
}
}
@@ -41,6 +41,7 @@ export const DELIVERY_QUEUES = {
chatProviderEffects: "chat-provider-effects",
chatGitHubIngress: "chat-github-ingress",
chatGitHubPublications: "chat-github-publications",
chatGitHubAcknowledgements: "chat-github-acknowledgements",
chatFailedRunRetries: "chat-failed-run-retries",
chatTelegramMaintenance: "chat-telegram-maintenance",
chatReceivers: "chat-receivers",
+21 -14
View File
@@ -1,3 +1,4 @@
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
import { insertIssueComment } from "./issue-comment-writes.js";
import { randomUUID } from "node:crypto";
import { sanitizeQuarantinedCommentForHigherTrust } from "./source-trust.js";
@@ -195,6 +196,7 @@ export async function enqueueFastResponse(tx: Db, source: FastResponseSource) {
if (source.acceptedAt.getTime() + FAST_RESPONSE_DEADLINE_MS <= Date.now())
return;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.fastResponse);
if (source.endpointId && source.deliveryId) await notifyChatResponseAvailability(tx, { companyId: source.companyId, deliveryId: source.deliveryId });
await tx
.insert(fastResponseRequests)
.values({
@@ -549,20 +551,23 @@ export function fastResponseService(
}
}
async function skip(request: Request, reason: string) {
await db
.update(fastResponseRequests)
.set({
status: "skipped",
publicationStatus: "suppressed",
errorCode: reason,
finishedAt: new Date(),
})
.where(
and(
eq(fastResponseRequests.id, request.id),
eq(fastResponseRequests.status, "pending"),
),
);
await db.transaction(async tx => {
if (request.deliveryId) await notifyChatResponseAvailability(tx, { companyId: request.companyId, deliveryId: request.deliveryId });
await tx
.update(fastResponseRequests)
.set({
status: "skipped",
publicationStatus: "suppressed",
errorCode: reason,
finishedAt: new Date(),
})
.where(
and(
eq(fastResponseRequests.id, request.id),
eq(fastResponseRequests.status, "pending"),
),
);
});
}
async function settle(request: Request, outcome: FastResponseOutcome) {
await withBudgetSpan("budget.accounting.settlement", () => withAccountingTransaction(
@@ -625,6 +630,7 @@ export function fastResponseService(
},
);
const finishedAt = new Date();
if (request.deliveryId) await notifyChatResponseAvailability(tx, { companyId: request.companyId, deliveryId: request.deliveryId });
await tx
.update(fastResponseRequests)
.set({
@@ -993,6 +999,7 @@ export function fastResponseService(
idempotencyKey: `fast-response:${request.id}`,
payload: safe,
});
if (request.deliveryId) await notifyChatResponseAvailability(tx, { companyId: request.companyId, deliveryId: request.deliveryId });
await tx
.update(fastResponseRequests)
.set({
@@ -346,6 +346,19 @@ function controllerLeaseLossResultJson(signal: AbortSignal) {
: {};
}
/** Persist the resolved presentation and wake dependents in its committing transaction. */
export async function recordRunPresentationDecision(db: Db, input: {
companyId: string; runId: string; issueId?: string; decision: RunPresentationDecision;
}) {
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId });
await tx.update(heartbeatRuns).set({
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ presentationDecision: input.decision })}::jsonb`,
updatedAt: new Date(),
}).where(eq(heartbeatRuns.id, input.runId));
});
}
export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunCompletionDependencies) {
const {
getRun,
@@ -892,13 +905,8 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
};
}
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: livenessRun.companyId, issueId: issueId ?? undefined });
await tx.update(heartbeatRuns).set({
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ presentationDecision })}::jsonb`,
updatedAt: new Date(),
}).where(eq(heartbeatRuns.id, livenessRun.id));
});
await recordRunPresentationDecision(db, { companyId: livenessRun.companyId,
runId: livenessRun.id, issueId: issueId ?? undefined, decision: presentationDecision });
await appendRunEvent(livenessRun, {
eventType: "run.presentation.resolved",
stream: "system",