From 8aaf98315107ddf3dc045605f0989396d61e85cb Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 12:39:36 -0600 Subject: [PATCH] perf(db): send a transaction's work notifications in one statement at commit (#21436) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server signals queue work with database work signals, and it sends a PostgreSQL notification for each signal so that another server can wake during a deploy overlap > - Each signal sends its own `select pg_notify($1, $2)`, inside the open transaction, and waits for it > - On staging this is the most-called statement in the database: 31.8 per second, 15.1 million calls in total against 7.5 million `begin`, about 3.7 awaits per transaction > - Every await inside an open transaction holds its locks and its connection for one more turn of a saturated event loop > - This pull request collects a transaction's notifications and sends them in one statement just before commit > - The benefit is at most one notification round trip per transaction, with the same payloads delivered at the same commit ## Linked Issues or Issue Description **What existing behavior does this improve?** Database work signals send one `pg_notify` statement per signal, inside the transaction. **Subsystem affected** `packages/db` work signals, used by every delivery queue in the server. **Current behavior** `signalDatabaseWork` and `notifyDatabaseWorkOnCommit` each run `select pg_notify(channel, payload)` on the transaction and wait for it. `notifyIssueRecovery` alone signals 7 queues per call, and a transaction often calls it several times. A local legacy run (wake, start, 20 output lines, finish) sends 424 notification statements. Staging measures 31.8 per second, 8.5 `begin` per second. **Proposed behavior** A signal inside a transaction records its payload on that transaction. Before the outermost commit, the transaction sends every distinct payload in one statement. Payloads and the channel are byte-identical to today's, and PostgreSQL still delivers them only on commit. **Reason and benefit** At most one notification statement per transaction instead of one per signal. Fewer awaits while a transaction holds its locks and its connection. **Breaking changes** None. The listener protocol and the payload format do not change. ## What Changed - `packages/db/src/work-signals.ts`: a signal records `(channel, payload)` on the current transaction level. A released savepoint merges its records into its parent; a rolled-back savepoint drops them. The outermost transaction sends all distinct payloads in one statement (`select pg_notify(channel, payload) from jsonb_array_elements_text($2::jsonb)`) after its callback returns and before commit. A transaction that signals nothing sends no extra statement. - The record lives on the transaction object only. A retried callback gets a new transaction object and starts empty. - `publishDatabaseWorkHint` (outside a transaction) is unchanged and still sends at once. - `packages/db/src/work-signals.test.ts`: tests on real PostgreSQL with a LISTEN client. ## Verification - `pnpm --filter @paperclipai/db exec vitest run src/work-signals.test.ts`: 22 pass. 6 of the new tests fail on master. - New tests (real PostgreSQL, LISTEN client, asserting what is delivered as well as what is sent): - Distinct topics in one transaction: one statement at commit. The payloads delivered are identical to the format on master (`{version, origin, topic}`), in first-signal order. - A transaction that signals nothing: no notification statement. - Savepoints: a released savepoint two levels deep keeps its topic. A rolled-back savepoint drops its topic. A topic from a rolled-back savepoint that the outer level signals again is still delivered. - The callback throws: nothing is sent or delivered. - The commit-time send fails: the transaction fails and rolls back its write, and nothing is delivered. - 500 distinct topics: one statement, all 500 delivered. - Sequential, concurrent and retried transactions each start clean. - Server suites that use notifications pass: `deployment-notifications`, `run-start-handover`, `heartbeat-run-start-handover`, `agent-runtime-hooks`, `legacy-controller-lease`, `heartbeat-run-dispatch-lifecycle` (99 tests). Two of them hit statement timeouts once on a loaded machine and then passed 41/41 when run alone. - Before and after, one local legacy run (embedded PostgreSQL, process adapter, 20 output lines 50 ms apart, uninstrumented pass, alternating arms): `TSX_DISABLE_CACHE=1 UNITS=legacy TASKS=50 AGENTS=3 NODE_OPTIONS=--expose-gc pnpm --filter @paperclipai/server exec tsx scripts/query-cost/drive.ts` (harness from #21413) | Per legacy run | Before (2 runs) | After (2 runs) | | --- | --- | --- | | Statements | 883, 883 | 474, 491 | | `pg_notify` statements | 424 | 21 (one per signalling transaction) | | Notification round trips per signalling transaction | about 20 | 1 | | Transactions (BEGIN plus COMMIT) | 72 | 72 | | Allocated | 81.1, 79.3 MB | 76.3, 74.3 MB | | Wall time | 5.0, 5.2 s | 4.1, 7.7 s (noisy: loaded machine) | Statement counts are exact and stable. Allocation fell by about 5 MB per run (about 6%). Wall time varies too much on this machine to compare. - Staging, measured by the coordinator (Neon statistics, 64 seconds): 31.8 `pg_notify` per second, 8.5 `begin` per second. At about 44 KB of allocation per statement, notifications cost about 1.4 MB/s, about 2.4% of the 57.6 MB/s. The allocation saving is part of that, about 2%, and it is not the reason for this change. The reason is round trips inside open transactions: at most one notification statement per transaction instead of about 3.7. The per-second effect on staging is an estimate until it is measured after deploy. ## Risks - A failure of the notification send now surfaces at commit time, not at the signal call. No caller catches errors around a signal call and continues: the only `.catch` near a signal (`notifyBudgetStopsSettled`) wraps a root-level call that is not in a transaction and already fails before this change. - Order within a transaction: notifications are sent in first-signal order. A savepoint's records join its parent when the savepoint is released, which matches the order of sequential code. Listeners do not depend on order: each payload is an independent hint, and PostgreSQL already removes duplicate payloads within a transaction. - The notification is sent after the transaction's last statement instead of in the middle. Delivery still happens only at commit, as before. Remaining work, not in this pull request: sending stops being useful when no other server listens. The listener is open only during a startup handoff, while this process owns a native execution, or during a run-start handover, and a server drops its own notifications. A later change could send only while an overlap is possible. It would have to prove this interleaving: a new server announces itself, the old server is still outside its own window, and the old server must start sending before the new server relies on a hint. Otherwise the new server waits for its next reconciliation pass. ## Model Used - Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no documentation describes the send timing) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- packages/db/src/work-signals.test.ts | 137 ++++++++++++++++++++++++++- packages/db/src/work-signals.ts | 37 ++++++-- 2 files changed, 164 insertions(+), 10 deletions(-) diff --git a/packages/db/src/work-signals.test.ts b/packages/db/src/work-signals.test.ts index 0a50429fd64..b98a868af96 100644 --- a/packages/db/src/work-signals.test.ts +++ b/packages/db/src/work-signals.test.ts @@ -3,7 +3,7 @@ import { drizzle } from "drizzle-orm/postgres-js"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { createDb, withDedicatedDbConnection } from "./client.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js"; -import { databaseWorkScopeKey, databaseWorkPending, installDatabaseWorkSignals, notifyDatabaseWorkOnCommit, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js"; +import { DATABASE_WORK_CHANNEL, databaseWorkScopeKey, databaseWorkPending, enableDatabaseWorkNotifications, installDatabaseWorkSignals, notifyDatabaseWorkOnCommit, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js"; const support = await getEmbeddedPostgresTestSupport(); (support.supported ? describe : describe.skip)("database work signals", () => { @@ -221,3 +221,138 @@ describe("database work scope identity", () => { }); }); }); + +(support.supported ? describe : describe.skip)("database work notifications flushed once per transaction", () => { + let db: ReturnType; + let listener: ReturnType; + let temporary: Awaited>; + const delivered: string[] = []; + let notifyStatements = 0; + let sentPayloads: string[] = []; + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("work-signals-notify-"); + db = createDb(temporary.connectionString); + enableDatabaseWorkNotifications(db); + listener = createDb(temporary.connectionString, { maxConnections: 1 }); + await listener.$client.listen(DATABASE_WORK_CHANNEL, payload => { delivered.push(payload); }); + db.$client.options.debug = (_connection: number, query: string, parameters: unknown[]) => { + if (!/pg_notify/.test(query) || parameters.some(value => String(value).includes("fence"))) return; + notifyStatements += 1; + sentPayloads.push(...JSON.parse(String(parameters[parameters.length - 1])) as string[]); + }; + }, 90_000); + afterAll(async () => { + await listener?.$client.end({ timeout: 1 }); + await db?.$client.end({ timeout: 1 }); + await temporary?.cleanup(); + }); + const topic = (payload: string) => (JSON.parse(payload) as { topic: string }).topic; + async function run(action: (root: typeof db) => Promise) { + notifyStatements = 0; + sentPayloads = []; + delivered.length = 0; + let failure: unknown = null; + await action(db).catch(error => { failure = error; }); + await db.execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, ${JSON.stringify({ topic: "fence" })})`); + await vi.waitFor(() => expect(delivered.some(payload => topic(payload) === "fence")).toBe(true)); + const received = delivered.filter(payload => topic(payload) !== "fence"); + return { statements: notifyStatements, sent: sentPayloads.map(topic), delivered: received.map(topic), payloads: received, + failure: failure instanceof Error ? failure.message : failure }; + } + const rollback = () => { throw new Error("rollback"); }; + + it("sends every distinct topic of a transaction in one statement at commit, in first-signal order", async () => { + const result = await run(root => root.transaction(async tx => { + await signalDatabaseWork(tx, "a", "one"); + await signalDatabaseWork(tx, "b"); + await signalDatabaseWork(tx, "a", "two"); + await notifyDatabaseWorkOnCommit(tx, "c"); + expect(notifyStatements).toBe(0); + })); + expect(result).toMatchObject({ statements: 1, sent: ["a", "b", "c"], delivered: ["a", "b", "c"], failure: null }); + expect(result.payloads.map(payload => JSON.parse(payload))).toEqual(["a", "b", "c"].map(name => + ({ version: 1, origin: expect.any(String), topic: name }))); + }); + + it("sends nothing for a transaction that signals nothing", async () => { + expect(await run(root => root.transaction(async tx => { await tx.execute(sql`select 1`); }))) + .toMatchObject({ statements: 0, delivered: [] }); + }); + + it("keeps a released savepoint's topics, two levels deep, and drops a rolled-back savepoint's topics", async () => { + expect(await run(root => root.transaction(async tx => { + await tx.transaction(async first => { + await first.transaction(async second => { await signalDatabaseWork(second, "kept"); }); + await first.transaction(async second => { await signalDatabaseWork(second, "dropped"); rollback(); }).catch(() => {}); + }); + await tx.transaction(async nested => { await signalDatabaseWork(nested, "dropped-too"); rollback(); }).catch(() => {}); + }))).toMatchObject({ statements: 1, sent: ["kept"], delivered: ["kept"], failure: null }); + }); + + it("still sends a topic signalled in a rolled-back savepoint when the outer level signals it again", async () => { + expect(await run(root => root.transaction(async tx => { + await tx.transaction(async nested => { await signalDatabaseWork(nested, "a"); rollback(); }).catch(() => {}); + await signalDatabaseWork(tx, "a"); + }))).toMatchObject({ statements: 1, delivered: ["a"] }); + expect(await run(root => root.transaction(async tx => { + await signalDatabaseWork(tx, "a"); + await tx.transaction(async nested => { await signalDatabaseWork(nested, "a"); rollback(); }).catch(() => {}); + }))).toMatchObject({ statements: 1, delivered: ["a"] }); + }); + + it("sends nothing when the callback throws", async () => { + expect(await run(root => root.transaction(async tx => { await signalDatabaseWork(tx, "a"); rollback(); }))) + .toMatchObject({ statements: 0, delivered: [], failure: "rollback" }); + }); + + it("fails and rolls back the transaction when the commit-time send fails", async () => { + await db.execute(sql`create table if not exists work_signal_flush_test (id text primary key)`); + const result = await run(root => root.transaction(async tx => { + await tx.execute(sql`insert into work_signal_flush_test values ('flush')`); + await signalDatabaseWork(tx, "a"); + await tx.execute(sql`select 1 / 0`).catch(() => {}); + })); + expect(result.delivered).toEqual([]); + expect(result.statements).toBe(1); + expect(result.failure).toMatch(/pg_notify/); + expect(await db.execute(sql`select id from work_signal_flush_test where id = 'flush'`)).toHaveLength(0); + }); + + it("sends a large number of distinct topics in one bounded statement", async () => { + const names = Array.from({ length: 500 }, (_, index) => `topic-${index}`); + const result = await run(root => root.transaction(async tx => { for (const name of names) await signalDatabaseWork(tx, name); })); + expect(result.statements).toBe(1); + expect(result.delivered).toEqual(names); + }); + + it("rejects a signal recorded after its transaction or savepoint ended instead of losing it", async () => { + let outer: Parameters[0]>[0] | undefined; + let savepoint: typeof outer; + const result = await run(root => root.transaction(async tx => { + outer = tx; + await signalDatabaseWork(tx, "a"); + await tx.transaction(async nested => { savepoint = nested; await signalDatabaseWork(nested, "b"); }); + await expect(signalDatabaseWork(savepoint!, "late-savepoint")).rejects.toThrow("after its transaction or savepoint ended"); + await expect(notifyDatabaseWorkOnCommit(savepoint!, "late-savepoint")).rejects.toThrow("after its transaction or savepoint ended"); + })); + expect(result).toMatchObject({ statements: 1, delivered: ["a", "b"], failure: null }); + await expect(signalDatabaseWork(outer!, "late-outer")).rejects.toThrow("after its transaction or savepoint ended"); + await expect(notifyDatabaseWorkOnCommit(outer!, "late-outer")).rejects.toThrow("after its transaction or savepoint ended"); + }); + + it("starts every transaction clean, including concurrent transactions and a retried callback", async () => { + expect(await run(async root => { + await root.transaction(async tx => { await signalDatabaseWork(tx, "a"); }); + await root.transaction(async tx => { await signalDatabaseWork(tx, "a"); }); + await Promise.all([1, 2].map(() => root.transaction(async tx => { await signalDatabaseWork(tx, "b"); }))); + let attempts = 0; + const callback = async (tx: Parameters[0]>[0]) => { + attempts += 1; + await signalDatabaseWork(tx, "c"); + if (attempts === 1) rollback(); + }; + await root.transaction(callback).catch(() => {}); + await root.transaction(callback); + })).toMatchObject({ statements: 5, delivered: ["a", "a", "b", "b", "c"] }); + }); +}); diff --git a/packages/db/src/work-signals.ts b/packages/db/src/work-signals.ts index baf4ca24856..05b83a417cf 100644 --- a/packages/db/src/work-signals.ts +++ b/packages/db/src/work-signals.ts @@ -7,7 +7,7 @@ import { randomUUID } from "node:crypto"; type Listener = (event: "intent" | "settled", hints?: ReadonlySet) => void; type Transaction = { topics: Set; hints: Map>; xid?: Promise; uncertain: boolean }; type Scope = { key: object; listeners: Map>; transactions: Set; root: Connection; notificationOrigin: string; notify: boolean }; -type Context = { scope: Scope; transaction?: Transaction }; +type Context = { scope: Scope; transaction?: Transaction; pending?: Map; ended?: boolean }; type Connection = { transaction: (...args: any[]) => Promise; execute: (...args: any[]) => Promise }; const contexts = new WeakMap(); @@ -33,10 +33,21 @@ function install(connection: Connection, context: Context) { // savepoint must not wake a consumer before its parent commits. const state = context.transaction ?? { topics: new Set(), hints: new Map>(), uncertain: false }; try { + let level: Context | undefined; const result = await transaction.call(this, async (tx: Connection) => { - install(tx, { scope: context.scope, transaction: state }); - return callback(tx); + const current: Context = { scope: context.scope, transaction: state, pending: new Map() }; + level = current; + install(tx, current); + try { + const value = await callback(tx); + current.ended = true; + if (!context.transaction) await flushNotifications(tx, current.pending!); + return value; + } finally { current.ended = true; } }, ...options); + if (context.pending && level?.pending) { + for (const [key, payload] of level.pending) if (!context.pending.has(key)) context.pending.set(key, payload); + } if (!context.transaction) context.scope.transactions.delete(state); return result; } catch (error) { @@ -74,11 +85,19 @@ export async function signalDatabaseWork(connection: object, topic: string, hint // Keep a failed acquisition distinguishable from an uncertain queue write. try { await state.xid; } catch (error) { state.xid = undefined; throw error; } - if (context.scope.notify) { - // PostgreSQL delivers only after the outer commit, including savepoints. - // Queue rows remain authoritative; notification payloads contain no data. - await (connection as Connection).execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, ${notification(context.scope, topic)})`); - } + if (context.scope.notify) notifyAtCommit(context, notification(context.scope, topic)); +} + +function notifyAtCommit(context: Context, payload: string): void { + if (context.ended) throw new Error("Database work was signalled after its transaction or savepoint ended"); + const key = `${DATABASE_WORK_CHANNEL}\u0000${payload}`; + if (!context.pending!.has(key)) context.pending!.set(key, payload); +} + +async function flushNotifications(connection: Connection, pending: Map): Promise { + if (pending.size === 0) return; + await connection.execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, payload) + from jsonb_array_elements_text(${JSON.stringify([...pending.values()])}::jsonb) as notification(payload)`); } export const DATABASE_WORK_CHANNEL = "paperclip_work_v1"; @@ -101,7 +120,7 @@ export async function notifyDatabaseWorkOnCommit(connection: object, topic: stri const context = contexts.get(connection); if (!context?.scope.notify) return; if (!context.transaction) throw new Error("Database work must be registered inside a transaction"); - await (connection as Connection).execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, ${notification(context.scope, topic)})`); + notifyAtCommit(context, notification(context.scope, topic)); } /** A post-commit invalidation hint, for already-persisted live UI state. */