perf(db): send a transaction's work notifications in one statement at commit (#21436)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server signals queue work with database work signals, and it
sends a PostgreSQL notification for each signal so that another server
can wake during a deploy overlap
> - Each signal sends its own `select pg_notify($1, $2)`, inside the
open transaction, and waits for it
> - On staging this is the most-called statement in the database: 31.8
per second, 15.1 million calls in total against 7.5 million `begin`,
about 3.7 awaits per transaction
> - Every await inside an open transaction holds its locks and its
connection for one more turn of a saturated event loop
> - This pull request collects a transaction's notifications and sends
them in one statement just before commit
> - The benefit is at most one notification round trip per transaction,
with the same payloads delivered at the same commit

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Database work signals send one `pg_notify` statement per signal, inside
the transaction.

**Subsystem affected**
`packages/db` work signals, used by every delivery queue in the server.

**Current behavior**
`signalDatabaseWork` and `notifyDatabaseWorkOnCommit` each run `select
pg_notify(channel, payload)` on the transaction and wait for it.
`notifyIssueRecovery` alone signals 7 queues per call, and a transaction
often calls it several times. A local legacy run (wake, start, 20 output
lines, finish) sends 424 notification statements. Staging measures 31.8
per second, 8.5 `begin` per second.

**Proposed behavior**
A signal inside a transaction records its payload on that transaction.
Before the outermost commit, the transaction sends every distinct
payload in one statement. Payloads and the channel are byte-identical to
today's, and PostgreSQL still delivers them only on commit.

**Reason and benefit**
At most one notification statement per transaction instead of one per
signal. Fewer awaits while a transaction holds its locks and its
connection.

**Breaking changes**
None. The listener protocol and the payload format do not change.

## What Changed

- `packages/db/src/work-signals.ts`: a signal records `(channel,
payload)` on the current transaction level. A released savepoint merges
its records into its parent; a rolled-back savepoint drops them. The
outermost transaction sends all distinct payloads in one statement
(`select pg_notify(channel, payload) from
jsonb_array_elements_text($2::jsonb)`) after its callback returns and
before commit. A transaction that signals nothing sends no extra
statement.
- The record lives on the transaction object only. A retried callback
gets a new transaction object and starts empty.
- `publishDatabaseWorkHint` (outside a transaction) is unchanged and
still sends at once.
- `packages/db/src/work-signals.test.ts`: tests on real PostgreSQL with
a LISTEN client.

## Verification

- `pnpm --filter @paperclipai/db exec vitest run
src/work-signals.test.ts`: 22 pass. 6 of the new tests fail on master.
- New tests (real PostgreSQL, LISTEN client, asserting what is delivered
as well as what is sent):
- Distinct topics in one transaction: one statement at commit. The
payloads delivered are identical to the format on master (`{version,
origin, topic}`), in first-signal order.
  - A transaction that signals nothing: no notification statement.
- Savepoints: a released savepoint two levels deep keeps its topic. A
rolled-back savepoint drops its topic. A topic from a rolled-back
savepoint that the outer level signals again is still delivered.
  - The callback throws: nothing is sent or delivered.
- The commit-time send fails: the transaction fails and rolls back its
write, and nothing is delivered.
  - 500 distinct topics: one statement, all 500 delivered.
  - Sequential, concurrent and retried transactions each start clean.
- Server suites that use notifications pass: `deployment-notifications`,
`run-start-handover`, `heartbeat-run-start-handover`,
`agent-runtime-hooks`, `legacy-controller-lease`,
`heartbeat-run-dispatch-lifecycle` (99 tests). Two of them hit statement
timeouts once on a loaded machine and then passed 41/41 when run alone.
- Before and after, one local legacy run (embedded PostgreSQL, process
adapter, 20 output lines 50 ms apart, uninstrumented pass, alternating
arms):
`TSX_DISABLE_CACHE=1 UNITS=legacy TASKS=50 AGENTS=3
NODE_OPTIONS=--expose-gc pnpm --filter @paperclipai/server exec tsx
scripts/query-cost/drive.ts` (harness from #21413)

  | Per legacy run | Before (2 runs) | After (2 runs) |
  | --- | --- | --- |
  | Statements | 883, 883 | 474, 491 |
  | `pg_notify` statements | 424 | 21 (one per signalling transaction) |
  | Notification round trips per signalling transaction | about 20 | 1 |
  | Transactions (BEGIN plus COMMIT) | 72 | 72 |
  | Allocated | 81.1, 79.3 MB | 76.3, 74.3 MB |
  | Wall time | 5.0, 5.2 s | 4.1, 7.7 s (noisy: loaded machine) |

Statement counts are exact and stable. Allocation fell by about 5 MB per
run (about 6%). Wall time varies too much on this machine to compare.
- Staging, measured by the coordinator (Neon statistics, 64 seconds):
31.8 `pg_notify` per second, 8.5 `begin` per second. At about 44 KB of
allocation per statement, notifications cost about 1.4 MB/s, about 2.4%
of the 57.6 MB/s. The allocation saving is part of that, about 2%, and
it is not the reason for this change. The reason is round trips inside
open transactions: at most one notification statement per transaction
instead of about 3.7. The per-second effect on staging is an estimate
until it is measured after deploy.

## Risks

- A failure of the notification send now surfaces at commit time, not at
the signal call. No caller catches errors around a signal call and
continues: the only `.catch` near a signal (`notifyBudgetStopsSettled`)
wraps a root-level call that is not in a transaction and already fails
before this change.
- Order within a transaction: notifications are sent in first-signal
order. A savepoint's records join its parent when the savepoint is
released, which matches the order of sequential code. Listeners do not
depend on order: each payload is an independent hint, and PostgreSQL
already removes duplicate payloads within a transaction.
- The notification is sent after the transaction's last statement
instead of in the middle. Delivery still happens only at commit, as
before.

Remaining work, not in this pull request: sending stops being useful
when no other server listens. The listener is open only during a startup
handoff, while this process owns a native execution, or during a
run-start handover, and a server drops its own notifications. A later
change could send only while an overlap is possible. It would have to
prove this interleaving: a new server announces itself, the old server
is still outside its own window, and the old server must start sending
before the new server relies on a hint. Otherwise the new server waits
for its next reconciliation pass.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
documentation describes the send timing)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 12:39:36 -06:00
1 parent c6b497d831
commit 8aaf983151
2 files changed
+164 -10

No files matched your search

+136 -1
View File
@@ -3,7 +3,7 @@ import { drizzle } from "drizzle-orm/postgres-js";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { createDb, withDedicatedDbConnection } from "./client.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js";
import { databaseWorkScopeKey, databaseWorkPending, installDatabaseWorkSignals, notifyDatabaseWorkOnCommit, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js";
import { DATABASE_WORK_CHANNEL, databaseWorkScopeKey, databaseWorkPending, enableDatabaseWorkNotifications, installDatabaseWorkSignals, notifyDatabaseWorkOnCommit, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js";
const support = await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("database work signals", () => {
@@ -221,3 +221,138 @@ describe("database work scope identity", () => {
});
});
});
(support.supported ? describe : describe.skip)("database work notifications flushed once per transaction", () => {
let db: ReturnType<typeof createDb>;
let listener: ReturnType<typeof createDb>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
const delivered: string[] = [];
let notifyStatements = 0;
let sentPayloads: string[] = [];
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("work-signals-notify-");
db = createDb(temporary.connectionString);
enableDatabaseWorkNotifications(db);
listener = createDb(temporary.connectionString, { maxConnections: 1 });
await listener.$client.listen(DATABASE_WORK_CHANNEL, payload => { delivered.push(payload); });
db.$client.options.debug = (_connection: number, query: string, parameters: unknown[]) => {
if (!/pg_notify/.test(query) || parameters.some(value => String(value).includes("fence"))) return;
notifyStatements += 1;
sentPayloads.push(...JSON.parse(String(parameters[parameters.length - 1])) as string[]);
};
}, 90_000);
afterAll(async () => {
await listener?.$client.end({ timeout: 1 });
await db?.$client.end({ timeout: 1 });
await temporary?.cleanup();
});
const topic = (payload: string) => (JSON.parse(payload) as { topic: string }).topic;
async function run(action: (root: typeof db) => Promise<unknown>) {
notifyStatements = 0;
sentPayloads = [];
delivered.length = 0;
let failure: unknown = null;
await action(db).catch(error => { failure = error; });
await db.execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, ${JSON.stringify({ topic: "fence" })})`);
await vi.waitFor(() => expect(delivered.some(payload => topic(payload) === "fence")).toBe(true));
const received = delivered.filter(payload => topic(payload) !== "fence");
return { statements: notifyStatements, sent: sentPayloads.map(topic), delivered: received.map(topic), payloads: received,
failure: failure instanceof Error ? failure.message : failure };
}
const rollback = () => { throw new Error("rollback"); };
it("sends every distinct topic of a transaction in one statement at commit, in first-signal order", async () => {
const result = await run(root => root.transaction(async tx => {
await signalDatabaseWork(tx, "a", "one");
await signalDatabaseWork(tx, "b");
await signalDatabaseWork(tx, "a", "two");
await notifyDatabaseWorkOnCommit(tx, "c");
expect(notifyStatements).toBe(0);
}));
expect(result).toMatchObject({ statements: 1, sent: ["a", "b", "c"], delivered: ["a", "b", "c"], failure: null });
expect(result.payloads.map(payload => JSON.parse(payload))).toEqual(["a", "b", "c"].map(name =>
({ version: 1, origin: expect.any(String), topic: name })));
});
it("sends nothing for a transaction that signals nothing", async () => {
expect(await run(root => root.transaction(async tx => { await tx.execute(sql`select 1`); })))
.toMatchObject({ statements: 0, delivered: [] });
});
it("keeps a released savepoint's topics, two levels deep, and drops a rolled-back savepoint's topics", async () => {
expect(await run(root => root.transaction(async tx => {
await tx.transaction(async first => {
await first.transaction(async second => { await signalDatabaseWork(second, "kept"); });
await first.transaction(async second => { await signalDatabaseWork(second, "dropped"); rollback(); }).catch(() => {});
});
await tx.transaction(async nested => { await signalDatabaseWork(nested, "dropped-too"); rollback(); }).catch(() => {});
}))).toMatchObject({ statements: 1, sent: ["kept"], delivered: ["kept"], failure: null });
});
it("still sends a topic signalled in a rolled-back savepoint when the outer level signals it again", async () => {
expect(await run(root => root.transaction(async tx => {
await tx.transaction(async nested => { await signalDatabaseWork(nested, "a"); rollback(); }).catch(() => {});
await signalDatabaseWork(tx, "a");
}))).toMatchObject({ statements: 1, delivered: ["a"] });
expect(await run(root => root.transaction(async tx => {
await signalDatabaseWork(tx, "a");
await tx.transaction(async nested => { await signalDatabaseWork(nested, "a"); rollback(); }).catch(() => {});
}))).toMatchObject({ statements: 1, delivered: ["a"] });
});
it("sends nothing when the callback throws", async () => {
expect(await run(root => root.transaction(async tx => { await signalDatabaseWork(tx, "a"); rollback(); })))
.toMatchObject({ statements: 0, delivered: [], failure: "rollback" });
});
it("fails and rolls back the transaction when the commit-time send fails", async () => {
await db.execute(sql`create table if not exists work_signal_flush_test (id text primary key)`);
const result = await run(root => root.transaction(async tx => {
await tx.execute(sql`insert into work_signal_flush_test values ('flush')`);
await signalDatabaseWork(tx, "a");
await tx.execute(sql`select 1 / 0`).catch(() => {});
}));
expect(result.delivered).toEqual([]);
expect(result.statements).toBe(1);
expect(result.failure).toMatch(/pg_notify/);
expect(await db.execute(sql`select id from work_signal_flush_test where id = 'flush'`)).toHaveLength(0);
});
it("sends a large number of distinct topics in one bounded statement", async () => {
const names = Array.from({ length: 500 }, (_, index) => `topic-${index}`);
const result = await run(root => root.transaction(async tx => { for (const name of names) await signalDatabaseWork(tx, name); }));
expect(result.statements).toBe(1);
expect(result.delivered).toEqual(names);
});
it("rejects a signal recorded after its transaction or savepoint ended instead of losing it", async () => {
let outer: Parameters<Parameters<typeof db.transaction>[0]>[0] | undefined;
let savepoint: typeof outer;
const result = await run(root => root.transaction(async tx => {
outer = tx;
await signalDatabaseWork(tx, "a");
await tx.transaction(async nested => { savepoint = nested; await signalDatabaseWork(nested, "b"); });
await expect(signalDatabaseWork(savepoint!, "late-savepoint")).rejects.toThrow("after its transaction or savepoint ended");
await expect(notifyDatabaseWorkOnCommit(savepoint!, "late-savepoint")).rejects.toThrow("after its transaction or savepoint ended");
}));
expect(result).toMatchObject({ statements: 1, delivered: ["a", "b"], failure: null });
await expect(signalDatabaseWork(outer!, "late-outer")).rejects.toThrow("after its transaction or savepoint ended");
await expect(notifyDatabaseWorkOnCommit(outer!, "late-outer")).rejects.toThrow("after its transaction or savepoint ended");
});
it("starts every transaction clean, including concurrent transactions and a retried callback", async () => {
expect(await run(async root => {
await root.transaction(async tx => { await signalDatabaseWork(tx, "a"); });
await root.transaction(async tx => { await signalDatabaseWork(tx, "a"); });
await Promise.all([1, 2].map(() => root.transaction(async tx => { await signalDatabaseWork(tx, "b"); })));
let attempts = 0;
const callback = async (tx: Parameters<Parameters<typeof root.transaction>[0]>[0]) => {
attempts += 1;
await signalDatabaseWork(tx, "c");
if (attempts === 1) rollback();
};
await root.transaction(callback).catch(() => {});
await root.transaction(callback);
})).toMatchObject({ statements: 5, delivered: ["a", "a", "b", "b", "c"] });
});
});
+28 -9
View File
@@ -7,7 +7,7 @@ import { randomUUID } from "node:crypto";
type Listener = (event: "intent" | "settled", hints?: ReadonlySet<string>) => void;
type Transaction = { topics: Set<string>; hints: Map<string, Set<string>>; xid?: Promise<string>; uncertain: boolean };
type Scope = { key: object; listeners: Map<string, Set<Listener>>; transactions: Set<Transaction>; root: Connection; notificationOrigin: string; notify: boolean };
type Context = { scope: Scope; transaction?: Transaction };
type Context = { scope: Scope; transaction?: Transaction; pending?: Map<string, string>; ended?: boolean };
type Connection = { transaction: (...args: any[]) => Promise<any>; execute: (...args: any[]) => Promise<any> };
const contexts = new WeakMap<object, Context>();
@@ -33,10 +33,21 @@ function install(connection: Connection, context: Context) {
// savepoint must not wake a consumer before its parent commits.
const state = context.transaction ?? { topics: new Set<string>(), hints: new Map<string, Set<string>>(), uncertain: false };
try {
let level: Context | undefined;
const result = await transaction.call(this, async (tx: Connection) => {
install(tx, { scope: context.scope, transaction: state });
return callback(tx);
const current: Context = { scope: context.scope, transaction: state, pending: new Map() };
level = current;
install(tx, current);
try {
const value = await callback(tx);
current.ended = true;
if (!context.transaction) await flushNotifications(tx, current.pending!);
return value;
} finally { current.ended = true; }
}, ...options);
if (context.pending && level?.pending) {
for (const [key, payload] of level.pending) if (!context.pending.has(key)) context.pending.set(key, payload);
}
if (!context.transaction) context.scope.transactions.delete(state);
return result;
} catch (error) {
@@ -74,11 +85,19 @@ export async function signalDatabaseWork(connection: object, topic: string, hint
// Keep a failed acquisition distinguishable from an uncertain queue write.
try { await state.xid; }
catch (error) { state.xid = undefined; throw error; }
if (context.scope.notify) {
// PostgreSQL delivers only after the outer commit, including savepoints.
// Queue rows remain authoritative; notification payloads contain no data.
await (connection as Connection).execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, ${notification(context.scope, topic)})`);
}
if (context.scope.notify) notifyAtCommit(context, notification(context.scope, topic));
}
function notifyAtCommit(context: Context, payload: string): void {
if (context.ended) throw new Error("Database work was signalled after its transaction or savepoint ended");
const key = `${DATABASE_WORK_CHANNEL}\u0000${payload}`;
if (!context.pending!.has(key)) context.pending!.set(key, payload);
}
async function flushNotifications(connection: Connection, pending: Map<string, string>): Promise<void> {
if (pending.size === 0) return;
await connection.execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, payload)
from jsonb_array_elements_text(${JSON.stringify([...pending.values()])}::jsonb) as notification(payload)`);
}
export const DATABASE_WORK_CHANNEL = "paperclip_work_v1";
@@ -101,7 +120,7 @@ export async function notifyDatabaseWorkOnCommit(connection: object, topic: stri
const context = contexts.get(connection);
if (!context?.scope.notify) return;
if (!context.transaction) throw new Error("Database work must be registered inside a transaction");
await (connection as Connection).execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, ${notification(context.scope, topic)})`);
notifyAtCommit(context, notification(context.scope, topic));
}
/** A post-commit invalidation hint, for already-persisted live UI state. */