mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
perf(db): send a transaction's work notifications in one statement at commit (#21436)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server signals queue work with database work signals, and it
sends a PostgreSQL notification for each signal so that another server
can wake during a deploy overlap
> - Each signal sends its own `select pg_notify($1, $2)`, inside the
open transaction, and waits for it
> - On staging this is the most-called statement in the database: 31.8
per second, 15.1 million calls in total against 7.5 million `begin`,
about 3.7 awaits per transaction
> - Every await inside an open transaction holds its locks and its
connection for one more turn of a saturated event loop
> - This pull request collects a transaction's notifications and sends
them in one statement just before commit
> - The benefit is at most one notification round trip per transaction,
with the same payloads delivered at the same commit
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Database work signals send one `pg_notify` statement per signal, inside
the transaction.
**Subsystem affected**
`packages/db` work signals, used by every delivery queue in the server.
**Current behavior**
`signalDatabaseWork` and `notifyDatabaseWorkOnCommit` each run `select
pg_notify(channel, payload)` on the transaction and wait for it.
`notifyIssueRecovery` alone signals 7 queues per call, and a transaction
often calls it several times. A local legacy run (wake, start, 20 output
lines, finish) sends 424 notification statements. Staging measures 31.8
per second, 8.5 `begin` per second.
**Proposed behavior**
A signal inside a transaction records its payload on that transaction.
Before the outermost commit, the transaction sends every distinct
payload in one statement. Payloads and the channel are byte-identical to
today's, and PostgreSQL still delivers them only on commit.
**Reason and benefit**
At most one notification statement per transaction instead of one per
signal. Fewer awaits while a transaction holds its locks and its
connection.
**Breaking changes**
None. The listener protocol and the payload format do not change.
## What Changed
- `packages/db/src/work-signals.ts`: a signal records `(channel,
payload)` on the current transaction level. A released savepoint merges
its records into its parent; a rolled-back savepoint drops them. The
outermost transaction sends all distinct payloads in one statement
(`select pg_notify(channel, payload) from
jsonb_array_elements_text($2::jsonb)`) after its callback returns and
before commit. A transaction that signals nothing sends no extra
statement.
- The record lives on the transaction object only. A retried callback
gets a new transaction object and starts empty.
- `publishDatabaseWorkHint` (outside a transaction) is unchanged and
still sends at once.
- `packages/db/src/work-signals.test.ts`: tests on real PostgreSQL with
a LISTEN client.
## Verification
- `pnpm --filter @paperclipai/db exec vitest run
src/work-signals.test.ts`: 22 pass. 6 of the new tests fail on master.
- New tests (real PostgreSQL, LISTEN client, asserting what is delivered
as well as what is sent):
- Distinct topics in one transaction: one statement at commit. The
payloads delivered are identical to the format on master (`{version,
origin, topic}`), in first-signal order.
- A transaction that signals nothing: no notification statement.
- Savepoints: a released savepoint two levels deep keeps its topic. A
rolled-back savepoint drops its topic. A topic from a rolled-back
savepoint that the outer level signals again is still delivered.
- The callback throws: nothing is sent or delivered.
- The commit-time send fails: the transaction fails and rolls back its
write, and nothing is delivered.
- 500 distinct topics: one statement, all 500 delivered.
- Sequential, concurrent and retried transactions each start clean.
- Server suites that use notifications pass: `deployment-notifications`,
`run-start-handover`, `heartbeat-run-start-handover`,
`agent-runtime-hooks`, `legacy-controller-lease`,
`heartbeat-run-dispatch-lifecycle` (99 tests). Two of them hit statement
timeouts once on a loaded machine and then passed 41/41 when run alone.
- Before and after, one local legacy run (embedded PostgreSQL, process
adapter, 20 output lines 50 ms apart, uninstrumented pass, alternating
arms):
`TSX_DISABLE_CACHE=1 UNITS=legacy TASKS=50 AGENTS=3
NODE_OPTIONS=--expose-gc pnpm --filter @paperclipai/server exec tsx
scripts/query-cost/drive.ts` (harness from #21413)
| Per legacy run | Before (2 runs) | After (2 runs) |
| --- | --- | --- |
| Statements | 883, 883 | 474, 491 |
| `pg_notify` statements | 424 | 21 (one per signalling transaction) |
| Notification round trips per signalling transaction | about 20 | 1 |
| Transactions (BEGIN plus COMMIT) | 72 | 72 |
| Allocated | 81.1, 79.3 MB | 76.3, 74.3 MB |
| Wall time | 5.0, 5.2 s | 4.1, 7.7 s (noisy: loaded machine) |
Statement counts are exact and stable. Allocation fell by about 5 MB per
run (about 6%). Wall time varies too much on this machine to compare.
- Staging, measured by the coordinator (Neon statistics, 64 seconds):
31.8 `pg_notify` per second, 8.5 `begin` per second. At about 44 KB of
allocation per statement, notifications cost about 1.4 MB/s, about 2.4%
of the 57.6 MB/s. The allocation saving is part of that, about 2%, and
it is not the reason for this change. The reason is round trips inside
open transactions: at most one notification statement per transaction
instead of about 3.7. The per-second effect on staging is an estimate
until it is measured after deploy.
## Risks
- A failure of the notification send now surfaces at commit time, not at
the signal call. No caller catches errors around a signal call and
continues: the only `.catch` near a signal (`notifyBudgetStopsSettled`)
wraps a root-level call that is not in a transaction and already fails
before this change.
- Order within a transaction: notifications are sent in first-signal
order. A savepoint's records join its parent when the savepoint is
released, which matches the order of sequential code. Listeners do not
depend on order: each payload is an independent hint, and PostgreSQL
already removes duplicate payloads within a transaction.
- The notification is sent after the transaction's last statement
instead of in the middle. Delivery still happens only at commit, as
before.
Remaining work, not in this pull request: sending stops being useful
when no other server listens. The listener is open only during a startup
handoff, while this process owns a native execution, or during a
run-start handover, and a server drops its own notifications. A later
change could send only while an overlap is possible. It would have to
prove this interleaving: a new server announces itself, the old server
is still outside its own window, and the old server must start sending
before the new server relies on a hint. Otherwise the new server waits
for its next reconciliation pass.
## Model Used
- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
documentation describes the send timing)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c6b497d831
commit
8aaf983151
2 files changed
+164
-10
No files matched your search
@@ -3,7 +3,7 @@ import { drizzle } from "drizzle-orm/postgres-js";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { createDb, withDedicatedDbConnection } from "./client.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js";
|
||||
import { databaseWorkScopeKey, databaseWorkPending, installDatabaseWorkSignals, notifyDatabaseWorkOnCommit, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js";
|
||||
import { DATABASE_WORK_CHANNEL, databaseWorkScopeKey, databaseWorkPending, enableDatabaseWorkNotifications, installDatabaseWorkSignals, notifyDatabaseWorkOnCommit, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("database work signals", () => {
|
||||
@@ -221,3 +221,138 @@ describe("database work scope identity", () => {
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
(support.supported ? describe : describe.skip)("database work notifications flushed once per transaction", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let listener: ReturnType<typeof createDb>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
const delivered: string[] = [];
|
||||
let notifyStatements = 0;
|
||||
let sentPayloads: string[] = [];
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("work-signals-notify-");
|
||||
db = createDb(temporary.connectionString);
|
||||
enableDatabaseWorkNotifications(db);
|
||||
listener = createDb(temporary.connectionString, { maxConnections: 1 });
|
||||
await listener.$client.listen(DATABASE_WORK_CHANNEL, payload => { delivered.push(payload); });
|
||||
db.$client.options.debug = (_connection: number, query: string, parameters: unknown[]) => {
|
||||
if (!/pg_notify/.test(query) || parameters.some(value => String(value).includes("fence"))) return;
|
||||
notifyStatements += 1;
|
||||
sentPayloads.push(...JSON.parse(String(parameters[parameters.length - 1])) as string[]);
|
||||
};
|
||||
}, 90_000);
|
||||
afterAll(async () => {
|
||||
await listener?.$client.end({ timeout: 1 });
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await temporary?.cleanup();
|
||||
});
|
||||
const topic = (payload: string) => (JSON.parse(payload) as { topic: string }).topic;
|
||||
async function run(action: (root: typeof db) => Promise<unknown>) {
|
||||
notifyStatements = 0;
|
||||
sentPayloads = [];
|
||||
delivered.length = 0;
|
||||
let failure: unknown = null;
|
||||
await action(db).catch(error => { failure = error; });
|
||||
await db.execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, ${JSON.stringify({ topic: "fence" })})`);
|
||||
await vi.waitFor(() => expect(delivered.some(payload => topic(payload) === "fence")).toBe(true));
|
||||
const received = delivered.filter(payload => topic(payload) !== "fence");
|
||||
return { statements: notifyStatements, sent: sentPayloads.map(topic), delivered: received.map(topic), payloads: received,
|
||||
failure: failure instanceof Error ? failure.message : failure };
|
||||
}
|
||||
const rollback = () => { throw new Error("rollback"); };
|
||||
|
||||
it("sends every distinct topic of a transaction in one statement at commit, in first-signal order", async () => {
|
||||
const result = await run(root => root.transaction(async tx => {
|
||||
await signalDatabaseWork(tx, "a", "one");
|
||||
await signalDatabaseWork(tx, "b");
|
||||
await signalDatabaseWork(tx, "a", "two");
|
||||
await notifyDatabaseWorkOnCommit(tx, "c");
|
||||
expect(notifyStatements).toBe(0);
|
||||
}));
|
||||
expect(result).toMatchObject({ statements: 1, sent: ["a", "b", "c"], delivered: ["a", "b", "c"], failure: null });
|
||||
expect(result.payloads.map(payload => JSON.parse(payload))).toEqual(["a", "b", "c"].map(name =>
|
||||
({ version: 1, origin: expect.any(String), topic: name })));
|
||||
});
|
||||
|
||||
it("sends nothing for a transaction that signals nothing", async () => {
|
||||
expect(await run(root => root.transaction(async tx => { await tx.execute(sql`select 1`); })))
|
||||
.toMatchObject({ statements: 0, delivered: [] });
|
||||
});
|
||||
|
||||
it("keeps a released savepoint's topics, two levels deep, and drops a rolled-back savepoint's topics", async () => {
|
||||
expect(await run(root => root.transaction(async tx => {
|
||||
await tx.transaction(async first => {
|
||||
await first.transaction(async second => { await signalDatabaseWork(second, "kept"); });
|
||||
await first.transaction(async second => { await signalDatabaseWork(second, "dropped"); rollback(); }).catch(() => {});
|
||||
});
|
||||
await tx.transaction(async nested => { await signalDatabaseWork(nested, "dropped-too"); rollback(); }).catch(() => {});
|
||||
}))).toMatchObject({ statements: 1, sent: ["kept"], delivered: ["kept"], failure: null });
|
||||
});
|
||||
|
||||
it("still sends a topic signalled in a rolled-back savepoint when the outer level signals it again", async () => {
|
||||
expect(await run(root => root.transaction(async tx => {
|
||||
await tx.transaction(async nested => { await signalDatabaseWork(nested, "a"); rollback(); }).catch(() => {});
|
||||
await signalDatabaseWork(tx, "a");
|
||||
}))).toMatchObject({ statements: 1, delivered: ["a"] });
|
||||
expect(await run(root => root.transaction(async tx => {
|
||||
await signalDatabaseWork(tx, "a");
|
||||
await tx.transaction(async nested => { await signalDatabaseWork(nested, "a"); rollback(); }).catch(() => {});
|
||||
}))).toMatchObject({ statements: 1, delivered: ["a"] });
|
||||
});
|
||||
|
||||
it("sends nothing when the callback throws", async () => {
|
||||
expect(await run(root => root.transaction(async tx => { await signalDatabaseWork(tx, "a"); rollback(); })))
|
||||
.toMatchObject({ statements: 0, delivered: [], failure: "rollback" });
|
||||
});
|
||||
|
||||
it("fails and rolls back the transaction when the commit-time send fails", async () => {
|
||||
await db.execute(sql`create table if not exists work_signal_flush_test (id text primary key)`);
|
||||
const result = await run(root => root.transaction(async tx => {
|
||||
await tx.execute(sql`insert into work_signal_flush_test values ('flush')`);
|
||||
await signalDatabaseWork(tx, "a");
|
||||
await tx.execute(sql`select 1 / 0`).catch(() => {});
|
||||
}));
|
||||
expect(result.delivered).toEqual([]);
|
||||
expect(result.statements).toBe(1);
|
||||
expect(result.failure).toMatch(/pg_notify/);
|
||||
expect(await db.execute(sql`select id from work_signal_flush_test where id = 'flush'`)).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("sends a large number of distinct topics in one bounded statement", async () => {
|
||||
const names = Array.from({ length: 500 }, (_, index) => `topic-${index}`);
|
||||
const result = await run(root => root.transaction(async tx => { for (const name of names) await signalDatabaseWork(tx, name); }));
|
||||
expect(result.statements).toBe(1);
|
||||
expect(result.delivered).toEqual(names);
|
||||
});
|
||||
|
||||
it("rejects a signal recorded after its transaction or savepoint ended instead of losing it", async () => {
|
||||
let outer: Parameters<Parameters<typeof db.transaction>[0]>[0] | undefined;
|
||||
let savepoint: typeof outer;
|
||||
const result = await run(root => root.transaction(async tx => {
|
||||
outer = tx;
|
||||
await signalDatabaseWork(tx, "a");
|
||||
await tx.transaction(async nested => { savepoint = nested; await signalDatabaseWork(nested, "b"); });
|
||||
await expect(signalDatabaseWork(savepoint!, "late-savepoint")).rejects.toThrow("after its transaction or savepoint ended");
|
||||
await expect(notifyDatabaseWorkOnCommit(savepoint!, "late-savepoint")).rejects.toThrow("after its transaction or savepoint ended");
|
||||
}));
|
||||
expect(result).toMatchObject({ statements: 1, delivered: ["a", "b"], failure: null });
|
||||
await expect(signalDatabaseWork(outer!, "late-outer")).rejects.toThrow("after its transaction or savepoint ended");
|
||||
await expect(notifyDatabaseWorkOnCommit(outer!, "late-outer")).rejects.toThrow("after its transaction or savepoint ended");
|
||||
});
|
||||
|
||||
it("starts every transaction clean, including concurrent transactions and a retried callback", async () => {
|
||||
expect(await run(async root => {
|
||||
await root.transaction(async tx => { await signalDatabaseWork(tx, "a"); });
|
||||
await root.transaction(async tx => { await signalDatabaseWork(tx, "a"); });
|
||||
await Promise.all([1, 2].map(() => root.transaction(async tx => { await signalDatabaseWork(tx, "b"); })));
|
||||
let attempts = 0;
|
||||
const callback = async (tx: Parameters<Parameters<typeof root.transaction>[0]>[0]) => {
|
||||
attempts += 1;
|
||||
await signalDatabaseWork(tx, "c");
|
||||
if (attempts === 1) rollback();
|
||||
};
|
||||
await root.transaction(callback).catch(() => {});
|
||||
await root.transaction(callback);
|
||||
})).toMatchObject({ statements: 5, delivered: ["a", "a", "b", "b", "c"] });
|
||||
});
|
||||
});
|
||||
@@ -7,7 +7,7 @@ import { randomUUID } from "node:crypto";
|
||||
type Listener = (event: "intent" | "settled", hints?: ReadonlySet<string>) => void;
|
||||
type Transaction = { topics: Set<string>; hints: Map<string, Set<string>>; xid?: Promise<string>; uncertain: boolean };
|
||||
type Scope = { key: object; listeners: Map<string, Set<Listener>>; transactions: Set<Transaction>; root: Connection; notificationOrigin: string; notify: boolean };
|
||||
type Context = { scope: Scope; transaction?: Transaction };
|
||||
type Context = { scope: Scope; transaction?: Transaction; pending?: Map<string, string>; ended?: boolean };
|
||||
type Connection = { transaction: (...args: any[]) => Promise<any>; execute: (...args: any[]) => Promise<any> };
|
||||
const contexts = new WeakMap<object, Context>();
|
||||
|
||||
@@ -33,10 +33,21 @@ function install(connection: Connection, context: Context) {
|
||||
// savepoint must not wake a consumer before its parent commits.
|
||||
const state = context.transaction ?? { topics: new Set<string>(), hints: new Map<string, Set<string>>(), uncertain: false };
|
||||
try {
|
||||
let level: Context | undefined;
|
||||
const result = await transaction.call(this, async (tx: Connection) => {
|
||||
install(tx, { scope: context.scope, transaction: state });
|
||||
return callback(tx);
|
||||
const current: Context = { scope: context.scope, transaction: state, pending: new Map() };
|
||||
level = current;
|
||||
install(tx, current);
|
||||
try {
|
||||
const value = await callback(tx);
|
||||
current.ended = true;
|
||||
if (!context.transaction) await flushNotifications(tx, current.pending!);
|
||||
return value;
|
||||
} finally { current.ended = true; }
|
||||
}, ...options);
|
||||
if (context.pending && level?.pending) {
|
||||
for (const [key, payload] of level.pending) if (!context.pending.has(key)) context.pending.set(key, payload);
|
||||
}
|
||||
if (!context.transaction) context.scope.transactions.delete(state);
|
||||
return result;
|
||||
} catch (error) {
|
||||
@@ -74,11 +85,19 @@ export async function signalDatabaseWork(connection: object, topic: string, hint
|
||||
// Keep a failed acquisition distinguishable from an uncertain queue write.
|
||||
try { await state.xid; }
|
||||
catch (error) { state.xid = undefined; throw error; }
|
||||
if (context.scope.notify) {
|
||||
// PostgreSQL delivers only after the outer commit, including savepoints.
|
||||
// Queue rows remain authoritative; notification payloads contain no data.
|
||||
await (connection as Connection).execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, ${notification(context.scope, topic)})`);
|
||||
}
|
||||
if (context.scope.notify) notifyAtCommit(context, notification(context.scope, topic));
|
||||
}
|
||||
|
||||
function notifyAtCommit(context: Context, payload: string): void {
|
||||
if (context.ended) throw new Error("Database work was signalled after its transaction or savepoint ended");
|
||||
const key = `${DATABASE_WORK_CHANNEL}\u0000${payload}`;
|
||||
if (!context.pending!.has(key)) context.pending!.set(key, payload);
|
||||
}
|
||||
|
||||
async function flushNotifications(connection: Connection, pending: Map<string, string>): Promise<void> {
|
||||
if (pending.size === 0) return;
|
||||
await connection.execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, payload)
|
||||
from jsonb_array_elements_text(${JSON.stringify([...pending.values()])}::jsonb) as notification(payload)`);
|
||||
}
|
||||
|
||||
export const DATABASE_WORK_CHANNEL = "paperclip_work_v1";
|
||||
@@ -101,7 +120,7 @@ export async function notifyDatabaseWorkOnCommit(connection: object, topic: stri
|
||||
const context = contexts.get(connection);
|
||||
if (!context?.scope.notify) return;
|
||||
if (!context.transaction) throw new Error("Database work must be registered inside a transaction");
|
||||
await (connection as Connection).execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, ${notification(context.scope, topic)})`);
|
||||
notifyAtCommit(context, notification(context.scope, topic));
|
||||
}
|
||||
|
||||
/** A post-commit invalidation hint, for already-persisted live UI state. */
|
||||
|
||||
Reference in new issue
Block a user