mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
perf(db): keep dedicated connections for the next burst (#21769)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process runs every agent on one Node main thread, and each new database connection costs a TLS handshake, about 0.36 ms of SCRAM key derivation on that thread, and a type-fetch query > - `withDedicatedDbConnection` gives lock holders a private session; since #21693 it keeps at most 4 released sessions and closes the rest > - When the same burst of more than 4 dedicated holders comes back, every holder above 4 opens a new connection again > - A harness on embedded PostgreSQL measured 504 new connections per 1,000 uses for bursts of 8 and 760 for bursts of 16 > - This pull request keeps released dedicated clients for reuse while idle plus in-use clients stay within `DATABASE_DEDICATED_CONNECTION_MAX`; the driver idle timeout still closes their sessions when they are not used > - The benefit is 8 and 16 new connections per 1,000 uses for the same bursts, with no change to any pool maximum or to `databaseConnectionBudget()` ## Linked Issues or Issue Description **What is the problem?** A staging measurement showed `SET application_name='paperclip-runner-checkpoint'` at 3.17/s and `SET application_name='paperclip'` at 3.86/s with 7 live runs. This looked like connection churn. The investigation found two separate facts: 1. The `SET` statements are not new client connections. postgres.js sends `application_name` in the startup packet, never as a `SET` statement (`StartupMessage` in postgres.js `src/connection.js`). PgBouncer in transaction mode sends `SET application_name` itself when it gives a client transaction a server connection that a client with another `application_name` used last. With one shared server pool, every checkpoint transaction that lands after a main-pool transaction costs one `SET`, and the reverse. The rate follows checkpoint transactions, not connects. The checkpoint pool opens at most 3 connections per 1,000 saves (measured below). 2. Dedicated connections did churn by construction for recurring bursts of more than 4 holders (measured below). This pull request fixes that. **Steps to reproduce** Run `packages/db/src/connection-reuse-census.test.ts`. It counts `pg_stat_database.sessions` around 1,000 operations. **Expected behavior** A recurring burst reuses the sessions that the previous burst opened. ## What Changed - `withDedicatedDbConnection` keeps a released client idle when fewer than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle, as before, or when idle plus in-use clients stay within the dedicated limit (`DATABASE_DEDICATED_CONNECTION_MAX`, default 24). A client whose action failed, an ended client and clients after shutdown are still closed. - Each retained client keeps the main pool's `idle_timeout` (default 60 s), so its session closes after that idle time, as the main pool's do. The next use reconnects. - The worst case stays at `2 x DATABASE_DEDICATED_CONNECTION_MAX + 4`: a new client opens only when no client is idle, and a client is kept above 4 only while idle plus in-use is within the limit. `databaseConnectionBudget()` and every pool maximum are unchanged. - New `connection-reuse-census.test.ts` measures connections opened per 1,000 checkpoint saves and per 1,000 dedicated uses. - `dedicated-connection-pool.test.ts`: the burst test now checks that the next burst reuses all sessions and that they close after the idle timeout (1 s in the test). Before, it checked that at most 4 stayed open. - `doc/DATABASE.md` describes the new retention rule. ## Verification Connections opened per 1,000 operations, embedded PostgreSQL, `pg_stat_database.sessions` (load average about 10.5 on a shared machine): | Operation | Before | After | | --- | --- | --- | | Checkpoint saves, concurrency 1 / 7 / 24 | 1 / 3 / 3 | 1 / 3 / 3 | | Dedicated uses, steady concurrency 1 / 4 / 8 / 16 | 1 / 4 / 8 / 16 | 1 / 4 / 8 / 16 | | Dedicated uses, recurring bursts of 8 | 504 | 8 | | Dedicated uses, recurring bursts of 16 | 760 | 16 | - `pnpm --filter @paperclipai/db exec vitest run src/connection-reuse-census.test.ts src/dedicated-connection-pool.test.ts src/dedicated-connection-admission.test.ts src/client-options.test.ts`: 38 passed. The burst cases fail on `master` (504 and 760 against a bound of 8 and 16). ## Risks - After a burst, up to `DATABASE_DEDICATED_CONNECTION_MAX` dedicated sessions stay open for up to `DATABASE_IDLE_TIMEOUT_SECONDS` (60 s) instead of closing at once above 4. This stays inside the connection budget. With `DATABASE_IDLE_TIMEOUT_SECONDS=0` they stay open until shutdown, like the main pool. - Retained sessions keep the existing rule: actions keep session state transaction-scoped. - The PgBouncer `SET application_name` statements stay. Removing them needs one shared `application_name` for all pools, which removes the per-pool names that operators and tests use. That is a separate decision. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
8f1d39da79
commit
531db3c10b
4 files changed
+89
-11
No files matched your search
+6
-3
@@ -139,9 +139,12 @@ DATABASE_APPLICATION_NAME=paperclip # application_name in pg_stat_activity; def
|
||||
|
||||
`withDedicatedDbConnection` gives an action exclusive use of one session outside
|
||||
the main pool (application name `paperclip-workspace-finalization-lock`). Each
|
||||
`createDb` owner keeps at most `DEDICATED_CONNECTION_IDLE_MAX` (4) of these
|
||||
single-connection clients idle for reuse; they follow the same idle timeout and
|
||||
max lifetime as the main pool. At most `DATABASE_DEDICATED_CONNECTION_MAX`
|
||||
`createDb` owner keeps released single-connection clients idle for reuse: always
|
||||
up to `DEDICATED_CONNECTION_IDLE_MAX` (4), and more while idle plus in-use
|
||||
clients stay within `DATABASE_DEDICATED_CONNECTION_MAX`, so a recurring burst
|
||||
reuses the sessions the previous burst opened instead of reconnecting. They
|
||||
follow the same idle timeout and max lifetime as the main pool, so a retained
|
||||
client closes its session after `DATABASE_IDLE_TIMEOUT_SECONDS` without use. At most `DATABASE_DEDICATED_CONNECTION_MAX`
|
||||
(default 24) dedicated clients are in use at once per owner. A caller that
|
||||
finds the limit reached waits in a first-in, first-out queue. A caller that
|
||||
already holds a dedicated client, or runs inside an open transaction of the
|
||||
|
||||
@@ -446,7 +446,8 @@ export async function withDedicatedDbConnection<T>(db: Db, action: (dedicated: D
|
||||
} finally {
|
||||
hold.active = false;
|
||||
try {
|
||||
if (dedicated && completed && !set.closed && !set.ended.has(dedicated) && set.idle.length < DEDICATED_CONNECTION_IDLE_MAX) {
|
||||
if (dedicated && completed && !set.closed && !set.ended.has(dedicated)
|
||||
&& (set.idle.length < DEDICATED_CONNECTION_IDLE_MAX || set.idle.length + set.inUse <= set.limit)) {
|
||||
set.idle.push(dedicated);
|
||||
} else if (dedicated && !set.ended.has(dedicated)) {
|
||||
await dedicated.$client.end({ timeout: 1 });
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
import { sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { checkpointDb, closeCheckpointDb, closeDedicatedDbConnections, createDb, withDedicatedDbConnection, type Db } from "./client.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
|
||||
async function sessionsOpened(observer: Db) {
|
||||
const [row] = await observer.execute<{ sessions: string }>(sql`select sessions::text as sessions from pg_stat_database where datname = current_database()`);
|
||||
return Number(row!.sessions);
|
||||
}
|
||||
|
||||
async function runConcurrently(total: number, concurrency: number, work: () => Promise<unknown>) {
|
||||
let next = 0;
|
||||
await Promise.all(Array.from({ length: concurrency }, async () => {
|
||||
while (next < total) { next++; await work(); }
|
||||
}));
|
||||
}
|
||||
|
||||
describe.skipIf(!support.supported)("connections opened per 1,000 operations", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let observer: Db;
|
||||
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("connection-reuse-");
|
||||
observer = createDb(temporary.connectionString, { maxConnections: 1, applicationName: "connection-reuse-observer" });
|
||||
await sessionsOpened(observer);
|
||||
}, 90_000);
|
||||
afterAll(async () => {
|
||||
await observer?.$client.end({ timeout: 1 });
|
||||
await temporary?.cleanup();
|
||||
});
|
||||
|
||||
async function measure(open: (db: Db) => Promise<void>) {
|
||||
const db = createDb(temporary.connectionString, { maxConnections: 10 });
|
||||
await db.execute(sql`select 1`);
|
||||
const before = await sessionsOpened(observer);
|
||||
try {
|
||||
await open(db);
|
||||
return await sessionsOpened(observer) - before;
|
||||
} finally {
|
||||
await closeCheckpointDb(db, 1);
|
||||
await closeDedicatedDbConnections(db, 1);
|
||||
await db.$client.end({ timeout: 1 });
|
||||
}
|
||||
}
|
||||
|
||||
it.each([1, 7, 24])("checkpoint saves at concurrency %i reuse the checkpoint pool", async concurrency => {
|
||||
const opened = await measure(db => runConcurrently(1_000, concurrency, () =>
|
||||
checkpointDb(db).transaction(async tx => { await tx.execute(sql`select 1`); })));
|
||||
expect(opened).toBeLessThanOrEqual(3);
|
||||
});
|
||||
|
||||
it.each([1, 4, 8, 16])("dedicated connection uses at concurrency %i", async concurrency => {
|
||||
const opened = await measure(db => runConcurrently(1_000, concurrency, () =>
|
||||
withDedicatedDbConnection(db, connection => connection.transaction(async tx => { await tx.execute(sql`select 1`); }))));
|
||||
expect(opened).toBeLessThanOrEqual(concurrency);
|
||||
});
|
||||
|
||||
it.each([8, 16])("bursts of %i dedicated connection uses reuse the connections the last burst opened", async width => {
|
||||
const opened = await measure(async db => {
|
||||
for (let burst = 0; burst < 1_000 / width; burst++) {
|
||||
await Promise.all(Array.from({ length: width }, () =>
|
||||
withDedicatedDbConnection(db, connection => connection.transaction(async tx => { await tx.execute(sql`select 1`); }))));
|
||||
}
|
||||
});
|
||||
expect(opened).toBeLessThanOrEqual(width);
|
||||
});
|
||||
});
|
||||
@@ -73,8 +73,9 @@ describe.skipIf(!support.supported)("withDedicatedDbConnection connection reuse"
|
||||
expect(seen.size).toBeLessThanOrEqual(DEDICATED_CONNECTION_IDLE_MAX);
|
||||
});
|
||||
|
||||
it("gives concurrent callers exclusive sessions and keeps at most the cap open afterwards", async () => {
|
||||
const db = owner();
|
||||
it("gives concurrent callers exclusive sessions, reuses them for the next burst and closes them once idle", async () => {
|
||||
const db = createDb(temporary.connectionString, { maxConnections: 2, idleTimeoutSeconds: 1 });
|
||||
owners.push(db);
|
||||
const concurrency = DEDICATED_CONNECTION_IDLE_MAX + 3;
|
||||
let release!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
@@ -94,12 +95,16 @@ describe.skipIf(!support.supported)("withDedicatedDbConnection connection reuse"
|
||||
release();
|
||||
const sessions = await Promise.all(held);
|
||||
expect(new Set(sessions.map(key)).size).toBe(concurrency);
|
||||
expect((await eventually(openDedicatedSessions, open => open.length <= DEDICATED_CONNECTION_IDLE_MAX)).length)
|
||||
.toBeLessThanOrEqual(DEDICATED_CONNECTION_IDLE_MAX);
|
||||
const retained = new Set(sessions.map(key));
|
||||
for (let i = 0; i < 20; i++) {
|
||||
expect(retained.has(key(await withDedicatedDbConnection(db, session)))).toBe(true);
|
||||
}
|
||||
const nextBurst = await Promise.all(Array.from({ length: concurrency }, () =>
|
||||
withDedicatedDbConnection(db, dedicated => dedicated.transaction(async tx => {
|
||||
const mine = await session(tx);
|
||||
await tx.execute(sql`select pg_sleep(0.05)`);
|
||||
return mine;
|
||||
}))));
|
||||
expect(nextBurst.every(value => retained.has(key(value)))).toBe(true);
|
||||
expect(await eventually(openDedicatedSessions, open => open.length === 0)).toEqual([]);
|
||||
expect(retained.has(key(await withDedicatedDbConnection(db, session)))).toBe(false);
|
||||
});
|
||||
|
||||
it("discards a connection whose action failed mid-transaction", async () => {
|
||||
|
||||
Reference in new issue
Block a user