Files
PaperClipAI/packages/db/src/dedicated-connection-pool.test.ts
T
DottaandClaude Opus 5.5 531db3c10b perf(db): keep dedicated connections for the next burst (#21769)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process runs every agent on one Node main thread, and
each new database connection costs a TLS handshake, about 0.36 ms of
SCRAM key derivation on that thread, and a type-fetch query
> - `withDedicatedDbConnection` gives lock holders a private session;
since #21693 it keeps at most 4 released sessions and closes the rest
> - When the same burst of more than 4 dedicated holders comes back,
every holder above 4 opens a new connection again
> - A harness on embedded PostgreSQL measured 504 new connections per
1,000 uses for bursts of 8 and 760 for bursts of 16
> - This pull request keeps released dedicated clients for reuse while
idle plus in-use clients stay within
`DATABASE_DEDICATED_CONNECTION_MAX`; the driver idle timeout still
closes their sessions when they are not used
> - The benefit is 8 and 16 new connections per 1,000 uses for the same
bursts, with no change to any pool maximum or to
`databaseConnectionBudget()`

## Linked Issues or Issue Description

**What is the problem?**
A staging measurement showed `SET
application_name='paperclip-runner-checkpoint'` at 3.17/s and `SET
application_name='paperclip'` at 3.86/s with 7 live runs. This looked
like connection churn. The investigation found two separate facts:

1. The `SET` statements are not new client connections. postgres.js
sends `application_name` in the startup packet, never as a `SET`
statement (`StartupMessage` in postgres.js `src/connection.js`).
PgBouncer in transaction mode sends `SET application_name` itself when
it gives a client transaction a server connection that a client with
another `application_name` used last. With one shared server pool, every
checkpoint transaction that lands after a main-pool transaction costs
one `SET`, and the reverse. The rate follows checkpoint transactions,
not connects. The checkpoint pool opens at most 3 connections per 1,000
saves (measured below).
2. Dedicated connections did churn by construction for recurring bursts
of more than 4 holders (measured below). This pull request fixes that.

**Steps to reproduce**
Run `packages/db/src/connection-reuse-census.test.ts`. It counts
`pg_stat_database.sessions` around 1,000 operations.

**Expected behavior**
A recurring burst reuses the sessions that the previous burst opened.

## What Changed

- `withDedicatedDbConnection` keeps a released client idle when fewer
than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle, as before, or when
idle plus in-use clients stay within the dedicated limit
(`DATABASE_DEDICATED_CONNECTION_MAX`, default 24). A client whose action
failed, an ended client and clients after shutdown are still closed.
- Each retained client keeps the main pool's `idle_timeout` (default 60
s), so its session closes after that idle time, as the main pool's do.
The next use reconnects.
- The worst case stays at `2 x DATABASE_DEDICATED_CONNECTION_MAX + 4`: a
new client opens only when no client is idle, and a client is kept above
4 only while idle plus in-use is within the limit.
`databaseConnectionBudget()` and every pool maximum are unchanged.
- New `connection-reuse-census.test.ts` measures connections opened per
1,000 checkpoint saves and per 1,000 dedicated uses.
- `dedicated-connection-pool.test.ts`: the burst test now checks that
the next burst reuses all sessions and that they close after the idle
timeout (1 s in the test). Before, it checked that at most 4 stayed
open.
- `doc/DATABASE.md` describes the new retention rule.

## Verification

Connections opened per 1,000 operations, embedded PostgreSQL,
`pg_stat_database.sessions` (load average about 10.5 on a shared
machine):

| Operation | Before | After |
| --- | --- | --- |
| Checkpoint saves, concurrency 1 / 7 / 24 | 1 / 3 / 3 | 1 / 3 / 3 |
| Dedicated uses, steady concurrency 1 / 4 / 8 / 16 | 1 / 4 / 8 / 16 | 1
/ 4 / 8 / 16 |
| Dedicated uses, recurring bursts of 8 | 504 | 8 |
| Dedicated uses, recurring bursts of 16 | 760 | 16 |

- `pnpm --filter @paperclipai/db exec vitest run
src/connection-reuse-census.test.ts
src/dedicated-connection-pool.test.ts
src/dedicated-connection-admission.test.ts src/client-options.test.ts`:
38 passed. The burst cases fail on `master` (504 and 760 against a bound
of 8 and 16).

## Risks

- After a burst, up to `DATABASE_DEDICATED_CONNECTION_MAX` dedicated
sessions stay open for up to `DATABASE_IDLE_TIMEOUT_SECONDS` (60 s)
instead of closing at once above 4. This stays inside the connection
budget. With `DATABASE_IDLE_TIMEOUT_SECONDS=0` they stay open until
shutdown, like the main pool.
- Retained sessions keep the existing rule: actions keep session state
transaction-scoped.
- The PgBouncer `SET application_name` statements stay. Removing them
needs one shared `application_name` for all pools, which removes the
per-pool names that operators and tests use. That is a separate
decision.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:51:55 -06:00

200 lines
9.0 KiB
TypeScript

import { sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
DEDICATED_CONNECTION_IDLE_MAX,
closeDedicatedDbConnections,
createDb,
withDedicatedDbConnection,
type Db,
} from "./client.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js";
const support = await getEmbeddedPostgresTestSupport();
const APPLICATION_NAME = "paperclip-workspace-finalization-lock";
type Session = { pid: number; backendStart: string };
async function session(db: Pick<Db, "execute">): Promise<Session> {
const [row] = await db.execute<{ pid: number; backend_start: string }>(sql`select pg_backend_pid() as pid,
(select backend_start::text from pg_stat_activity where pid = pg_backend_pid()) as backend_start`);
return { pid: Number(row.pid), backendStart: String(row.backend_start) };
}
const key = (value: Session) => `${value.pid}@${value.backendStart}`;
async function eventually<T>(read: () => Promise<T>, accept: (value: T) => boolean): Promise<T> {
const deadline = Date.now() + 5_000;
for (;;) {
const value = await read();
if (accept(value) || Date.now() > deadline) return value;
await new Promise(resolve => setTimeout(resolve, 25));
}
}
describe.skipIf(!support.supported)("withDedicatedDbConnection connection reuse", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let observer: Db;
const owners: Db[] = [];
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("dedicated-pool-");
observer = createDb(temporary.connectionString, { maxConnections: 1, applicationName: "dedicated-pool-observer" });
}, 90_000);
afterEach(async () => {
for (const db of owners.splice(0)) {
await closeDedicatedDbConnections(db, 1);
await db.$client.end({ timeout: 1 });
}
});
afterAll(async () => {
await observer?.$client.end({ timeout: 1 });
await temporary?.cleanup();
});
function owner() {
const db = createDb(temporary.connectionString, { maxConnections: 2 });
owners.push(db);
return db;
}
async function openDedicatedSessions(): Promise<Session[]> {
const rows = await observer.execute<{ pid: number; backend_start: string }>(sql`select pid, backend_start::text as backend_start
from pg_stat_activity where application_name = ${APPLICATION_NAME} and datname = current_database()`);
return rows.map(row => ({ pid: Number(row.pid), backendStart: String(row.backend_start) }));
}
it("reuses a bounded set of physical connections across 200 sequential calls", async () => {
const db = owner();
const seen = new Set<string>();
for (let i = 0; i < 200; i++) {
seen.add(key(await withDedicatedDbConnection(db, dedicated => dedicated.transaction(tx => session(tx)))));
}
expect(seen.size).toBe(1);
expect(seen.size).toBeLessThanOrEqual(DEDICATED_CONNECTION_IDLE_MAX);
});
it("gives concurrent callers exclusive sessions, reuses them for the next burst and closes them once idle", async () => {
const db = createDb(temporary.connectionString, { maxConnections: 2, idleTimeoutSeconds: 1 });
owners.push(db);
const concurrency = DEDICATED_CONNECTION_IDLE_MAX + 3;
let release!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
let arrived = 0;
let allArrived!: () => void;
const everyoneHolding = new Promise<void>(resolve => { allArrived = resolve; });
const held = Array.from({ length: concurrency }, () => withDedicatedDbConnection(db, dedicated =>
dedicated.transaction(async tx => {
const mine = await session(tx);
await tx.execute(sql`select pg_advisory_xact_lock(${mine.pid})`);
if (++arrived === concurrency) allArrived();
await gate;
expect(await session(tx)).toEqual(mine);
return mine;
})));
await everyoneHolding;
release();
const sessions = await Promise.all(held);
expect(new Set(sessions.map(key)).size).toBe(concurrency);
const retained = new Set(sessions.map(key));
const nextBurst = await Promise.all(Array.from({ length: concurrency }, () =>
withDedicatedDbConnection(db, dedicated => dedicated.transaction(async tx => {
const mine = await session(tx);
await tx.execute(sql`select pg_sleep(0.05)`);
return mine;
}))));
expect(nextBurst.every(value => retained.has(key(value)))).toBe(true);
expect(await eventually(openDedicatedSessions, open => open.length === 0)).toEqual([]);
expect(retained.has(key(await withDedicatedDbConnection(db, session)))).toBe(false);
});
it("discards a connection whose action failed mid-transaction", async () => {
const db = owner();
let failed: Session | undefined;
await expect(withDedicatedDbConnection(db, dedicated => dedicated.transaction(async tx => {
failed = await session(tx);
await tx.execute(sql`select 1 / 0`);
}))).rejects.toThrow();
expect(failed).toBeDefined();
const next = await withDedicatedDbConnection(db, session);
expect(key(next)).not.toBe(key(failed!));
const open = await eventually(openDedicatedSessions, rows => !rows.map(key).includes(key(failed!)));
expect(open.map(key)).not.toContain(key(failed!));
});
it("discards a connection whose socket broke during the action", async () => {
const db = owner();
let broken: Session | undefined;
await expect(withDedicatedDbConnection(db, async dedicated => {
broken = await session(dedicated);
await observer.execute(sql`select pg_terminate_backend(${broken.pid})`);
await session(dedicated);
})).rejects.toThrow();
const next = await withDedicatedDbConnection(db, session);
expect(next.pid).not.toBe(broken!.pid);
});
it("recovers when an idle retained connection is terminated between calls", async () => {
const db = owner();
const first = await withDedicatedDbConnection(db, session);
await observer.execute(sql`select pg_terminate_backend(${first.pid})`);
await eventually(openDedicatedSessions, rows => !rows.some(row => row.pid === first.pid));
await new Promise(resolve => setTimeout(resolve, 50));
const second = await withDedicatedDbConnection(db, session);
expect(second.pid).not.toBe(first.pid);
});
it("never reuses a connection the action ended itself", async () => {
const db = owner();
const first = await withDedicatedDbConnection(db, async dedicated => {
const value = await session(dedicated);
await dedicated.$client.end({ timeout: 0 });
return value;
});
const second = await withDedicatedDbConnection(db, session);
expect(second.pid).not.toBe(first.pid);
});
it("keeps transaction-local settings out of the next action", async () => {
const db = owner();
await withDedicatedDbConnection(db, dedicated => dedicated.transaction(tx =>
tx.execute(sql`select set_config('statement_timeout', '1234', true), set_config('lock_timeout', '567', true)`)));
const [row] = await withDedicatedDbConnection(db, dedicated => dedicated.execute<{ statement_timeout: string; lock_timeout: string }>(
sql`select current_setting('statement_timeout') as statement_timeout, current_setting('lock_timeout') as lock_timeout`));
expect(row).toEqual({ statement_timeout: "0", lock_timeout: "0" });
});
it("closes every retained connection on shutdown and stays usable without retaining more", async () => {
const db = owner();
await Promise.all(Array.from({ length: DEDICATED_CONNECTION_IDLE_MAX }, () =>
withDedicatedDbConnection(db, dedicated => dedicated.transaction(async tx => {
await tx.execute(sql`select pg_sleep(0.05)`);
}))));
expect((await openDedicatedSessions()).length).toBeGreaterThan(0);
await closeDedicatedDbConnections(db, 1);
expect(await eventually(openDedicatedSessions, rows => rows.length === 0)).toEqual([]);
await withDedicatedDbConnection(db, session);
expect(await eventually(openDedicatedSessions, rows => rows.length === 0)).toEqual([]);
});
it("closes retained connections when the owner client ends", async () => {
const db = createDb(temporary.connectionString, { maxConnections: 1 });
await withDedicatedDbConnection(db, session);
expect((await openDedicatedSessions()).length).toBe(1);
await db.$client.end({ timeout: 1 });
expect(await eventually(openDedicatedSessions, rows => rows.length === 0)).toEqual([]);
});
it("shares one retained set with nested dedicated actions without blocking them", async () => {
const db = owner();
const [outer, inner] = await withDedicatedDbConnection(db, async dedicated => {
const mine = await session(dedicated);
const nested = await withDedicatedDbConnection(dedicated, session);
return [mine, nested];
});
expect(inner.pid).not.toBe(outer.pid);
const reused = new Set<number>();
for (let i = 0; i < 10; i++) reused.add((await withDedicatedDbConnection(db, session)).pid);
expect([...reused].every(pid => pid === outer.pid || pid === inner.pid)).toBe(true);
});
});