Files
PaperClipAI/packages/db/src/client.ts
T
DottaandClaude Opus 5.5 531db3c10b perf(db): keep dedicated connections for the next burst (#21769)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process runs every agent on one Node main thread, and
each new database connection costs a TLS handshake, about 0.36 ms of
SCRAM key derivation on that thread, and a type-fetch query
> - `withDedicatedDbConnection` gives lock holders a private session;
since #21693 it keeps at most 4 released sessions and closes the rest
> - When the same burst of more than 4 dedicated holders comes back,
every holder above 4 opens a new connection again
> - A harness on embedded PostgreSQL measured 504 new connections per
1,000 uses for bursts of 8 and 760 for bursts of 16
> - This pull request keeps released dedicated clients for reuse while
idle plus in-use clients stay within
`DATABASE_DEDICATED_CONNECTION_MAX`; the driver idle timeout still
closes their sessions when they are not used
> - The benefit is 8 and 16 new connections per 1,000 uses for the same
bursts, with no change to any pool maximum or to
`databaseConnectionBudget()`

## Linked Issues or Issue Description

**What is the problem?**
A staging measurement showed `SET
application_name='paperclip-runner-checkpoint'` at 3.17/s and `SET
application_name='paperclip'` at 3.86/s with 7 live runs. This looked
like connection churn. The investigation found two separate facts:

1. The `SET` statements are not new client connections. postgres.js
sends `application_name` in the startup packet, never as a `SET`
statement (`StartupMessage` in postgres.js `src/connection.js`).
PgBouncer in transaction mode sends `SET application_name` itself when
it gives a client transaction a server connection that a client with
another `application_name` used last. With one shared server pool, every
checkpoint transaction that lands after a main-pool transaction costs
one `SET`, and the reverse. The rate follows checkpoint transactions,
not connects. The checkpoint pool opens at most 3 connections per 1,000
saves (measured below).
2. Dedicated connections did churn by construction for recurring bursts
of more than 4 holders (measured below). This pull request fixes that.

**Steps to reproduce**
Run `packages/db/src/connection-reuse-census.test.ts`. It counts
`pg_stat_database.sessions` around 1,000 operations.

**Expected behavior**
A recurring burst reuses the sessions that the previous burst opened.

## What Changed

- `withDedicatedDbConnection` keeps a released client idle when fewer
than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle, as before, or when
idle plus in-use clients stay within the dedicated limit
(`DATABASE_DEDICATED_CONNECTION_MAX`, default 24). A client whose action
failed, an ended client and clients after shutdown are still closed.
- Each retained client keeps the main pool's `idle_timeout` (default 60
s), so its session closes after that idle time, as the main pool's do.
The next use reconnects.
- The worst case stays at `2 x DATABASE_DEDICATED_CONNECTION_MAX + 4`: a
new client opens only when no client is idle, and a client is kept above
4 only while idle plus in-use is within the limit.
`databaseConnectionBudget()` and every pool maximum are unchanged.
- New `connection-reuse-census.test.ts` measures connections opened per
1,000 checkpoint saves and per 1,000 dedicated uses.
- `dedicated-connection-pool.test.ts`: the burst test now checks that
the next burst reuses all sessions and that they close after the idle
timeout (1 s in the test). Before, it checked that at most 4 stayed
open.
- `doc/DATABASE.md` describes the new retention rule.

## Verification

Connections opened per 1,000 operations, embedded PostgreSQL,
`pg_stat_database.sessions` (load average about 10.5 on a shared
machine):

| Operation | Before | After |
| --- | --- | --- |
| Checkpoint saves, concurrency 1 / 7 / 24 | 1 / 3 / 3 | 1 / 3 / 3 |
| Dedicated uses, steady concurrency 1 / 4 / 8 / 16 | 1 / 4 / 8 / 16 | 1
/ 4 / 8 / 16 |
| Dedicated uses, recurring bursts of 8 | 504 | 8 |
| Dedicated uses, recurring bursts of 16 | 760 | 16 |

- `pnpm --filter @paperclipai/db exec vitest run
src/connection-reuse-census.test.ts
src/dedicated-connection-pool.test.ts
src/dedicated-connection-admission.test.ts src/client-options.test.ts`:
38 passed. The burst cases fail on `master` (504 and 760 against a bound
of 8 and 16).

## Risks

- After a burst, up to `DATABASE_DEDICATED_CONNECTION_MAX` dedicated
sessions stay open for up to `DATABASE_IDLE_TIMEOUT_SECONDS` (60 s)
instead of closing at once above 4. This stays inside the connection
budget. With `DATABASE_IDLE_TIMEOUT_SECONDS=0` they stay open until
shutdown, like the main pool.
- Retained sessions keep the existing rule: actions keep session state
transaction-scoped.
- The PgBouncer `SET application_name` statements stay. Removing them
needs one shared `application_name` for all pools, which removes the
per-pool names that operators and tests use. That is a separate
decision.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:51:55 -06:00

1638 lines
63 KiB
TypeScript

import { AsyncLocalStorage } from "node:async_hooks";
import { createHash } from "node:crypto";
import { readFile, readdir } from "node:fs/promises";
import { fileURLToPath } from "node:url";
import postgres from "postgres";
import { installDatabaseWorkSignals } from "./work-signals.js";
import { installDatabaseResponseCapture } from "./response-capture.js";
import { createPreparedQueryState, installPreparedQueries, preparedQueryStats, type PreparedQueryState, type PreparedQueryStats } from "./prepared-queries.js";
import { drizzleWithLazyTransactionQueries } from "./lazy-transaction-queries.js";
import * as schema from "./schema/index.js";
const MIGRATIONS_FOLDER = fileURLToPath(new URL("./migrations", import.meta.url));
const DRIZZLE_MIGRATIONS_TABLE = "__drizzle_migrations";
const MIGRATIONS_JOURNAL_JSON = fileURLToPath(new URL("./migrations/meta/_journal.json", import.meta.url));
const PROJECT_ENVIRONMENT_OVERLAP = {
dropFile: "0466_drop_project_environment.sql",
dropSha256: "07ca1c666c6471fdb9ce7a0a5fb31bf1a074138466fa8daae50b53199b8e0d7c",
repairFile: "0467_fair_human_torch.sql",
repairSha256: "50ce0eb2567570b098bea1de92fe6a220d139e4ba83438195fbbad1be51e31a1",
} as const;
function createUtilitySql(url: string) {
return postgres(url, { max: 1, onnotice: () => {} });
}
type RegisteredPostgresClient = ReturnType<typeof postgres>;
/**
* Derives a registry key from a connection URL's host and port only. We must
* not retain or log the full URL, because it carries credentials.
*/
function hostPortKey(url: string): string {
const parsed = new URL(url);
return `${parsed.hostname}:${parsed.port || "5432"}`;
}
/**
* Same as `hostPortKey`, but returns `null` instead of throwing when the URL
* does not parse. `postgres(url)` tolerates a value `new URL()` rejects (an
* empty string falls back to the `PG*` environment variables), so `createDb`
* must tolerate it too: skip the registry entry and let the driver decide
* the outcome, instead of throwing an error the driver itself would not.
*/
function hostPortKeyOrNull(url: string): string | null {
try {
return hostPortKey(url);
} catch (error) {
if (error instanceof TypeError && (error as NodeJS.ErrnoException).code === "ERR_INVALID_URL") return null;
throw error;
}
}
// Tracks every client `createDb` hands out, keyed by host and port, so a test
// fixture can end them before it stops the Postgres cluster they point at. A
// `WeakRef` plus `FinalizationRegistry` means a long-lived process (a real
// server) retains nothing extra: an unreferenced client is pruned on its own.
const clientsByHostPort = new Map<string, Set<WeakRef<RegisteredPostgresClient>>>();
const clientFinalizer = new FinalizationRegistry<{ hostPortKey: string; ref: WeakRef<RegisteredPostgresClient> }>(
({ hostPortKey, ref }) => {
const refs = clientsByHostPort.get(hostPortKey);
if (!refs) return;
refs.delete(ref);
if (refs.size === 0) clientsByHostPort.delete(hostPortKey);
},
);
function registerClient(key: string, client: RegisteredPostgresClient): void {
const ref = new WeakRef(client);
let refs = clientsByHostPort.get(key);
if (!refs) {
refs = new Set();
clientsByHostPort.set(key, refs);
}
refs.add(ref);
clientFinalizer.register(client, { hostPortKey: key, ref }, ref);
}
/**
* Ends every live client `createDb` handed out for the given URL's host and
* port, then forgets them. Call this before stopping a Postgres cluster: a
* client that outlives the cluster it points at can crash the process (a
* reserved connection's deferred write firing after the socket is gone).
* Swallows individual `end()` errors so one bad client cannot block the rest.
* An owned database on a shared test host must select database scope instead;
* its teardown cannot close another fixture's clients on that host.
*/
export async function closeRegisteredClients(url: string, options?: { scope: "database" }): Promise<void> {
const key = hostPortKeyOrNull(url);
if (key === null) return;
const database = options?.scope === "database" ? decodeURIComponent(new URL(url).pathname.slice(1)) : undefined;
if (database === "") throw new Error("Database-scoped teardown requires an explicit database name");
const refs = clientsByHostPort.get(key);
if (!refs) return;
const clients: RegisteredPostgresClient[] = [];
for (const ref of refs) {
const client = ref.deref();
if (client && database !== undefined && client.options.database !== database) continue;
refs.delete(ref);
clientFinalizer.unregister(ref);
if (client) clients.push(client);
}
if (refs.size === 0) clientsByHostPort.delete(key);
await Promise.all(clients.map((client) => client.end({ timeout: 1 }).catch(() => {})));
}
function isSafeIdentifier(value: string): boolean {
return /^[A-Za-z_][A-Za-z0-9_]*$/.test(value);
}
function quoteIdentifier(value: string): string {
if (!isSafeIdentifier(value)) throw new Error(`Unsafe SQL identifier: ${value}`);
return `"${value.replaceAll("\"", "\"\"")}"`;
}
function quoteLiteral(value: string): string {
return `'${value.replaceAll("'", "''")}'`;
}
function splitMigrationStatements(content: string): string[] {
return content
.split("--> statement-breakpoint")
.map((statement) => statement.trim())
.filter((statement) => statement.length > 0);
}
export type MigrationState =
| {
status: "upToDate";
tableCount: number;
availableMigrations: string[];
appliedMigrations: string[];
journalEntryCount: number;
}
| {
status: "needsMigrations";
tableCount: number;
availableMigrations: string[];
appliedMigrations: string[];
pendingMigrations: string[];
journalEntryCount: number;
reason: "no-migration-journal-empty-db" | "no-migration-journal-non-empty-db" | "pending-migrations";
};
export interface DatabaseClientOptions {
prepare?: boolean;
/** postgres.js `max` — connection pool size (driver default: 10). */
maxConnections?: number;
/** postgres.js `idle_timeout` in seconds (driver default: disabled). */
idleTimeoutSeconds?: number;
/** postgres.js `connect_timeout` in seconds (driver default: 30). */
connectTimeoutSeconds?: number;
/**
* postgres.js `max_lifetime` in seconds. Bounds how long one pooled
* connection is reused before the client replaces it (driver default: a
* random value between 30 and 60 minutes).
*/
maxLifetimeSeconds?: number;
/**
* postgres.js `connection.application_name`, shown in
* `pg_stat_activity.application_name`. Lets an operator tell Paperclip's
* pool apart from other clients of the same database (driver default:
* `postgres.js`).
*/
applicationName?: string;
/**
* Connection-level `statement_timeout` in milliseconds, sent as a startup
* parameter (no extra round trip). For dedicated single-purpose pools only.
*/
statementTimeoutMs?: number;
dedicatedConnectionMax?: number;
dedicatedConnectionWaitMs?: number;
}
/**
* Idle pooled connections close after this many seconds unless
* `DATABASE_IDLE_TIMEOUT_SECONDS` says otherwise. The driver default keeps an
* idle connection open forever, so a process that stops issuing queries still
* holds every backend it ever opened. Set `DATABASE_IDLE_TIMEOUT_SECONDS=0`
* to restore the driver default.
*/
export const DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS = 60;
/** `application_name` reported to PostgreSQL unless `DATABASE_APPLICATION_NAME` overrides it. */
export const DEFAULT_DATABASE_APPLICATION_NAME = "paperclip";
function envBoolean(env: NodeJS.ProcessEnv, name: string): boolean | undefined {
const value = env[name]?.trim().toLowerCase();
if (value === undefined || value === "") return undefined;
if (value === "true" || value === "1") return true;
if (value === "false" || value === "0") return false;
throw new Error(`${name} must be "true" or "false", got: ${env[name]}`);
}
function envPositiveInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
const value = env[name]?.trim();
if (value === undefined || value === "") return undefined;
if (!/^[1-9]\d*$/.test(value)) {
throw new Error(`${name} must be a positive integer, got: ${env[name]}`);
}
return Number.parseInt(value, 10);
}
function envNonNegativeInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
const value = env[name]?.trim();
if (value === undefined || value === "") return undefined;
if (!/^(?:0|[1-9]\d*)$/.test(value)) {
throw new Error(`${name} must be a non-negative integer, got: ${env[name]}`);
}
return Number.parseInt(value, 10);
}
function envNonEmptyString(env: NodeJS.ProcessEnv, name: string): string | undefined {
const value = env[name]?.trim();
if (value === undefined || value === "") return undefined;
return value;
}
/**
* Database client tuning from the environment, so hosted deployments can
* adapt to their connection topology (pooled endpoints, network latency)
* without editing source. Every variable is optional. This function returns
* only the values the environment sets; `resolveDatabaseClientOptions` adds
* Paperclip's own defaults on top, and the driver defaults apply to the rest
* — self-hosted setups need none of these.
*/
export function databaseClientOptionsFromEnv(env: NodeJS.ProcessEnv = process.env): DatabaseClientOptions {
const options: DatabaseClientOptions = {};
const prepare = envBoolean(env, "DATABASE_PREPARED_STATEMENTS");
if (prepare !== undefined) options.prepare = prepare;
const maxConnections = envPositiveInteger(env, "DATABASE_POOL_MAX");
if (maxConnections !== undefined) options.maxConnections = maxConnections;
// `0` is allowed here: it disables idle reaping (the driver default).
const idleTimeoutSeconds = envNonNegativeInteger(env, "DATABASE_IDLE_TIMEOUT_SECONDS");
if (idleTimeoutSeconds !== undefined) options.idleTimeoutSeconds = idleTimeoutSeconds;
const connectTimeoutSeconds = envPositiveInteger(env, "DATABASE_CONNECT_TIMEOUT_SECONDS");
if (connectTimeoutSeconds !== undefined) options.connectTimeoutSeconds = connectTimeoutSeconds;
const maxLifetimeSeconds = envPositiveInteger(env, "DATABASE_MAX_LIFETIME_SECONDS");
if (maxLifetimeSeconds !== undefined) options.maxLifetimeSeconds = maxLifetimeSeconds;
const applicationName = envNonEmptyString(env, "DATABASE_APPLICATION_NAME");
if (applicationName !== undefined) options.applicationName = applicationName;
const dedicatedConnectionMax = envPositiveInteger(env, "DATABASE_DEDICATED_CONNECTION_MAX");
if (dedicatedConnectionMax !== undefined) options.dedicatedConnectionMax = dedicatedConnectionMax;
const dedicatedConnectionWaitMs = envPositiveInteger(env, "DATABASE_DEDICATED_CONNECTION_WAIT_MS");
if (dedicatedConnectionWaitMs !== undefined) options.dedicatedConnectionWaitMs = dedicatedConnectionWaitMs;
return options;
}
/**
* Fills in Paperclip's defaults for the options the caller left unset: idle
* connections are reaped after `DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS`, and the
* pool identifies itself as `DEFAULT_DATABASE_APPLICATION_NAME`. Everything
* else stays at the driver default. An explicit value (including
* `idleTimeoutSeconds: 0`) always wins over the default.
*/
export function resolveDatabaseClientOptions(options: DatabaseClientOptions): DatabaseClientOptions {
return {
...options,
idleTimeoutSeconds: options.idleTimeoutSeconds ?? DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS,
applicationName: options.applicationName ?? DEFAULT_DATABASE_APPLICATION_NAME,
};
}
export function postgresJsOptions(options: DatabaseClientOptions): Record<string, unknown> {
const driverOptions: Record<string, unknown> = {};
if (options.prepare !== undefined) driverOptions.prepare = options.prepare;
if (options.maxConnections !== undefined) driverOptions.max = options.maxConnections;
if (options.idleTimeoutSeconds !== undefined) driverOptions.idle_timeout = options.idleTimeoutSeconds;
if (options.connectTimeoutSeconds !== undefined) driverOptions.connect_timeout = options.connectTimeoutSeconds;
if (options.maxLifetimeSeconds !== undefined) driverOptions.max_lifetime = options.maxLifetimeSeconds;
const connection: Record<string, string> = {};
if (options.applicationName !== undefined) connection.application_name = options.applicationName;
if (options.statementTimeoutMs !== undefined) connection.statement_timeout = String(options.statementTimeoutMs);
if (Object.keys(connection).length) driverOptions.connection = connection;
return driverOptions;
}
// A long advisory-lock transaction must not borrow the normal pool that its
// work needs for progress writes. Keep connection configuration private to the
// originating Db lifetime; callers receive neither URLs nor credentials.
const ownedDatabaseListenerFactories = new WeakMap<object, () => Db>();
export const DEDICATED_CONNECTION_IDLE_MAX = 4;
export const DEFAULT_DEDICATED_CONNECTION_MAX = 24;
export const DEFAULT_DEDICATED_CONNECTION_WAIT_MS = 30_000;
type DedicatedConnectionWaiter = { grant(): void; timer: ReturnType<typeof setTimeout> };
type DedicatedConnectionSet = {
create(): Db;
idle: Db[];
ended: WeakSet<Db>;
closed: boolean;
limit: number;
waitMs: number;
inUse: number;
waiters: DedicatedConnectionWaiter[];
stats: DedicatedDbConnectionStats;
};
export type DedicatedDbConnectionStats = { limit: number; inUse: number; peakInUse: number; waiting: number;
queued: number; overLimit: number; nestedOverLimit: number };
const dedicatedConnectionSets = new WeakMap<object, DedicatedConnectionSet>();
type DedicatedConnectionHold = { set: DedicatedConnectionSet; active: boolean };
const heldDedicatedConnections = new AsyncLocalStorage<readonly DedicatedConnectionHold[]>();
const openTransactions = new AsyncLocalStorage<{ active: boolean }>();
function dedicatedConnectionSet(create: DedicatedConnectionSet["create"], resolved: DatabaseClientOptions): DedicatedConnectionSet {
const limit = resolved.dedicatedConnectionMax ?? DEFAULT_DEDICATED_CONNECTION_MAX;
return { create, idle: [], ended: new WeakSet(), closed: false, limit, inUse: 0, waiters: [],
waitMs: resolved.dedicatedConnectionWaitMs ?? DEFAULT_DEDICATED_CONNECTION_WAIT_MS,
stats: { limit, inUse: 0, peakInUse: 0, waiting: 0, queued: 0, overLimit: 0, nestedOverLimit: 0 } };
}
function takeDedicatedSlot(set: DedicatedConnectionSet) {
set.inUse++;
set.stats.inUse = set.inUse;
set.stats.peakInUse = Math.max(set.stats.peakInUse, set.inUse);
}
function releaseDedicatedSlot(set: DedicatedConnectionSet) {
set.inUse--;
set.stats.inUse = set.inUse;
while (set.waiters.length && set.inUse < set.limit) set.waiters.shift()!.grant();
set.stats.waiting = set.waiters.length;
}
function holdsLiveDedicatedOrTransaction(set: DedicatedConnectionSet): boolean {
return Boolean(heldDedicatedConnections.getStore()?.some(hold => hold.active && hold.set === set)
|| openTransactions.getStore()?.active);
}
async function admitDedicatedConnection(set: DedicatedConnectionSet, nested: boolean): Promise<void> {
if (nested || (set.inUse < set.limit && set.waiters.length === 0)) {
if (nested && set.inUse >= set.limit) set.stats.nestedOverLimit++;
takeDedicatedSlot(set);
return;
}
set.stats.queued++;
await new Promise<void>(resolve => {
const waiter: DedicatedConnectionWaiter = {
grant: () => { clearTimeout(waiter.timer); takeDedicatedSlot(set); resolve(); },
timer: setTimeout(() => {
const index = set.waiters.indexOf(waiter);
if (index < 0) return;
set.waiters.splice(index, 1);
set.stats.waiting = set.waiters.length;
set.stats.overLimit++;
takeDedicatedSlot(set);
resolve();
}, set.waitMs),
};
set.waiters.push(waiter);
set.stats.waiting = set.waiters.length;
});
}
function markOpenTransactions(db: Db) {
const transaction = db.transaction.bind(db) as (...args: unknown[]) => Promise<unknown>;
db.transaction = ((fn: (tx: unknown) => Promise<unknown>, ...rest: unknown[]) => transaction(async (tx: unknown) => {
const open = { active: true };
try { return await openTransactions.run(open, () => fn(tx)); }
finally { open.active = false; }
}, ...rest)) as Db["transaction"];
}
export const DRIVER_DEFAULT_POOL_MAX_CONNECTIONS = 10;
export const PRIVATE_POOL_MAX_CONNECTIONS = {
liveness: 2, legacyControllerLease: 1, controllerInstanceLease: 1, checkpoint: 3, fileStorage: 2,
} as const;
export const DATABASE_LISTENER_CONNECTIONS = 2;
export function databaseConnectionBudget(options: DatabaseClientOptions = {}) {
const main = options.maxConnections ?? DRIVER_DEFAULT_POOL_MAX_CONNECTIONS;
const privatePools = Object.values(PRIVATE_POOL_MAX_CONNECTIONS).reduce((sum, value) => sum + value, 0);
const dedicatedLimit = options.dedicatedConnectionMax ?? DEFAULT_DEDICATED_CONNECTION_MAX;
const dedicated = 2 * dedicatedLimit + DEDICATED_CONNECTION_IDLE_MAX;
return { main, privatePools, listeners: DATABASE_LISTENER_CONNECTIONS, dedicatedLimit, dedicated,
total: main + privatePools + DATABASE_LISTENER_CONNECTIONS + dedicated };
}
const preparedQueryStates = new WeakMap<object, PreparedQueryState>();
export function preparedStatementStats(db: Db): PreparedQueryStats | null {
const state = preparedQueryStates.get(db);
return state ? preparedQueryStats(state) : null;
}
export function dedicatedDbConnectionStats(db: Db): DedicatedDbConnectionStats | null {
const set = dedicatedConnectionSets.get(db);
return set ? { ...set.stats } : null;
}
function trackClientEnd(client: Db, onEnd: (timeoutSeconds: number | undefined) => Promise<void> | void): void {
const sql = client.$client;
const end = sql.end;
sql.end = async (options?: { timeout?: number }) => {
await Promise.all([onEnd(options?.timeout), end(options)]);
};
}
async function endDedicatedConnections(clients: Db[], timeoutSeconds: number): Promise<void> {
await Promise.all(clients.map(client => client.$client.end({ timeout: timeoutSeconds }).catch(() => undefined)));
}
export async function closeDedicatedDbConnections(db: Db, timeoutSeconds: number): Promise<void> {
const set = dedicatedConnectionSets.get(db);
if (!set) return;
set.closed = true;
await endDedicatedConnections(set.idle.splice(0), timeoutSeconds);
}
export interface OwnedDatabaseListener {
listen: Db["$client"]["listen"];
close(): Promise<void>;
}
export function createOwnedDatabaseListener(db: Db): OwnedDatabaseListener {
const factory = ownedDatabaseListenerFactories.get(db);
if (!factory) throw new Error("owned_listener_requires_create_db");
const owned = factory();
return {
listen: owned.$client.listen.bind(owned.$client),
close: () => owned.$client.end({ timeout: 1 }),
};
}
export type DedicatedDbConnectionOptions = { admitWithoutWaiting?: boolean };
export async function withDedicatedDbConnection<T>(db: Db, action: (dedicated: Db) => Promise<T>,
options: DedicatedDbConnectionOptions = {}): Promise<T> {
const set = dedicatedConnectionSets.get(db);
if (!set) throw new Error("dedicated_connection_requires_create_db");
await admitDedicatedConnection(set, options.admitWithoutWaiting === true || holdsLiveDedicatedOrTransaction(set));
const hold: DedicatedConnectionHold = { set, active: true };
let dedicated: Db | undefined;
let completed = false;
try {
while (!dedicated && set.idle.length) {
const candidate = set.idle.pop()!;
if (!set.ended.has(candidate)) dedicated = candidate;
}
dedicated ??= set.create();
const connection = dedicated;
const result = await heldDedicatedConnections.run([...(heldDedicatedConnections.getStore() ?? []), hold], () => action(connection));
completed = true;
return result;
} finally {
hold.active = false;
try {
if (dedicated && completed && !set.closed && !set.ended.has(dedicated)
&& (set.idle.length < DEDICATED_CONNECTION_IDLE_MAX || set.idle.length + set.inUse <= set.limit)) {
set.idle.push(dedicated);
} else if (dedicated && !set.ended.has(dedicated)) {
await dedicated.$client.end({ timeout: 1 });
}
} finally {
releaseDedicatedSlot(set);
}
}
}
// postgres.js pipelines a query onto a busy connection when none is idle, so
// a short liveness write can wait behind an unrelated slow statement with no
// client-side bound. Liveness writes (controller-lease renewal) get a small
// private pool, created on first use and shared for the owner's lifetime.
const livenessDbFactories = new WeakMap<object, () => Db>();
const livenessDbs = new WeakMap<object, Db>();
/** The private liveness pool for a `createDb` client; other clients (test
* doubles, transactions) are returned unchanged. */
export function livenessDb(db: Db): Db {
const existing = livenessDbs.get(db);
if (existing) return existing;
const factory = livenessDbFactories.get(db);
if (!factory) return db;
const created = factory();
livenessDbs.set(db, created);
return created;
}
const controllerLeaseDbFactories = new WeakMap<object, () => Db>();
const controllerLeaseDbs = new WeakMap<object, Db>();
export function controllerLeaseDb(db: Db): Db {
const existing = controllerLeaseDbs.get(db);
if (existing) return existing;
const factory = controllerLeaseDbFactories.get(db);
if (!factory) return db;
const created = factory();
controllerLeaseDbs.set(db, created);
return created;
}
const controllerInstanceLeaseDbFactories = new WeakMap<object, (statementTimeoutMs: number | null) => Db>();
const controllerInstanceLeaseDbs = new WeakMap<object, Db>();
/** One private connection for the server's controller-instance lease, so its
* renewal never waits behind request or worker traffic on the main pool.
* `statementTimeoutMs` (first call wins) is a connection-level
* `statement_timeout`: the server cancels a slow renewal, so it never holds the
* single connection that the next renewal needs, at no extra round trip. */
export function controllerInstanceLeaseDb(db: Db, options: { statementTimeoutMs?: number | null } = {}): Db {
const existing = controllerInstanceLeaseDbs.get(db);
if (existing) return existing;
const factory = controllerInstanceLeaseDbFactories.get(db);
if (!factory) return db;
const created = factory(options.statementTimeoutMs ?? null);
controllerInstanceLeaseDbs.set(db, created);
return created;
}
/** Replaces the lease connection with one that sends no `statement_timeout`
* startup parameter, for a connection pooler that refuses it. */
export function controllerInstanceLeaseDbWithoutStatementTimeout(db: Db): Db {
const factory = controllerInstanceLeaseDbFactories.get(db);
if (!factory) return db;
const previous = controllerInstanceLeaseDbs.get(db);
const created = factory(null);
controllerInstanceLeaseDbs.set(db, created);
void previous?.$client.end({ timeout: 0 }).catch(() => undefined);
return created;
}
/** Ends the liveness pool if one was created. The ended pool stays cached, so
* a late renewal fails like a query on the ended primary pool. */
export async function closeLivenessDb(db: Db, timeoutSeconds: number): Promise<void> {
await Promise.all([livenessDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
controllerLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
controllerInstanceLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds })]);
}
// Control checkpoints must not occupy the request or lease-renewal pools
// during an aligned burst. Share at most three lazy connections per owner.
const checkpointDbFactories = new WeakMap<object, () => Db>();
const checkpointDbs = new WeakMap<object, Db>();
const closedCheckpointOwners = new WeakSet<object>();
/** The bounded controller checkpoint pool; transactions and test doubles keep
* their existing database when they have no owning client factory. */
export function checkpointDb(db: Db): Db {
const existing = checkpointDbs.get(db);
if (existing) return existing;
const factory = checkpointDbFactories.get(db);
if (!factory) return db;
if (closedCheckpointOwners.has(db)) throw new Error("checkpoint_pool_closed");
const created = factory();
checkpointDbs.set(db, created);
return created;
}
/** Fence lazy creation immediately, and keep any ended pool cached. */
export async function closeCheckpointDb(db: Db, timeoutSeconds: number): Promise<void> {
closedCheckpointOwners.add(db);
await checkpointDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
}
// Managed file publication holds an agent lock while transferring bytes. A
// stalled object store must not consume request or controller-renewal capacity.
const fileStorageDbFactories = new WeakMap<object, () => Db>();
const fileStorageDbs = new WeakMap<object, Db>();
const closedFileStorageOwners = new WeakSet<object>();
export function fileStorageDb(db: Db): Db {
const existing = fileStorageDbs.get(db);
if (existing) return existing;
const factory = fileStorageDbFactories.get(db);
if (!factory) return db; // Preserve the caller's transaction/savepoint.
if (closedFileStorageOwners.has(db)) throw new Error("file_storage_pool_closed");
const created = factory();
fileStorageDbs.set(db, created);
fileStorageDbs.set(created, created);
return created;
}
export async function closeFileStorageDb(db: Db, timeoutSeconds: number): Promise<void> {
closedFileStorageOwners.add(db);
await fileStorageDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
}
export function createDb(url: string, options?: DatabaseClientOptions) {
const resolved = resolveDatabaseClientOptions(options ?? databaseClientOptionsFromEnv());
return createDbClient(url, resolved);
}
function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: object, dedicatedSet?: DedicatedConnectionSet) {
const sql = postgres(url, postgresJsOptions(resolved));
const preparedState = resolved.prepare === true
? (owner && preparedQueryStates.get(owner)) || createPreparedQueryState()
: undefined;
if (preparedState) installPreparedQueries(sql, preparedState);
const key = hostPortKeyOrNull(url);
if (key) registerClient(key, sql);
// A disconnect can lose the response after a statement has committed.
// postgres.js calls that error "write CONNECTION_CLOSED" too, so the
// message cannot establish that replay is safe. Leave retries to callers
// that know the complete operation is idempotent.
const db = installDatabaseWorkSignals(drizzleWithLazyTransactionQueries(sql, schema), owner);
installDatabaseResponseCapture(db, schema);
markOpenTransactions(db);
if (preparedState) preparedQueryStates.set(db, preparedState);
const ownerSet = owner ? dedicatedConnectionSets.get(owner) : undefined;
if (dedicatedSet) {
trackClientEnd(db, () => { dedicatedSet.ended.add(db); });
dedicatedConnectionSets.set(db, dedicatedSet);
} else if (ownerSet) {
dedicatedConnectionSets.set(db, ownerSet);
} else {
const set: DedicatedConnectionSet = dedicatedConnectionSet(() => createDbClient(url, {
...resolved, maxConnections: 1, applicationName: "paperclip-workspace-finalization-lock",
}, db, set), resolved);
dedicatedConnectionSets.set(db, set);
trackClientEnd(db, timeoutSeconds => closeDedicatedDbConnections(db, timeoutSeconds ?? 1));
}
ownedDatabaseListenerFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: 1,
}, db));
livenessDbFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.liveness, applicationName: "paperclip-controller-lease",
}, db));
controllerLeaseDbFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.legacyControllerLease, applicationName: "paperclip-legacy-controller-lease",
}, db));
controllerInstanceLeaseDbFactories.set(db, statementTimeoutMs => createDbClient(url, {
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.controllerInstanceLease, applicationName: "paperclip-controller-instance-lease",
...(statementTimeoutMs !== null ? { statementTimeoutMs } : {}),
}, db));
checkpointDbFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.checkpoint, applicationName: "paperclip-runner-checkpoint",
}, db));
fileStorageDbFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.fileStorage, applicationName: "paperclip-file-storage",
}, db));
return db;
}
export async function getPostgresDataDirectory(url: string): Promise<string | null> {
const sql = createUtilitySql(url);
try {
const rows = await sql<{ data_directory: string | null }[]>`
SELECT current_setting('data_directory', true) AS data_directory
`;
const actual = rows[0]?.data_directory;
return typeof actual === "string" && actual.length > 0 ? actual : null;
} catch {
return null;
} finally {
await sql.end();
}
}
async function listMigrationFiles(): Promise<string[]> {
const entries = await readdir(MIGRATIONS_FOLDER, { withFileTypes: true });
return entries
.filter((entry) => entry.isFile() && entry.name.endsWith(".sql"))
.map((entry) => entry.name)
.sort((a, b) => a.localeCompare(b));
}
type MigrationJournalFile = {
entries?: Array<{ idx?: number; tag?: string; when?: number }>;
};
type JournalMigrationEntry = {
fileName: string;
folderMillis: number;
order: number;
};
async function listJournalMigrationEntries(): Promise<JournalMigrationEntry[]> {
try {
const raw = await readFile(MIGRATIONS_JOURNAL_JSON, "utf8");
const parsed = JSON.parse(raw) as MigrationJournalFile;
if (!Array.isArray(parsed.entries)) return [];
return parsed.entries
.map((entry, entryIndex) => {
if (typeof entry?.tag !== "string") return null;
if (typeof entry?.when !== "number" || !Number.isFinite(entry.when)) return null;
const order = Number.isInteger(entry.idx) ? Number(entry.idx) : entryIndex;
return { fileName: `${entry.tag}.sql`, folderMillis: entry.when, order };
})
.filter((entry): entry is JournalMigrationEntry => entry !== null);
} catch {
return [];
}
}
async function listJournalMigrationFiles(): Promise<string[]> {
const entries = await listJournalMigrationEntries();
return entries.map((entry) => entry.fileName);
}
async function readMigrationFileContent(migrationFile: string): Promise<string> {
return readFile(new URL(`./migrations/${migrationFile}`, import.meta.url), "utf8");
}
async function orderMigrationsByJournal(migrationFiles: string[]): Promise<string[]> {
const journalEntries = await listJournalMigrationEntries();
const orderByFileName = new Map(journalEntries.map((entry) => [entry.fileName, entry.order]));
return [...migrationFiles].sort((left, right) => {
const leftOrder = orderByFileName.get(left);
const rightOrder = orderByFileName.get(right);
if (leftOrder === undefined && rightOrder === undefined) return left.localeCompare(right);
if (leftOrder === undefined) return 1;
if (rightOrder === undefined) return -1;
if (leftOrder === rightOrder) return left.localeCompare(right);
return leftOrder - rightOrder;
});
}
type SqlExecutor = Pick<ReturnType<typeof postgres>, "unsafe">;
async function runInTransaction(sql: SqlExecutor, action: () => Promise<void>): Promise<void> {
await sql.unsafe("BEGIN");
try {
await action();
await sql.unsafe("COMMIT");
} catch (error) {
try {
await sql.unsafe("ROLLBACK");
} catch {
// Ignore rollback failures and surface the original error.
}
throw error;
}
}
async function latestMigrationCreatedAt(
sql: SqlExecutor,
qualifiedTable: string,
): Promise<number | null> {
const rows = await sql.unsafe<{ created_at: string | number | null }[]>(
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC NULLS LAST LIMIT 1`,
);
const value = Number(rows[0]?.created_at ?? Number.NaN);
return Number.isFinite(value) ? value : null;
}
function normalizeFolderMillis(value: number | null | undefined): number {
if (typeof value === "number" && Number.isFinite(value) && value >= 0) {
return Math.trunc(value);
}
return Date.now();
}
async function ensureMigrationJournalTable(
sql: ReturnType<typeof postgres>,
): Promise<{ migrationTableSchema: string; columnNames: Set<string> }> {
let migrationTableSchema = await discoverMigrationTableSchema(sql);
if (!migrationTableSchema) {
const drizzleSchema = quoteIdentifier("drizzle");
const migrationTable = quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE);
await sql.unsafe(`CREATE SCHEMA IF NOT EXISTS ${drizzleSchema}`);
await sql.unsafe(
`CREATE TABLE IF NOT EXISTS ${drizzleSchema}.${migrationTable} (id SERIAL PRIMARY KEY, hash text NOT NULL, created_at bigint)`,
);
migrationTableSchema = (await discoverMigrationTableSchema(sql)) ?? "drizzle";
}
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
return { migrationTableSchema, columnNames };
}
async function migrationHistoryEntryExists(
sql: SqlExecutor,
qualifiedTable: string,
columnNames: Set<string>,
migrationFile: string,
hash: string,
): Promise<boolean> {
const predicates: string[] = [];
if (columnNames.has("hash")) predicates.push(`hash = ${quoteLiteral(hash)}`);
if (columnNames.has("name")) predicates.push(`name = ${quoteLiteral(migrationFile)}`);
if (predicates.length === 0) return false;
const rows = await sql.unsafe<{ one: number }[]>(
`SELECT 1 AS one FROM ${qualifiedTable} WHERE ${predicates.join(" OR ")} LIMIT 1`,
);
return rows.length > 0;
}
async function recordMigrationHistoryEntry(
sql: SqlExecutor,
qualifiedTable: string,
columnNames: Set<string>,
migrationFile: string,
hash: string,
folderMillis: number,
): Promise<void> {
const insertColumns: string[] = [];
const insertValues: string[] = [];
if (columnNames.has("hash")) {
insertColumns.push(quoteIdentifier("hash"));
insertValues.push(quoteLiteral(hash));
}
if (columnNames.has("name")) {
insertColumns.push(quoteIdentifier("name"));
insertValues.push(quoteLiteral(migrationFile));
}
if (columnNames.has("created_at")) {
const latestCreatedAt = await latestMigrationCreatedAt(sql, qualifiedTable);
const createdAt = latestCreatedAt === null
? normalizeFolderMillis(folderMillis)
: Math.max(latestCreatedAt + 1, normalizeFolderMillis(folderMillis));
insertColumns.push(quoteIdentifier("created_at"));
insertValues.push(quoteLiteral(String(createdAt)));
}
if (insertColumns.length === 0) return;
await sql.unsafe(
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
);
}
// These idempotent migrations require top-level statements for batch commits
// or concurrent index builds. History is recorded only after completion.
const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
"0313_private_task_access.sql",
"0314_private_task_draft_assets.sql",
"0332_little_night_nurse.sql",
"0345_luxuriant_carmella_unuscione.sql",
"0369_late_logan.sql",
"0381_mean_scream.sql",
"0382_narrow_the_executioner.sql",
"0385_open_madame_web.sql",
"0386_steady_epoch.sql",
"0387_boring_professor_monster.sql",
"0388_ambitious_micromax.sql",
"0389_brave_hulk.sql",
"0390_salty_killraven.sql",
"0391_third_roulette.sql",
"0392_small_frightful_four.sql",
"0393_open_grandmaster.sql",
"0395_tranquil_firestar.sql",
"0396_tan_paladin.sql",
"0397_known_madrox.sql",
"0398_yielding_random.sql",
"0399_ambitious_onslaught.sql",
"0400_absent_phantom_reporter.sql",
"0401_yummy_ultimates.sql",
"0402_ancient_malice.sql",
"0403_natural_cerise.sql",
"0406_zippy_war_machine.sql",
"0407_greedy_star_brand.sql",
"0408_short_red_skull.sql",
"0409_uneven_zuras.sql",
"0410_freezing_kinsey_walden.sql",
"0411_real_scrambler.sql",
"0412_tearful_squadron_supreme.sql",
"0413_tired_kronos.sql",
"0414_closed_edwin_jarvis.sql",
"0419_slippery_impossible_man.sql",
"0432_strong_mole_man.sql",
"0435_premium_puff_adder.sql",
"0436_legal_reavers.sql",
"0447_legacy_controller_process.sql",
"0450_petite_frightful_four.sql",
"0453_curious_ben_parker.sql",
"0454_lumpy_ronan.sql",
"0455_pending_budget_cancellation_index.sql",
"0456_chat_review_lookup_indexes.sql",
"0457_native_session_run_index.sql",
"0458_chat_admission_issue_index.sql",
"0459_environment_lease_backup_scope_index.sql",
"0460_workspace_activity_context_index.sql",
"0461_created_from_legacy_origin_run_index.sql",
"0470_heartbeat_runs_queued_created_index.sql",
"0472_runner_api_capture_pending_deletes.sql",
"0476_wakeup_request_run_index.sql",
"0486_heartbeat_runs_company_finished_index.sql",
"0487_heartbeat_runs_settled_process_indexes.sql",
"0469_zippy_photon.sql",
"0384_funny_adam_warlock.sql",
"0383_tool_review_company_expiry.sql",
]);
async function applyPendingMigrationsManually(
url: string,
pendingMigrations: string[],
): Promise<void> {
if (pendingMigrations.length === 0) return;
const orderedPendingMigrations = await orderMigrationsByJournal(pendingMigrations);
const journalEntries = await listJournalMigrationEntries();
const folderMillisByFileName = new Map(
journalEntries.map((entry) => [entry.fileName, normalizeFolderMillis(entry.folderMillis)]),
);
const pool = createUtilitySql(url);
try {
const sql = await pool.reserve();
try {
// A session lock survives each batch COMMIT. Reserve the connection so
// pool rotation cannot release it while another migrator is waiting.
// A blocking lock SELECT retains its statement snapshot while waiting.
// A partial concurrent index build can wait for that snapshot, forming a
// deadlock with this lock. Each failed try ends its snapshot before sleep.
while (!(await sql<{ locked: boolean }[]>`
SELECT pg_try_advisory_lock(hashtextextended('paperclip:migrations', 0)) AS locked`)[0]?.locked) {
await new Promise((resolve) => setTimeout(resolve, 100));
}
const { migrationTableSchema, columnNames } = await ensureMigrationJournalTable(sql);
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
for (const migrationFile of orderedPendingMigrations) {
const migrationContent = await readMigrationFileContent(migrationFile);
const hash = createHash("sha256").update(migrationContent).digest("hex");
const existingEntry = await migrationHistoryEntryExists(
sql,
qualifiedTable,
columnNames,
migrationFile,
hash,
);
if (existingEntry) continue;
const applyMigration = async () => {
for (const statement of splitMigrationStatements(migrationContent)) {
if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
// A cancelled concurrent build leaves an invalid index. IF NOT
// EXISTS alone would skip it on retry and journal an incomplete index.
const index = statement.replace(/^\s*--.*$/gm, "").trim()
.match(/^CREATE (?:UNIQUE )?INDEX CONCURRENTLY IF NOT EXISTS "([A-Za-z_][A-Za-z0-9_]*)"/i);
if (index) {
const invalid = await sql<{ invalid: boolean }[]>`
SELECT NOT i.indisvalid AS invalid FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public' AND c.relname = ${index[1]}`;
if (invalid[0]?.invalid) {
await sql.unsafe(`DROP INDEX CONCURRENTLY ${quoteIdentifier("public")}.${quoteIdentifier(index[1])}`);
}
}
}
// Older dev schemas can lack a replaced constraint. Preserve the
// published SQL/hash while allowing the harmless drop to proceed.
const executable = statement.replace(
/^(\s*ALTER TABLE "[^"]+" DROP CONSTRAINT) (?!IF EXISTS\b)/i,
"$1 IF EXISTS ",
);
await sql.unsafe(executable);
}
await recordMigrationHistoryEntry(
sql,
qualifiedTable,
columnNames,
migrationFile,
hash,
folderMillisByFileName.get(migrationFile) ?? Date.now(),
);
};
if (migrationFile === PROJECT_ENVIRONMENT_OVERLAP.dropFile) {
const repairFile = PROJECT_ENVIRONMENT_OVERLAP.repairFile;
const repairContent = await readMigrationFileContent(repairFile);
const repairHash = createHash("sha256").update(repairContent).digest("hex");
const dropIndex = orderedPendingMigrations.indexOf(migrationFile);
if (hash !== PROJECT_ENVIRONMENT_OVERLAP.dropSha256
|| repairHash !== PROJECT_ENVIRONMENT_OVERLAP.repairSha256
|| orderedPendingMigrations[dropIndex + 1] !== repairFile
|| await migrationHistoryEntryExists(sql, qualifiedTable, columnNames, repairFile, repairHash)) {
throw new Error("Project environment migration pair does not match the qualified overlap repair");
}
await runInTransaction(sql, async () => {
await sql.unsafe('LOCK TABLE "public"."projects" IN ACCESS EXCLUSIVE MODE');
await sql.unsafe(`CREATE TEMP TABLE paperclip_project_environment_overlap ON COMMIT DROP AS
SELECT id, environment_id FROM "public"."projects" WHERE environment_id IS NOT NULL`);
await applyMigration();
for (const statement of splitMigrationStatements(repairContent)) await sql.unsafe(statement);
await sql.unsafe(`UPDATE "public"."projects" AS project SET environment_id = legacy.environment_id
FROM pg_temp.paperclip_project_environment_overlap AS legacy WHERE project.id = legacy.id`);
await recordMigrationHistoryEntry(sql, qualifiedTable, columnNames, repairFile, repairHash,
folderMillisByFileName.get(repairFile) ?? Date.now());
});
} else if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
await applyMigration();
} else {
await runInTransaction(sql, applyMigration);
}
}
} finally {
try {
await sql`SELECT pg_advisory_unlock(hashtextextended('paperclip:migrations', 0))`;
} catch {
// Preserve the migration error if the session died; PostgreSQL releases
// its lock on disconnect, and pool.end() closes any remaining connection.
}
sql.release();
}
} finally {
await pool.end();
}
}
async function mapHashesToMigrationFiles(migrationFiles: string[]): Promise<Map<string, string>> {
const mapped = new Map<string, string>();
await Promise.all(
migrationFiles.map(async (migrationFile) => {
const content = await readMigrationFileContent(migrationFile);
const hash = createHash("sha256").update(content).digest("hex");
mapped.set(hash, migrationFile);
if (migrationFile === "0355_agent_tool_definition_hooks.sql") {
mapped.set("7ac76adac0220da522c78e51da20490aeb0f90512b4370a05af9f8dd0f75eda6", migrationFile);
}
}),
);
return mapped;
}
async function getMigrationTableColumnNames(
sql: ReturnType<typeof postgres>,
migrationTableSchema: string,
): Promise<Set<string>> {
const columns = await sql.unsafe<{ column_name: string }[]>(
`
SELECT column_name
FROM information_schema.columns
WHERE table_schema = ${quoteLiteral(migrationTableSchema)}
AND table_name = ${quoteLiteral(DRIZZLE_MIGRATIONS_TABLE)}
`,
);
return new Set(columns.map((column) => column.column_name));
}
async function tableExists(
sql: ReturnType<typeof postgres>,
tableName: string,
): Promise<boolean> {
const rows = await sql<{ exists: boolean }[]>`
SELECT EXISTS (
SELECT 1
FROM information_schema.tables
WHERE table_schema = 'public'
AND table_name = ${tableName}
) AS exists
`;
return rows[0]?.exists ?? false;
}
async function columnExists(
sql: ReturnType<typeof postgres>,
tableName: string,
columnName: string,
): Promise<boolean> {
const rows = await sql<{ exists: boolean }[]>`
SELECT EXISTS (
SELECT 1
FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = ${tableName}
AND column_name = ${columnName}
) AS exists
`;
return rows[0]?.exists ?? false;
}
async function columnHasDataType(
sql: ReturnType<typeof postgres>,
tableName: string,
columnName: string,
dataType: string,
): Promise<boolean> {
const rows = await sql<{ dataType: string; udtName: string }[]>`
SELECT data_type AS "dataType", udt_name AS "udtName"
FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = ${tableName}
AND column_name = ${columnName}
`;
const expected = dataType.toLowerCase();
return rows.some((row) => (
row.dataType.toLowerCase() === expected || row.udtName.toLowerCase() === expected
));
}
async function indexExists(
sql: ReturnType<typeof postgres>,
indexName: string,
): Promise<boolean> {
const rows = await sql<{ exists: boolean }[]>`
SELECT EXISTS (
SELECT 1
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public'
AND c.relkind = 'i'
AND c.relname = ${indexName}
) AS exists
`;
return rows[0]?.exists ?? false;
}
async function constraintExists(
sql: ReturnType<typeof postgres>,
constraintName: string,
): Promise<boolean> {
const rows = await sql<{ exists: boolean }[]>`
SELECT EXISTS (
SELECT 1
FROM pg_constraint c
JOIN pg_namespace n ON n.oid = c.connamespace
WHERE n.nspname = 'public'
AND c.conname = ${constraintName}
) AS exists
`;
return rows[0]?.exists ?? false;
}
async function functionExists(
sql: ReturnType<typeof postgres>,
functionName: string,
): Promise<boolean> {
const rows = await sql<{ exists: boolean }[]>`
SELECT EXISTS (
SELECT 1
FROM pg_proc p
JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname = 'public'
AND p.proname = ${functionName}
) AS exists
`;
return rows[0]?.exists ?? false;
}
async function triggerExists(
sql: ReturnType<typeof postgres>,
triggerName: string,
): Promise<boolean> {
const rows = await sql<{ exists: boolean }[]>`
SELECT EXISTS (
SELECT 1
FROM pg_trigger t
JOIN pg_class c ON c.oid = t.tgrelid
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public'
AND t.tgname = ${triggerName}
AND NOT t.tgisinternal
) AS exists
`;
return rows[0]?.exists ?? false;
}
async function heartbeatEventSequencesAreUnique(
sql: ReturnType<typeof postgres>,
): Promise<boolean> {
const rows = await sql<{ unique: boolean }[]>`
SELECT NOT EXISTS (
SELECT 1
FROM heartbeat_run_events
GROUP BY run_id, seq
HAVING count(*) > 1
) AS unique
`;
return rows[0]?.unique ?? false;
}
async function heartbeatNextEventSequencesAreCurrent(
sql: ReturnType<typeof postgres>,
): Promise<boolean> {
const rows = await sql<{ current: boolean }[]>`
SELECT NOT EXISTS (
SELECT 1
FROM heartbeat_runs run
WHERE run.next_event_seq IS DISTINCT FROM COALESCE((
SELECT max(event.seq) + 1
FROM heartbeat_run_events event
WHERE event.run_id = run.id
), 1)
) AS current
`;
return rows[0]?.current ?? false;
}
async function migrationStatementAlreadyApplied(
sql: ReturnType<typeof postgres>,
statement: string,
): Promise<boolean> {
const normalized = statement
.replace(/^\s*--.*$/gm, "")
.replace(/\s+/g, " ")
.trim();
const createTableMatch = normalized.match(/^CREATE TABLE(?: IF NOT EXISTS)? "([^"]+)"/i);
if (createTableMatch) {
return tableExists(sql, createTableMatch[1]);
}
const addColumnMatch = normalized.match(
/^ALTER TABLE "([^"]+)" ADD COLUMN(?: IF NOT EXISTS)? "([^"]+)"/i,
);
if (addColumnMatch) {
return columnExists(sql, addColumnMatch[1], addColumnMatch[2]);
}
const alterColumnTypeMatch = normalized.match(
/^ALTER TABLE "([^"]+)" ALTER COLUMN "([^"]+)" SET DATA TYPE ([A-Za-z0-9_]+)/i,
);
if (alterColumnTypeMatch) {
return columnHasDataType(
sql,
alterColumnTypeMatch[1],
alterColumnTypeMatch[2],
alterColumnTypeMatch[3],
);
}
const createIndexMatch = normalized.match(/^CREATE (?:UNIQUE )?INDEX(?: IF NOT EXISTS)? "([^"]+)"/i);
if (createIndexMatch) {
return indexExists(sql, createIndexMatch[1]);
}
const addConstraintMatch = normalized.match(/^ALTER TABLE "([^"]+)" ADD CONSTRAINT "([^"]+)"/i);
if (addConstraintMatch) {
return constraintExists(sql, addConstraintMatch[2]);
}
const createFunctionMatch = normalized.match(
/^CREATE OR REPLACE FUNCTION "?([A-Za-z_][A-Za-z0-9_]*)"?\s*\(/i,
);
if (createFunctionMatch) {
return functionExists(sql, createFunctionMatch[1]);
}
const createTriggerMatch = normalized.match(
/^CREATE TRIGGER "?([A-Za-z_][A-Za-z0-9_]*)"?/i,
);
if (createTriggerMatch) {
return triggerExists(sql, createTriggerMatch[1]);
}
// These native-runner repairs have persistent postconditions. Verify them
// instead of replaying them when a restored database is missing only the
// migration-history row.
if (
normalized.startsWith("WITH ranked AS (")
&& normalized.includes('UPDATE "heartbeat_run_events" AS event')
) {
return heartbeatEventSequencesAreUnique(sql);
}
if (
normalized.startsWith('UPDATE "heartbeat_runs" AS run')
&& normalized.includes('SET "next_event_seq" = COALESCE')
) {
return heartbeatNextEventSequencesAreCurrent(sql);
}
// If we cannot reason about a statement safely, require manual migration.
return false;
}
async function migrationContentAlreadyApplied(
sql: ReturnType<typeof postgres>,
migrationContent: string,
): Promise<boolean> {
const statements = splitMigrationStatements(migrationContent);
if (statements.length === 0) return false;
for (const statement of statements) {
const applied = await migrationStatementAlreadyApplied(sql, statement);
if (!applied) return false;
}
return true;
}
async function loadAppliedMigrations(
sql: ReturnType<typeof postgres>,
migrationTableSchema: string,
availableMigrations: string[],
): Promise<string[]> {
const quotedSchema = quoteIdentifier(migrationTableSchema);
const qualifiedTable = `${quotedSchema}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
if (columnNames.has("name")) {
const rows = await sql.unsafe<{ name: string }[]>(`SELECT name FROM ${qualifiedTable} ORDER BY id`);
return rows.map((row) => row.name).filter((name): name is string => Boolean(name));
}
if (columnNames.has("hash")) {
const rows = await sql.unsafe<{ hash: string }[]>(`SELECT hash FROM ${qualifiedTable} ORDER BY id`);
const hashesToMigrationFiles = await mapHashesToMigrationFiles(availableMigrations);
const appliedFromHashes = rows
.map((row) => hashesToMigrationFiles.get(row.hash))
.filter((name): name is string => Boolean(name));
if (appliedFromHashes.length > 0) {
// Best-effort: when all hashes resolve, this is authoritative.
if (appliedFromHashes.length === rows.length) return appliedFromHashes;
// Partial hash resolution can happen when files have changed; return what we can trust.
return appliedFromHashes;
}
// Fallback only when hashes are unavailable/unresolved.
if (columnNames.has("created_at")) {
const journalEntries = await listJournalMigrationEntries();
if (journalEntries.length > 0) {
const lastDbRows = await sql.unsafe<{ created_at: string | number | null }[]>(
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC LIMIT 1`,
);
const lastCreatedAt = Number(lastDbRows[0]?.created_at ?? -1);
if (Number.isFinite(lastCreatedAt) && lastCreatedAt >= 0) {
return journalEntries
.filter((entry) => availableMigrations.includes(entry.fileName))
.filter((entry) => entry.folderMillis <= lastCreatedAt)
.map((entry) => entry.fileName)
.slice(0, rows.length);
}
}
}
}
const rows = await sql.unsafe<{ id: number }[]>(`SELECT id FROM ${qualifiedTable} ORDER BY id`);
const journalMigrationFiles = await listJournalMigrationFiles();
const appliedFromIds = rows
.map((row) => journalMigrationFiles[row.id - 1])
.filter((name): name is string => Boolean(name));
if (appliedFromIds.length > 0) return appliedFromIds;
return availableMigrations.slice(0, Math.max(0, rows.length));
}
export type MigrationHistoryReconcileResult = {
repairedMigrations: string[];
remainingMigrations: string[];
};
export async function reconcilePendingMigrationHistory(
url: string,
): Promise<MigrationHistoryReconcileResult> {
const state = await inspectMigrations(url);
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
return { repairedMigrations: [], remainingMigrations: [] };
}
const sql = createUtilitySql(url);
const repairedMigrations: string[] = [];
try {
const journalEntries = await listJournalMigrationEntries();
const folderMillisByFile = new Map(journalEntries.map((entry) => [entry.fileName, entry.folderMillis]));
const migrationTableSchema = await discoverMigrationTableSchema(sql);
if (!migrationTableSchema) {
return { repairedMigrations, remainingMigrations: state.pendingMigrations };
}
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
for (const migrationFile of state.pendingMigrations) {
const migrationContent = await readMigrationFileContent(migrationFile);
const alreadyApplied = await migrationContentAlreadyApplied(sql, migrationContent);
if (!alreadyApplied) break;
const hash = createHash("sha256").update(migrationContent).digest("hex");
const folderMillis = folderMillisByFile.get(migrationFile) ?? Date.now();
const existingByHash = columnNames.has("hash")
? await sql.unsafe<{ created_at: string | number | null }[]>(
`SELECT created_at FROM ${qualifiedTable} WHERE hash = ${quoteLiteral(hash)} ORDER BY created_at DESC LIMIT 1`,
)
: [];
const existingByName = columnNames.has("name")
? await sql.unsafe<{ created_at: string | number | null }[]>(
`SELECT created_at FROM ${qualifiedTable} WHERE name = ${quoteLiteral(migrationFile)} ORDER BY created_at DESC LIMIT 1`,
)
: [];
if (existingByHash.length > 0 || existingByName.length > 0) {
if (columnNames.has("created_at")) {
const existingHashCreatedAt = Number(existingByHash[0]?.created_at ?? -1);
if (existingByHash.length > 0 && Number.isFinite(existingHashCreatedAt) && existingHashCreatedAt < folderMillis) {
await sql.unsafe(
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE hash = ${quoteLiteral(hash)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
);
}
const existingNameCreatedAt = Number(existingByName[0]?.created_at ?? -1);
if (existingByName.length > 0 && Number.isFinite(existingNameCreatedAt) && existingNameCreatedAt < folderMillis) {
await sql.unsafe(
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE name = ${quoteLiteral(migrationFile)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
);
}
}
repairedMigrations.push(migrationFile);
continue;
}
const insertColumns: string[] = [];
const insertValues: string[] = [];
if (columnNames.has("hash")) {
insertColumns.push(quoteIdentifier("hash"));
insertValues.push(quoteLiteral(hash));
}
if (columnNames.has("name")) {
insertColumns.push(quoteIdentifier("name"));
insertValues.push(quoteLiteral(migrationFile));
}
if (columnNames.has("created_at")) {
insertColumns.push(quoteIdentifier("created_at"));
insertValues.push(quoteLiteral(String(folderMillis)));
}
if (insertColumns.length === 0) break;
await sql.unsafe(
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
);
repairedMigrations.push(migrationFile);
}
} finally {
await sql.end();
}
const refreshed = await inspectMigrations(url);
return {
repairedMigrations,
remainingMigrations:
refreshed.status === "needsMigrations" ? refreshed.pendingMigrations : [],
};
}
async function discoverMigrationTableSchema(sql: ReturnType<typeof postgres>): Promise<string | null> {
const rows = await sql<{ schemaName: string }[]>`
SELECT n.nspname AS "schemaName"
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE c.relname = ${DRIZZLE_MIGRATIONS_TABLE} AND c.relkind = 'r'
`;
if (rows.length === 0) return null;
const drizzleSchema = rows.find(({ schemaName }) => schemaName === "drizzle");
if (drizzleSchema) return drizzleSchema.schemaName;
const publicSchema = rows.find(({ schemaName }) => schemaName === "public");
if (publicSchema) return publicSchema.schemaName;
return rows[0]?.schemaName ?? null;
}
export async function inspectMigrations(url: string): Promise<MigrationState> {
const sql = createUtilitySql(url);
try {
const availableMigrations = await listMigrationFiles();
const tableCountResult = await sql<{ count: number }[]>`
select count(*)::int as count
from information_schema.tables
where table_schema = 'public'
and table_type = 'BASE TABLE'
`;
const tableCount = tableCountResult[0]?.count ?? 0;
const migrationTableSchema = await discoverMigrationTableSchema(sql);
if (!migrationTableSchema) {
if (tableCount > 0) {
return {
status: "needsMigrations",
tableCount,
availableMigrations,
appliedMigrations: [],
pendingMigrations: availableMigrations,
journalEntryCount: 0,
reason: "no-migration-journal-non-empty-db",
};
}
return {
status: "needsMigrations",
tableCount,
availableMigrations,
appliedMigrations: [],
pendingMigrations: availableMigrations,
journalEntryCount: 0,
reason: "no-migration-journal-empty-db",
};
}
const qualifiedMigrationTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
const journalCountRows = await sql.unsafe<{ count: number }[]>(
`SELECT count(*)::int AS count FROM ${qualifiedMigrationTable}`,
);
const journalEntryCount = journalCountRows[0]?.count ?? 0;
const appliedMigrations = await loadAppliedMigrations(sql, migrationTableSchema, availableMigrations);
const pendingMigrations = availableMigrations.filter((name) => !appliedMigrations.includes(name));
if (pendingMigrations.length === 0) {
return {
status: "upToDate",
tableCount,
availableMigrations,
appliedMigrations,
journalEntryCount,
};
}
return {
status: "needsMigrations",
tableCount,
availableMigrations,
appliedMigrations,
pendingMigrations,
journalEntryCount,
reason: "pending-migrations",
};
} finally {
await sql.end();
}
}
export async function applyPendingMigrations(url: string): Promise<void> {
const initialState = await inspectMigrations(url);
if (initialState.status === "upToDate") return;
if (initialState.reason === "no-migration-journal-empty-db") {
// Use the same per-file executor for bootstrap and upgrades: Drizzle wraps
// the whole journal in one transaction and cannot run committing backfills.
await applyPendingMigrationsManually(url, initialState.pendingMigrations);
const bootstrappedState = await inspectMigrations(url);
if (bootstrappedState.status === "upToDate") return;
throw new Error(
`Failed to bootstrap migrations: ${bootstrappedState.pendingMigrations.join(", ")}`,
);
}
if (initialState.reason === "no-migration-journal-non-empty-db") {
throw new Error(
"Database has tables but no migration journal; automatic migration is unsafe. Initialize migration history manually.",
);
}
let state = await inspectMigrations(url);
if (state.status === "upToDate") return;
const repair = await reconcilePendingMigrationHistory(url);
if (repair.repairedMigrations.length > 0) {
state = await inspectMigrations(url);
if (state.status === "upToDate") return;
}
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
throw new Error("Migrations are still pending after migration-history reconciliation; run inspectMigrations for details.");
}
await applyPendingMigrationsManually(url, state.pendingMigrations);
const finalState = await inspectMigrations(url);
if (finalState.status !== "upToDate") {
throw new Error(
`Failed to apply pending migrations: ${finalState.pendingMigrations.join(", ")}`,
);
}
}
export type MigrationBootstrapResult =
| { migrated: true; reason: "migrated-empty-db"; tableCount: 0 }
| { migrated: false; reason: "already-migrated"; tableCount: number }
| { migrated: false; reason: "not-empty-no-migration-journal"; tableCount: number };
export async function migratePostgresIfEmpty(url: string): Promise<MigrationBootstrapResult> {
const sql = createUtilitySql(url);
try {
const migrationTableSchema = await discoverMigrationTableSchema(sql);
const tableCountResult = await sql<{ count: number }[]>`
select count(*)::int as count
from information_schema.tables
where table_schema = 'public'
and table_type = 'BASE TABLE'
`;
const tableCount = tableCountResult[0]?.count ?? 0;
if (migrationTableSchema) {
return { migrated: false, reason: "already-migrated", tableCount };
}
if (tableCount > 0) {
return { migrated: false, reason: "not-empty-no-migration-journal", tableCount };
}
await applyPendingMigrationsManually(url, await listMigrationFiles());
return { migrated: true, reason: "migrated-empty-db", tableCount: 0 };
} finally {
await sql.end();
}
}
export async function ensurePostgresDatabase(
url: string,
databaseName: string,
): Promise<"created" | "exists"> {
if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(databaseName)) {
throw new Error(`Unsafe database name: ${databaseName}`);
}
const sql = createUtilitySql(url);
try {
const existing = await sql<{ one: number }[]>`
select 1 as one from pg_database where datname = ${databaseName} limit 1
`;
if (existing.length > 0) return "exists";
await sql.unsafe(`create database "${databaseName}" encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
return "created";
} finally {
await sql.end();
}
}
export async function resetPostgresDatabase(
url: string,
databaseName: string,
): Promise<"reset"> {
const quotedDatabaseName = quoteIdentifier(databaseName);
const sql = createUtilitySql(url);
try {
await sql`
select pg_terminate_backend(pid)
from pg_stat_activity
where datname = ${databaseName}
and pid <> pg_backend_pid()
`;
await sql.unsafe(`drop database if exists ${quotedDatabaseName}`);
await sql.unsafe(`create database ${quotedDatabaseName} encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
return "reset";
} finally {
await sql.end();
}
}
export type Db = ReturnType<typeof createDb>;