mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process runs every agent on one Node main thread, and each new database connection costs a TLS handshake, about 0.36 ms of SCRAM key derivation on that thread, and a type-fetch query > - `withDedicatedDbConnection` gives lock holders a private session; since #21693 it keeps at most 4 released sessions and closes the rest > - When the same burst of more than 4 dedicated holders comes back, every holder above 4 opens a new connection again > - A harness on embedded PostgreSQL measured 504 new connections per 1,000 uses for bursts of 8 and 760 for bursts of 16 > - This pull request keeps released dedicated clients for reuse while idle plus in-use clients stay within `DATABASE_DEDICATED_CONNECTION_MAX`; the driver idle timeout still closes their sessions when they are not used > - The benefit is 8 and 16 new connections per 1,000 uses for the same bursts, with no change to any pool maximum or to `databaseConnectionBudget()` ## Linked Issues or Issue Description **What is the problem?** A staging measurement showed `SET application_name='paperclip-runner-checkpoint'` at 3.17/s and `SET application_name='paperclip'` at 3.86/s with 7 live runs. This looked like connection churn. The investigation found two separate facts: 1. The `SET` statements are not new client connections. postgres.js sends `application_name` in the startup packet, never as a `SET` statement (`StartupMessage` in postgres.js `src/connection.js`). PgBouncer in transaction mode sends `SET application_name` itself when it gives a client transaction a server connection that a client with another `application_name` used last. With one shared server pool, every checkpoint transaction that lands after a main-pool transaction costs one `SET`, and the reverse. The rate follows checkpoint transactions, not connects. The checkpoint pool opens at most 3 connections per 1,000 saves (measured below). 2. Dedicated connections did churn by construction for recurring bursts of more than 4 holders (measured below). This pull request fixes that. **Steps to reproduce** Run `packages/db/src/connection-reuse-census.test.ts`. It counts `pg_stat_database.sessions` around 1,000 operations. **Expected behavior** A recurring burst reuses the sessions that the previous burst opened. ## What Changed - `withDedicatedDbConnection` keeps a released client idle when fewer than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle, as before, or when idle plus in-use clients stay within the dedicated limit (`DATABASE_DEDICATED_CONNECTION_MAX`, default 24). A client whose action failed, an ended client and clients after shutdown are still closed. - Each retained client keeps the main pool's `idle_timeout` (default 60 s), so its session closes after that idle time, as the main pool's do. The next use reconnects. - The worst case stays at `2 x DATABASE_DEDICATED_CONNECTION_MAX + 4`: a new client opens only when no client is idle, and a client is kept above 4 only while idle plus in-use is within the limit. `databaseConnectionBudget()` and every pool maximum are unchanged. - New `connection-reuse-census.test.ts` measures connections opened per 1,000 checkpoint saves and per 1,000 dedicated uses. - `dedicated-connection-pool.test.ts`: the burst test now checks that the next burst reuses all sessions and that they close after the idle timeout (1 s in the test). Before, it checked that at most 4 stayed open. - `doc/DATABASE.md` describes the new retention rule. ## Verification Connections opened per 1,000 operations, embedded PostgreSQL, `pg_stat_database.sessions` (load average about 10.5 on a shared machine): | Operation | Before | After | | --- | --- | --- | | Checkpoint saves, concurrency 1 / 7 / 24 | 1 / 3 / 3 | 1 / 3 / 3 | | Dedicated uses, steady concurrency 1 / 4 / 8 / 16 | 1 / 4 / 8 / 16 | 1 / 4 / 8 / 16 | | Dedicated uses, recurring bursts of 8 | 504 | 8 | | Dedicated uses, recurring bursts of 16 | 760 | 16 | - `pnpm --filter @paperclipai/db exec vitest run src/connection-reuse-census.test.ts src/dedicated-connection-pool.test.ts src/dedicated-connection-admission.test.ts src/client-options.test.ts`: 38 passed. The burst cases fail on `master` (504 and 760 against a bound of 8 and 16). ## Risks - After a burst, up to `DATABASE_DEDICATED_CONNECTION_MAX` dedicated sessions stay open for up to `DATABASE_IDLE_TIMEOUT_SECONDS` (60 s) instead of closing at once above 4. This stays inside the connection budget. With `DATABASE_IDLE_TIMEOUT_SECONDS=0` they stay open until shutdown, like the main pool. - Retained sessions keep the existing rule: actions keep session state transaction-scoped. - The PgBouncer `SET application_name` statements stay. Removing them needs one shared `application_name` for all pools, which removes the per-pool names that operators and tests use. That is a separate decision. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1638 lines
63 KiB
TypeScript
1638 lines
63 KiB
TypeScript
import { AsyncLocalStorage } from "node:async_hooks";
|
|
import { createHash } from "node:crypto";
|
|
import { readFile, readdir } from "node:fs/promises";
|
|
import { fileURLToPath } from "node:url";
|
|
import postgres from "postgres";
|
|
import { installDatabaseWorkSignals } from "./work-signals.js";
|
|
import { installDatabaseResponseCapture } from "./response-capture.js";
|
|
import { createPreparedQueryState, installPreparedQueries, preparedQueryStats, type PreparedQueryState, type PreparedQueryStats } from "./prepared-queries.js";
|
|
import { drizzleWithLazyTransactionQueries } from "./lazy-transaction-queries.js";
|
|
import * as schema from "./schema/index.js";
|
|
|
|
const MIGRATIONS_FOLDER = fileURLToPath(new URL("./migrations", import.meta.url));
|
|
const DRIZZLE_MIGRATIONS_TABLE = "__drizzle_migrations";
|
|
const MIGRATIONS_JOURNAL_JSON = fileURLToPath(new URL("./migrations/meta/_journal.json", import.meta.url));
|
|
const PROJECT_ENVIRONMENT_OVERLAP = {
|
|
dropFile: "0466_drop_project_environment.sql",
|
|
dropSha256: "07ca1c666c6471fdb9ce7a0a5fb31bf1a074138466fa8daae50b53199b8e0d7c",
|
|
repairFile: "0467_fair_human_torch.sql",
|
|
repairSha256: "50ce0eb2567570b098bea1de92fe6a220d139e4ba83438195fbbad1be51e31a1",
|
|
} as const;
|
|
|
|
function createUtilitySql(url: string) {
|
|
return postgres(url, { max: 1, onnotice: () => {} });
|
|
}
|
|
|
|
type RegisteredPostgresClient = ReturnType<typeof postgres>;
|
|
|
|
/**
|
|
* Derives a registry key from a connection URL's host and port only. We must
|
|
* not retain or log the full URL, because it carries credentials.
|
|
*/
|
|
function hostPortKey(url: string): string {
|
|
const parsed = new URL(url);
|
|
return `${parsed.hostname}:${parsed.port || "5432"}`;
|
|
}
|
|
|
|
/**
|
|
* Same as `hostPortKey`, but returns `null` instead of throwing when the URL
|
|
* does not parse. `postgres(url)` tolerates a value `new URL()` rejects (an
|
|
* empty string falls back to the `PG*` environment variables), so `createDb`
|
|
* must tolerate it too: skip the registry entry and let the driver decide
|
|
* the outcome, instead of throwing an error the driver itself would not.
|
|
*/
|
|
function hostPortKeyOrNull(url: string): string | null {
|
|
try {
|
|
return hostPortKey(url);
|
|
} catch (error) {
|
|
if (error instanceof TypeError && (error as NodeJS.ErrnoException).code === "ERR_INVALID_URL") return null;
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
// Tracks every client `createDb` hands out, keyed by host and port, so a test
|
|
// fixture can end them before it stops the Postgres cluster they point at. A
|
|
// `WeakRef` plus `FinalizationRegistry` means a long-lived process (a real
|
|
// server) retains nothing extra: an unreferenced client is pruned on its own.
|
|
const clientsByHostPort = new Map<string, Set<WeakRef<RegisteredPostgresClient>>>();
|
|
const clientFinalizer = new FinalizationRegistry<{ hostPortKey: string; ref: WeakRef<RegisteredPostgresClient> }>(
|
|
({ hostPortKey, ref }) => {
|
|
const refs = clientsByHostPort.get(hostPortKey);
|
|
if (!refs) return;
|
|
refs.delete(ref);
|
|
if (refs.size === 0) clientsByHostPort.delete(hostPortKey);
|
|
},
|
|
);
|
|
|
|
function registerClient(key: string, client: RegisteredPostgresClient): void {
|
|
const ref = new WeakRef(client);
|
|
let refs = clientsByHostPort.get(key);
|
|
if (!refs) {
|
|
refs = new Set();
|
|
clientsByHostPort.set(key, refs);
|
|
}
|
|
refs.add(ref);
|
|
clientFinalizer.register(client, { hostPortKey: key, ref }, ref);
|
|
}
|
|
|
|
/**
|
|
* Ends every live client `createDb` handed out for the given URL's host and
|
|
* port, then forgets them. Call this before stopping a Postgres cluster: a
|
|
* client that outlives the cluster it points at can crash the process (a
|
|
* reserved connection's deferred write firing after the socket is gone).
|
|
* Swallows individual `end()` errors so one bad client cannot block the rest.
|
|
* An owned database on a shared test host must select database scope instead;
|
|
* its teardown cannot close another fixture's clients on that host.
|
|
*/
|
|
export async function closeRegisteredClients(url: string, options?: { scope: "database" }): Promise<void> {
|
|
const key = hostPortKeyOrNull(url);
|
|
if (key === null) return;
|
|
const database = options?.scope === "database" ? decodeURIComponent(new URL(url).pathname.slice(1)) : undefined;
|
|
if (database === "") throw new Error("Database-scoped teardown requires an explicit database name");
|
|
const refs = clientsByHostPort.get(key);
|
|
if (!refs) return;
|
|
|
|
const clients: RegisteredPostgresClient[] = [];
|
|
for (const ref of refs) {
|
|
const client = ref.deref();
|
|
if (client && database !== undefined && client.options.database !== database) continue;
|
|
refs.delete(ref);
|
|
clientFinalizer.unregister(ref);
|
|
if (client) clients.push(client);
|
|
}
|
|
if (refs.size === 0) clientsByHostPort.delete(key);
|
|
|
|
await Promise.all(clients.map((client) => client.end({ timeout: 1 }).catch(() => {})));
|
|
}
|
|
|
|
function isSafeIdentifier(value: string): boolean {
|
|
return /^[A-Za-z_][A-Za-z0-9_]*$/.test(value);
|
|
}
|
|
|
|
function quoteIdentifier(value: string): string {
|
|
if (!isSafeIdentifier(value)) throw new Error(`Unsafe SQL identifier: ${value}`);
|
|
return `"${value.replaceAll("\"", "\"\"")}"`;
|
|
}
|
|
|
|
function quoteLiteral(value: string): string {
|
|
return `'${value.replaceAll("'", "''")}'`;
|
|
}
|
|
|
|
function splitMigrationStatements(content: string): string[] {
|
|
return content
|
|
.split("--> statement-breakpoint")
|
|
.map((statement) => statement.trim())
|
|
.filter((statement) => statement.length > 0);
|
|
}
|
|
|
|
export type MigrationState =
|
|
| {
|
|
status: "upToDate";
|
|
tableCount: number;
|
|
availableMigrations: string[];
|
|
appliedMigrations: string[];
|
|
journalEntryCount: number;
|
|
}
|
|
| {
|
|
status: "needsMigrations";
|
|
tableCount: number;
|
|
availableMigrations: string[];
|
|
appliedMigrations: string[];
|
|
pendingMigrations: string[];
|
|
journalEntryCount: number;
|
|
reason: "no-migration-journal-empty-db" | "no-migration-journal-non-empty-db" | "pending-migrations";
|
|
};
|
|
|
|
export interface DatabaseClientOptions {
|
|
prepare?: boolean;
|
|
/** postgres.js `max` — connection pool size (driver default: 10). */
|
|
maxConnections?: number;
|
|
/** postgres.js `idle_timeout` in seconds (driver default: disabled). */
|
|
idleTimeoutSeconds?: number;
|
|
/** postgres.js `connect_timeout` in seconds (driver default: 30). */
|
|
connectTimeoutSeconds?: number;
|
|
/**
|
|
* postgres.js `max_lifetime` in seconds. Bounds how long one pooled
|
|
* connection is reused before the client replaces it (driver default: a
|
|
* random value between 30 and 60 minutes).
|
|
*/
|
|
maxLifetimeSeconds?: number;
|
|
/**
|
|
* postgres.js `connection.application_name`, shown in
|
|
* `pg_stat_activity.application_name`. Lets an operator tell Paperclip's
|
|
* pool apart from other clients of the same database (driver default:
|
|
* `postgres.js`).
|
|
*/
|
|
applicationName?: string;
|
|
/**
|
|
* Connection-level `statement_timeout` in milliseconds, sent as a startup
|
|
* parameter (no extra round trip). For dedicated single-purpose pools only.
|
|
*/
|
|
statementTimeoutMs?: number;
|
|
dedicatedConnectionMax?: number;
|
|
dedicatedConnectionWaitMs?: number;
|
|
}
|
|
|
|
/**
|
|
* Idle pooled connections close after this many seconds unless
|
|
* `DATABASE_IDLE_TIMEOUT_SECONDS` says otherwise. The driver default keeps an
|
|
* idle connection open forever, so a process that stops issuing queries still
|
|
* holds every backend it ever opened. Set `DATABASE_IDLE_TIMEOUT_SECONDS=0`
|
|
* to restore the driver default.
|
|
*/
|
|
export const DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS = 60;
|
|
/** `application_name` reported to PostgreSQL unless `DATABASE_APPLICATION_NAME` overrides it. */
|
|
export const DEFAULT_DATABASE_APPLICATION_NAME = "paperclip";
|
|
|
|
function envBoolean(env: NodeJS.ProcessEnv, name: string): boolean | undefined {
|
|
const value = env[name]?.trim().toLowerCase();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (value === "true" || value === "1") return true;
|
|
if (value === "false" || value === "0") return false;
|
|
throw new Error(`${name} must be "true" or "false", got: ${env[name]}`);
|
|
}
|
|
|
|
function envPositiveInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (!/^[1-9]\d*$/.test(value)) {
|
|
throw new Error(`${name} must be a positive integer, got: ${env[name]}`);
|
|
}
|
|
return Number.parseInt(value, 10);
|
|
}
|
|
|
|
function envNonNegativeInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (!/^(?:0|[1-9]\d*)$/.test(value)) {
|
|
throw new Error(`${name} must be a non-negative integer, got: ${env[name]}`);
|
|
}
|
|
return Number.parseInt(value, 10);
|
|
}
|
|
|
|
function envNonEmptyString(env: NodeJS.ProcessEnv, name: string): string | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
return value;
|
|
}
|
|
|
|
/**
|
|
* Database client tuning from the environment, so hosted deployments can
|
|
* adapt to their connection topology (pooled endpoints, network latency)
|
|
* without editing source. Every variable is optional. This function returns
|
|
* only the values the environment sets; `resolveDatabaseClientOptions` adds
|
|
* Paperclip's own defaults on top, and the driver defaults apply to the rest
|
|
* — self-hosted setups need none of these.
|
|
*/
|
|
export function databaseClientOptionsFromEnv(env: NodeJS.ProcessEnv = process.env): DatabaseClientOptions {
|
|
const options: DatabaseClientOptions = {};
|
|
const prepare = envBoolean(env, "DATABASE_PREPARED_STATEMENTS");
|
|
if (prepare !== undefined) options.prepare = prepare;
|
|
const maxConnections = envPositiveInteger(env, "DATABASE_POOL_MAX");
|
|
if (maxConnections !== undefined) options.maxConnections = maxConnections;
|
|
// `0` is allowed here: it disables idle reaping (the driver default).
|
|
const idleTimeoutSeconds = envNonNegativeInteger(env, "DATABASE_IDLE_TIMEOUT_SECONDS");
|
|
if (idleTimeoutSeconds !== undefined) options.idleTimeoutSeconds = idleTimeoutSeconds;
|
|
const connectTimeoutSeconds = envPositiveInteger(env, "DATABASE_CONNECT_TIMEOUT_SECONDS");
|
|
if (connectTimeoutSeconds !== undefined) options.connectTimeoutSeconds = connectTimeoutSeconds;
|
|
const maxLifetimeSeconds = envPositiveInteger(env, "DATABASE_MAX_LIFETIME_SECONDS");
|
|
if (maxLifetimeSeconds !== undefined) options.maxLifetimeSeconds = maxLifetimeSeconds;
|
|
const applicationName = envNonEmptyString(env, "DATABASE_APPLICATION_NAME");
|
|
if (applicationName !== undefined) options.applicationName = applicationName;
|
|
const dedicatedConnectionMax = envPositiveInteger(env, "DATABASE_DEDICATED_CONNECTION_MAX");
|
|
if (dedicatedConnectionMax !== undefined) options.dedicatedConnectionMax = dedicatedConnectionMax;
|
|
const dedicatedConnectionWaitMs = envPositiveInteger(env, "DATABASE_DEDICATED_CONNECTION_WAIT_MS");
|
|
if (dedicatedConnectionWaitMs !== undefined) options.dedicatedConnectionWaitMs = dedicatedConnectionWaitMs;
|
|
return options;
|
|
}
|
|
|
|
/**
|
|
* Fills in Paperclip's defaults for the options the caller left unset: idle
|
|
* connections are reaped after `DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS`, and the
|
|
* pool identifies itself as `DEFAULT_DATABASE_APPLICATION_NAME`. Everything
|
|
* else stays at the driver default. An explicit value (including
|
|
* `idleTimeoutSeconds: 0`) always wins over the default.
|
|
*/
|
|
export function resolveDatabaseClientOptions(options: DatabaseClientOptions): DatabaseClientOptions {
|
|
return {
|
|
...options,
|
|
idleTimeoutSeconds: options.idleTimeoutSeconds ?? DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS,
|
|
applicationName: options.applicationName ?? DEFAULT_DATABASE_APPLICATION_NAME,
|
|
};
|
|
}
|
|
|
|
export function postgresJsOptions(options: DatabaseClientOptions): Record<string, unknown> {
|
|
const driverOptions: Record<string, unknown> = {};
|
|
if (options.prepare !== undefined) driverOptions.prepare = options.prepare;
|
|
if (options.maxConnections !== undefined) driverOptions.max = options.maxConnections;
|
|
if (options.idleTimeoutSeconds !== undefined) driverOptions.idle_timeout = options.idleTimeoutSeconds;
|
|
if (options.connectTimeoutSeconds !== undefined) driverOptions.connect_timeout = options.connectTimeoutSeconds;
|
|
if (options.maxLifetimeSeconds !== undefined) driverOptions.max_lifetime = options.maxLifetimeSeconds;
|
|
const connection: Record<string, string> = {};
|
|
if (options.applicationName !== undefined) connection.application_name = options.applicationName;
|
|
if (options.statementTimeoutMs !== undefined) connection.statement_timeout = String(options.statementTimeoutMs);
|
|
if (Object.keys(connection).length) driverOptions.connection = connection;
|
|
return driverOptions;
|
|
}
|
|
|
|
// A long advisory-lock transaction must not borrow the normal pool that its
|
|
// work needs for progress writes. Keep connection configuration private to the
|
|
// originating Db lifetime; callers receive neither URLs nor credentials.
|
|
const ownedDatabaseListenerFactories = new WeakMap<object, () => Db>();
|
|
|
|
export const DEDICATED_CONNECTION_IDLE_MAX = 4;
|
|
export const DEFAULT_DEDICATED_CONNECTION_MAX = 24;
|
|
export const DEFAULT_DEDICATED_CONNECTION_WAIT_MS = 30_000;
|
|
|
|
type DedicatedConnectionWaiter = { grant(): void; timer: ReturnType<typeof setTimeout> };
|
|
type DedicatedConnectionSet = {
|
|
create(): Db;
|
|
idle: Db[];
|
|
ended: WeakSet<Db>;
|
|
closed: boolean;
|
|
limit: number;
|
|
waitMs: number;
|
|
inUse: number;
|
|
waiters: DedicatedConnectionWaiter[];
|
|
stats: DedicatedDbConnectionStats;
|
|
};
|
|
export type DedicatedDbConnectionStats = { limit: number; inUse: number; peakInUse: number; waiting: number;
|
|
queued: number; overLimit: number; nestedOverLimit: number };
|
|
const dedicatedConnectionSets = new WeakMap<object, DedicatedConnectionSet>();
|
|
type DedicatedConnectionHold = { set: DedicatedConnectionSet; active: boolean };
|
|
const heldDedicatedConnections = new AsyncLocalStorage<readonly DedicatedConnectionHold[]>();
|
|
const openTransactions = new AsyncLocalStorage<{ active: boolean }>();
|
|
|
|
function dedicatedConnectionSet(create: DedicatedConnectionSet["create"], resolved: DatabaseClientOptions): DedicatedConnectionSet {
|
|
const limit = resolved.dedicatedConnectionMax ?? DEFAULT_DEDICATED_CONNECTION_MAX;
|
|
return { create, idle: [], ended: new WeakSet(), closed: false, limit, inUse: 0, waiters: [],
|
|
waitMs: resolved.dedicatedConnectionWaitMs ?? DEFAULT_DEDICATED_CONNECTION_WAIT_MS,
|
|
stats: { limit, inUse: 0, peakInUse: 0, waiting: 0, queued: 0, overLimit: 0, nestedOverLimit: 0 } };
|
|
}
|
|
|
|
function takeDedicatedSlot(set: DedicatedConnectionSet) {
|
|
set.inUse++;
|
|
set.stats.inUse = set.inUse;
|
|
set.stats.peakInUse = Math.max(set.stats.peakInUse, set.inUse);
|
|
}
|
|
|
|
function releaseDedicatedSlot(set: DedicatedConnectionSet) {
|
|
set.inUse--;
|
|
set.stats.inUse = set.inUse;
|
|
while (set.waiters.length && set.inUse < set.limit) set.waiters.shift()!.grant();
|
|
set.stats.waiting = set.waiters.length;
|
|
}
|
|
|
|
function holdsLiveDedicatedOrTransaction(set: DedicatedConnectionSet): boolean {
|
|
return Boolean(heldDedicatedConnections.getStore()?.some(hold => hold.active && hold.set === set)
|
|
|| openTransactions.getStore()?.active);
|
|
}
|
|
|
|
async function admitDedicatedConnection(set: DedicatedConnectionSet, nested: boolean): Promise<void> {
|
|
if (nested || (set.inUse < set.limit && set.waiters.length === 0)) {
|
|
if (nested && set.inUse >= set.limit) set.stats.nestedOverLimit++;
|
|
takeDedicatedSlot(set);
|
|
return;
|
|
}
|
|
set.stats.queued++;
|
|
await new Promise<void>(resolve => {
|
|
const waiter: DedicatedConnectionWaiter = {
|
|
grant: () => { clearTimeout(waiter.timer); takeDedicatedSlot(set); resolve(); },
|
|
timer: setTimeout(() => {
|
|
const index = set.waiters.indexOf(waiter);
|
|
if (index < 0) return;
|
|
set.waiters.splice(index, 1);
|
|
set.stats.waiting = set.waiters.length;
|
|
set.stats.overLimit++;
|
|
takeDedicatedSlot(set);
|
|
resolve();
|
|
}, set.waitMs),
|
|
};
|
|
set.waiters.push(waiter);
|
|
set.stats.waiting = set.waiters.length;
|
|
});
|
|
}
|
|
|
|
function markOpenTransactions(db: Db) {
|
|
const transaction = db.transaction.bind(db) as (...args: unknown[]) => Promise<unknown>;
|
|
db.transaction = ((fn: (tx: unknown) => Promise<unknown>, ...rest: unknown[]) => transaction(async (tx: unknown) => {
|
|
const open = { active: true };
|
|
try { return await openTransactions.run(open, () => fn(tx)); }
|
|
finally { open.active = false; }
|
|
}, ...rest)) as Db["transaction"];
|
|
}
|
|
|
|
export const DRIVER_DEFAULT_POOL_MAX_CONNECTIONS = 10;
|
|
export const PRIVATE_POOL_MAX_CONNECTIONS = {
|
|
liveness: 2, legacyControllerLease: 1, controllerInstanceLease: 1, checkpoint: 3, fileStorage: 2,
|
|
} as const;
|
|
export const DATABASE_LISTENER_CONNECTIONS = 2;
|
|
|
|
export function databaseConnectionBudget(options: DatabaseClientOptions = {}) {
|
|
const main = options.maxConnections ?? DRIVER_DEFAULT_POOL_MAX_CONNECTIONS;
|
|
const privatePools = Object.values(PRIVATE_POOL_MAX_CONNECTIONS).reduce((sum, value) => sum + value, 0);
|
|
const dedicatedLimit = options.dedicatedConnectionMax ?? DEFAULT_DEDICATED_CONNECTION_MAX;
|
|
const dedicated = 2 * dedicatedLimit + DEDICATED_CONNECTION_IDLE_MAX;
|
|
return { main, privatePools, listeners: DATABASE_LISTENER_CONNECTIONS, dedicatedLimit, dedicated,
|
|
total: main + privatePools + DATABASE_LISTENER_CONNECTIONS + dedicated };
|
|
}
|
|
|
|
const preparedQueryStates = new WeakMap<object, PreparedQueryState>();
|
|
|
|
export function preparedStatementStats(db: Db): PreparedQueryStats | null {
|
|
const state = preparedQueryStates.get(db);
|
|
return state ? preparedQueryStats(state) : null;
|
|
}
|
|
|
|
export function dedicatedDbConnectionStats(db: Db): DedicatedDbConnectionStats | null {
|
|
const set = dedicatedConnectionSets.get(db);
|
|
return set ? { ...set.stats } : null;
|
|
}
|
|
|
|
function trackClientEnd(client: Db, onEnd: (timeoutSeconds: number | undefined) => Promise<void> | void): void {
|
|
const sql = client.$client;
|
|
const end = sql.end;
|
|
sql.end = async (options?: { timeout?: number }) => {
|
|
await Promise.all([onEnd(options?.timeout), end(options)]);
|
|
};
|
|
}
|
|
|
|
async function endDedicatedConnections(clients: Db[], timeoutSeconds: number): Promise<void> {
|
|
await Promise.all(clients.map(client => client.$client.end({ timeout: timeoutSeconds }).catch(() => undefined)));
|
|
}
|
|
|
|
export async function closeDedicatedDbConnections(db: Db, timeoutSeconds: number): Promise<void> {
|
|
const set = dedicatedConnectionSets.get(db);
|
|
if (!set) return;
|
|
set.closed = true;
|
|
await endDedicatedConnections(set.idle.splice(0), timeoutSeconds);
|
|
}
|
|
|
|
export interface OwnedDatabaseListener {
|
|
listen: Db["$client"]["listen"];
|
|
close(): Promise<void>;
|
|
}
|
|
|
|
export function createOwnedDatabaseListener(db: Db): OwnedDatabaseListener {
|
|
const factory = ownedDatabaseListenerFactories.get(db);
|
|
if (!factory) throw new Error("owned_listener_requires_create_db");
|
|
const owned = factory();
|
|
return {
|
|
listen: owned.$client.listen.bind(owned.$client),
|
|
close: () => owned.$client.end({ timeout: 1 }),
|
|
};
|
|
}
|
|
|
|
export type DedicatedDbConnectionOptions = { admitWithoutWaiting?: boolean };
|
|
|
|
export async function withDedicatedDbConnection<T>(db: Db, action: (dedicated: Db) => Promise<T>,
|
|
options: DedicatedDbConnectionOptions = {}): Promise<T> {
|
|
const set = dedicatedConnectionSets.get(db);
|
|
if (!set) throw new Error("dedicated_connection_requires_create_db");
|
|
await admitDedicatedConnection(set, options.admitWithoutWaiting === true || holdsLiveDedicatedOrTransaction(set));
|
|
const hold: DedicatedConnectionHold = { set, active: true };
|
|
let dedicated: Db | undefined;
|
|
let completed = false;
|
|
try {
|
|
while (!dedicated && set.idle.length) {
|
|
const candidate = set.idle.pop()!;
|
|
if (!set.ended.has(candidate)) dedicated = candidate;
|
|
}
|
|
dedicated ??= set.create();
|
|
const connection = dedicated;
|
|
const result = await heldDedicatedConnections.run([...(heldDedicatedConnections.getStore() ?? []), hold], () => action(connection));
|
|
completed = true;
|
|
return result;
|
|
} finally {
|
|
hold.active = false;
|
|
try {
|
|
if (dedicated && completed && !set.closed && !set.ended.has(dedicated)
|
|
&& (set.idle.length < DEDICATED_CONNECTION_IDLE_MAX || set.idle.length + set.inUse <= set.limit)) {
|
|
set.idle.push(dedicated);
|
|
} else if (dedicated && !set.ended.has(dedicated)) {
|
|
await dedicated.$client.end({ timeout: 1 });
|
|
}
|
|
} finally {
|
|
releaseDedicatedSlot(set);
|
|
}
|
|
}
|
|
}
|
|
|
|
// postgres.js pipelines a query onto a busy connection when none is idle, so
|
|
// a short liveness write can wait behind an unrelated slow statement with no
|
|
// client-side bound. Liveness writes (controller-lease renewal) get a small
|
|
// private pool, created on first use and shared for the owner's lifetime.
|
|
const livenessDbFactories = new WeakMap<object, () => Db>();
|
|
const livenessDbs = new WeakMap<object, Db>();
|
|
|
|
/** The private liveness pool for a `createDb` client; other clients (test
|
|
* doubles, transactions) are returned unchanged. */
|
|
export function livenessDb(db: Db): Db {
|
|
const existing = livenessDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = livenessDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory();
|
|
livenessDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
const controllerLeaseDbFactories = new WeakMap<object, () => Db>();
|
|
const controllerLeaseDbs = new WeakMap<object, Db>();
|
|
|
|
export function controllerLeaseDb(db: Db): Db {
|
|
const existing = controllerLeaseDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = controllerLeaseDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory();
|
|
controllerLeaseDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
const controllerInstanceLeaseDbFactories = new WeakMap<object, (statementTimeoutMs: number | null) => Db>();
|
|
const controllerInstanceLeaseDbs = new WeakMap<object, Db>();
|
|
|
|
/** One private connection for the server's controller-instance lease, so its
|
|
* renewal never waits behind request or worker traffic on the main pool.
|
|
* `statementTimeoutMs` (first call wins) is a connection-level
|
|
* `statement_timeout`: the server cancels a slow renewal, so it never holds the
|
|
* single connection that the next renewal needs, at no extra round trip. */
|
|
export function controllerInstanceLeaseDb(db: Db, options: { statementTimeoutMs?: number | null } = {}): Db {
|
|
const existing = controllerInstanceLeaseDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = controllerInstanceLeaseDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory(options.statementTimeoutMs ?? null);
|
|
controllerInstanceLeaseDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
/** Replaces the lease connection with one that sends no `statement_timeout`
|
|
* startup parameter, for a connection pooler that refuses it. */
|
|
export function controllerInstanceLeaseDbWithoutStatementTimeout(db: Db): Db {
|
|
const factory = controllerInstanceLeaseDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const previous = controllerInstanceLeaseDbs.get(db);
|
|
const created = factory(null);
|
|
controllerInstanceLeaseDbs.set(db, created);
|
|
void previous?.$client.end({ timeout: 0 }).catch(() => undefined);
|
|
return created;
|
|
}
|
|
|
|
/** Ends the liveness pool if one was created. The ended pool stays cached, so
|
|
* a late renewal fails like a query on the ended primary pool. */
|
|
export async function closeLivenessDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
await Promise.all([livenessDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
|
|
controllerLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
|
|
controllerInstanceLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds })]);
|
|
}
|
|
|
|
// Control checkpoints must not occupy the request or lease-renewal pools
|
|
// during an aligned burst. Share at most three lazy connections per owner.
|
|
const checkpointDbFactories = new WeakMap<object, () => Db>();
|
|
const checkpointDbs = new WeakMap<object, Db>();
|
|
const closedCheckpointOwners = new WeakSet<object>();
|
|
|
|
/** The bounded controller checkpoint pool; transactions and test doubles keep
|
|
* their existing database when they have no owning client factory. */
|
|
export function checkpointDb(db: Db): Db {
|
|
const existing = checkpointDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = checkpointDbFactories.get(db);
|
|
if (!factory) return db;
|
|
if (closedCheckpointOwners.has(db)) throw new Error("checkpoint_pool_closed");
|
|
const created = factory();
|
|
checkpointDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
/** Fence lazy creation immediately, and keep any ended pool cached. */
|
|
export async function closeCheckpointDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
closedCheckpointOwners.add(db);
|
|
await checkpointDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
|
|
}
|
|
|
|
// Managed file publication holds an agent lock while transferring bytes. A
|
|
// stalled object store must not consume request or controller-renewal capacity.
|
|
const fileStorageDbFactories = new WeakMap<object, () => Db>();
|
|
const fileStorageDbs = new WeakMap<object, Db>();
|
|
const closedFileStorageOwners = new WeakSet<object>();
|
|
export function fileStorageDb(db: Db): Db {
|
|
const existing = fileStorageDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = fileStorageDbFactories.get(db);
|
|
if (!factory) return db; // Preserve the caller's transaction/savepoint.
|
|
if (closedFileStorageOwners.has(db)) throw new Error("file_storage_pool_closed");
|
|
const created = factory();
|
|
fileStorageDbs.set(db, created);
|
|
fileStorageDbs.set(created, created);
|
|
return created;
|
|
}
|
|
export async function closeFileStorageDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
closedFileStorageOwners.add(db);
|
|
await fileStorageDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
|
|
}
|
|
|
|
export function createDb(url: string, options?: DatabaseClientOptions) {
|
|
const resolved = resolveDatabaseClientOptions(options ?? databaseClientOptionsFromEnv());
|
|
return createDbClient(url, resolved);
|
|
}
|
|
|
|
function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: object, dedicatedSet?: DedicatedConnectionSet) {
|
|
const sql = postgres(url, postgresJsOptions(resolved));
|
|
const preparedState = resolved.prepare === true
|
|
? (owner && preparedQueryStates.get(owner)) || createPreparedQueryState()
|
|
: undefined;
|
|
if (preparedState) installPreparedQueries(sql, preparedState);
|
|
const key = hostPortKeyOrNull(url);
|
|
if (key) registerClient(key, sql);
|
|
// A disconnect can lose the response after a statement has committed.
|
|
// postgres.js calls that error "write CONNECTION_CLOSED" too, so the
|
|
// message cannot establish that replay is safe. Leave retries to callers
|
|
// that know the complete operation is idempotent.
|
|
const db = installDatabaseWorkSignals(drizzleWithLazyTransactionQueries(sql, schema), owner);
|
|
installDatabaseResponseCapture(db, schema);
|
|
markOpenTransactions(db);
|
|
if (preparedState) preparedQueryStates.set(db, preparedState);
|
|
const ownerSet = owner ? dedicatedConnectionSets.get(owner) : undefined;
|
|
if (dedicatedSet) {
|
|
trackClientEnd(db, () => { dedicatedSet.ended.add(db); });
|
|
dedicatedConnectionSets.set(db, dedicatedSet);
|
|
} else if (ownerSet) {
|
|
dedicatedConnectionSets.set(db, ownerSet);
|
|
} else {
|
|
const set: DedicatedConnectionSet = dedicatedConnectionSet(() => createDbClient(url, {
|
|
...resolved, maxConnections: 1, applicationName: "paperclip-workspace-finalization-lock",
|
|
}, db, set), resolved);
|
|
dedicatedConnectionSets.set(db, set);
|
|
trackClientEnd(db, timeoutSeconds => closeDedicatedDbConnections(db, timeoutSeconds ?? 1));
|
|
}
|
|
ownedDatabaseListenerFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: 1,
|
|
}, db));
|
|
livenessDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.liveness, applicationName: "paperclip-controller-lease",
|
|
}, db));
|
|
controllerLeaseDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.legacyControllerLease, applicationName: "paperclip-legacy-controller-lease",
|
|
}, db));
|
|
controllerInstanceLeaseDbFactories.set(db, statementTimeoutMs => createDbClient(url, {
|
|
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.controllerInstanceLease, applicationName: "paperclip-controller-instance-lease",
|
|
...(statementTimeoutMs !== null ? { statementTimeoutMs } : {}),
|
|
}, db));
|
|
checkpointDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.checkpoint, applicationName: "paperclip-runner-checkpoint",
|
|
}, db));
|
|
fileStorageDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.fileStorage, applicationName: "paperclip-file-storage",
|
|
}, db));
|
|
return db;
|
|
}
|
|
|
|
export async function getPostgresDataDirectory(url: string): Promise<string | null> {
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
const rows = await sql<{ data_directory: string | null }[]>`
|
|
SELECT current_setting('data_directory', true) AS data_directory
|
|
`;
|
|
const actual = rows[0]?.data_directory;
|
|
return typeof actual === "string" && actual.length > 0 ? actual : null;
|
|
} catch {
|
|
return null;
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
async function listMigrationFiles(): Promise<string[]> {
|
|
const entries = await readdir(MIGRATIONS_FOLDER, { withFileTypes: true });
|
|
return entries
|
|
.filter((entry) => entry.isFile() && entry.name.endsWith(".sql"))
|
|
.map((entry) => entry.name)
|
|
.sort((a, b) => a.localeCompare(b));
|
|
}
|
|
|
|
type MigrationJournalFile = {
|
|
entries?: Array<{ idx?: number; tag?: string; when?: number }>;
|
|
};
|
|
|
|
type JournalMigrationEntry = {
|
|
fileName: string;
|
|
folderMillis: number;
|
|
order: number;
|
|
};
|
|
|
|
async function listJournalMigrationEntries(): Promise<JournalMigrationEntry[]> {
|
|
try {
|
|
const raw = await readFile(MIGRATIONS_JOURNAL_JSON, "utf8");
|
|
const parsed = JSON.parse(raw) as MigrationJournalFile;
|
|
if (!Array.isArray(parsed.entries)) return [];
|
|
return parsed.entries
|
|
.map((entry, entryIndex) => {
|
|
if (typeof entry?.tag !== "string") return null;
|
|
if (typeof entry?.when !== "number" || !Number.isFinite(entry.when)) return null;
|
|
const order = Number.isInteger(entry.idx) ? Number(entry.idx) : entryIndex;
|
|
return { fileName: `${entry.tag}.sql`, folderMillis: entry.when, order };
|
|
})
|
|
.filter((entry): entry is JournalMigrationEntry => entry !== null);
|
|
} catch {
|
|
return [];
|
|
}
|
|
}
|
|
|
|
async function listJournalMigrationFiles(): Promise<string[]> {
|
|
const entries = await listJournalMigrationEntries();
|
|
return entries.map((entry) => entry.fileName);
|
|
}
|
|
|
|
async function readMigrationFileContent(migrationFile: string): Promise<string> {
|
|
return readFile(new URL(`./migrations/${migrationFile}`, import.meta.url), "utf8");
|
|
}
|
|
|
|
async function orderMigrationsByJournal(migrationFiles: string[]): Promise<string[]> {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const orderByFileName = new Map(journalEntries.map((entry) => [entry.fileName, entry.order]));
|
|
return [...migrationFiles].sort((left, right) => {
|
|
const leftOrder = orderByFileName.get(left);
|
|
const rightOrder = orderByFileName.get(right);
|
|
if (leftOrder === undefined && rightOrder === undefined) return left.localeCompare(right);
|
|
if (leftOrder === undefined) return 1;
|
|
if (rightOrder === undefined) return -1;
|
|
if (leftOrder === rightOrder) return left.localeCompare(right);
|
|
return leftOrder - rightOrder;
|
|
});
|
|
}
|
|
|
|
type SqlExecutor = Pick<ReturnType<typeof postgres>, "unsafe">;
|
|
|
|
async function runInTransaction(sql: SqlExecutor, action: () => Promise<void>): Promise<void> {
|
|
await sql.unsafe("BEGIN");
|
|
try {
|
|
await action();
|
|
await sql.unsafe("COMMIT");
|
|
} catch (error) {
|
|
try {
|
|
await sql.unsafe("ROLLBACK");
|
|
} catch {
|
|
// Ignore rollback failures and surface the original error.
|
|
}
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
async function latestMigrationCreatedAt(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
): Promise<number | null> {
|
|
const rows = await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC NULLS LAST LIMIT 1`,
|
|
);
|
|
const value = Number(rows[0]?.created_at ?? Number.NaN);
|
|
return Number.isFinite(value) ? value : null;
|
|
}
|
|
|
|
function normalizeFolderMillis(value: number | null | undefined): number {
|
|
if (typeof value === "number" && Number.isFinite(value) && value >= 0) {
|
|
return Math.trunc(value);
|
|
}
|
|
return Date.now();
|
|
}
|
|
|
|
async function ensureMigrationJournalTable(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<{ migrationTableSchema: string; columnNames: Set<string> }> {
|
|
let migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
const drizzleSchema = quoteIdentifier("drizzle");
|
|
const migrationTable = quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE);
|
|
await sql.unsafe(`CREATE SCHEMA IF NOT EXISTS ${drizzleSchema}`);
|
|
await sql.unsafe(
|
|
`CREATE TABLE IF NOT EXISTS ${drizzleSchema}.${migrationTable} (id SERIAL PRIMARY KEY, hash text NOT NULL, created_at bigint)`,
|
|
);
|
|
migrationTableSchema = (await discoverMigrationTableSchema(sql)) ?? "drizzle";
|
|
}
|
|
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
return { migrationTableSchema, columnNames };
|
|
}
|
|
|
|
async function migrationHistoryEntryExists(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
columnNames: Set<string>,
|
|
migrationFile: string,
|
|
hash: string,
|
|
): Promise<boolean> {
|
|
const predicates: string[] = [];
|
|
if (columnNames.has("hash")) predicates.push(`hash = ${quoteLiteral(hash)}`);
|
|
if (columnNames.has("name")) predicates.push(`name = ${quoteLiteral(migrationFile)}`);
|
|
if (predicates.length === 0) return false;
|
|
|
|
const rows = await sql.unsafe<{ one: number }[]>(
|
|
`SELECT 1 AS one FROM ${qualifiedTable} WHERE ${predicates.join(" OR ")} LIMIT 1`,
|
|
);
|
|
return rows.length > 0;
|
|
}
|
|
|
|
async function recordMigrationHistoryEntry(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
columnNames: Set<string>,
|
|
migrationFile: string,
|
|
hash: string,
|
|
folderMillis: number,
|
|
): Promise<void> {
|
|
const insertColumns: string[] = [];
|
|
const insertValues: string[] = [];
|
|
|
|
if (columnNames.has("hash")) {
|
|
insertColumns.push(quoteIdentifier("hash"));
|
|
insertValues.push(quoteLiteral(hash));
|
|
}
|
|
if (columnNames.has("name")) {
|
|
insertColumns.push(quoteIdentifier("name"));
|
|
insertValues.push(quoteLiteral(migrationFile));
|
|
}
|
|
if (columnNames.has("created_at")) {
|
|
const latestCreatedAt = await latestMigrationCreatedAt(sql, qualifiedTable);
|
|
const createdAt = latestCreatedAt === null
|
|
? normalizeFolderMillis(folderMillis)
|
|
: Math.max(latestCreatedAt + 1, normalizeFolderMillis(folderMillis));
|
|
insertColumns.push(quoteIdentifier("created_at"));
|
|
insertValues.push(quoteLiteral(String(createdAt)));
|
|
}
|
|
|
|
if (insertColumns.length === 0) return;
|
|
|
|
await sql.unsafe(
|
|
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
|
|
);
|
|
}
|
|
|
|
// These idempotent migrations require top-level statements for batch commits
|
|
// or concurrent index builds. History is recorded only after completion.
|
|
const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
|
|
"0313_private_task_access.sql",
|
|
"0314_private_task_draft_assets.sql",
|
|
"0332_little_night_nurse.sql",
|
|
"0345_luxuriant_carmella_unuscione.sql",
|
|
"0369_late_logan.sql",
|
|
"0381_mean_scream.sql",
|
|
"0382_narrow_the_executioner.sql",
|
|
"0385_open_madame_web.sql",
|
|
"0386_steady_epoch.sql",
|
|
"0387_boring_professor_monster.sql",
|
|
"0388_ambitious_micromax.sql",
|
|
"0389_brave_hulk.sql",
|
|
"0390_salty_killraven.sql",
|
|
"0391_third_roulette.sql",
|
|
"0392_small_frightful_four.sql",
|
|
"0393_open_grandmaster.sql",
|
|
"0395_tranquil_firestar.sql",
|
|
"0396_tan_paladin.sql",
|
|
"0397_known_madrox.sql",
|
|
"0398_yielding_random.sql",
|
|
"0399_ambitious_onslaught.sql",
|
|
"0400_absent_phantom_reporter.sql",
|
|
"0401_yummy_ultimates.sql",
|
|
"0402_ancient_malice.sql",
|
|
"0403_natural_cerise.sql",
|
|
"0406_zippy_war_machine.sql",
|
|
"0407_greedy_star_brand.sql",
|
|
"0408_short_red_skull.sql",
|
|
"0409_uneven_zuras.sql",
|
|
"0410_freezing_kinsey_walden.sql",
|
|
"0411_real_scrambler.sql",
|
|
"0412_tearful_squadron_supreme.sql",
|
|
"0413_tired_kronos.sql",
|
|
"0414_closed_edwin_jarvis.sql",
|
|
"0419_slippery_impossible_man.sql",
|
|
"0432_strong_mole_man.sql",
|
|
"0435_premium_puff_adder.sql",
|
|
"0436_legal_reavers.sql",
|
|
"0447_legacy_controller_process.sql",
|
|
"0450_petite_frightful_four.sql",
|
|
"0453_curious_ben_parker.sql",
|
|
"0454_lumpy_ronan.sql",
|
|
"0455_pending_budget_cancellation_index.sql",
|
|
"0456_chat_review_lookup_indexes.sql",
|
|
"0457_native_session_run_index.sql",
|
|
"0458_chat_admission_issue_index.sql",
|
|
"0459_environment_lease_backup_scope_index.sql",
|
|
"0460_workspace_activity_context_index.sql",
|
|
"0461_created_from_legacy_origin_run_index.sql",
|
|
"0470_heartbeat_runs_queued_created_index.sql",
|
|
"0472_runner_api_capture_pending_deletes.sql",
|
|
"0476_wakeup_request_run_index.sql",
|
|
"0486_heartbeat_runs_company_finished_index.sql",
|
|
"0487_heartbeat_runs_settled_process_indexes.sql",
|
|
"0469_zippy_photon.sql",
|
|
|
|
|
|
"0384_funny_adam_warlock.sql",
|
|
"0383_tool_review_company_expiry.sql",
|
|
]);
|
|
|
|
async function applyPendingMigrationsManually(
|
|
url: string,
|
|
pendingMigrations: string[],
|
|
): Promise<void> {
|
|
if (pendingMigrations.length === 0) return;
|
|
|
|
const orderedPendingMigrations = await orderMigrationsByJournal(pendingMigrations);
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const folderMillisByFileName = new Map(
|
|
journalEntries.map((entry) => [entry.fileName, normalizeFolderMillis(entry.folderMillis)]),
|
|
);
|
|
|
|
const pool = createUtilitySql(url);
|
|
try {
|
|
const sql = await pool.reserve();
|
|
try {
|
|
// A session lock survives each batch COMMIT. Reserve the connection so
|
|
// pool rotation cannot release it while another migrator is waiting.
|
|
// A blocking lock SELECT retains its statement snapshot while waiting.
|
|
// A partial concurrent index build can wait for that snapshot, forming a
|
|
// deadlock with this lock. Each failed try ends its snapshot before sleep.
|
|
while (!(await sql<{ locked: boolean }[]>`
|
|
SELECT pg_try_advisory_lock(hashtextextended('paperclip:migrations', 0)) AS locked`)[0]?.locked) {
|
|
await new Promise((resolve) => setTimeout(resolve, 100));
|
|
}
|
|
const { migrationTableSchema, columnNames } = await ensureMigrationJournalTable(sql);
|
|
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
|
|
for (const migrationFile of orderedPendingMigrations) {
|
|
const migrationContent = await readMigrationFileContent(migrationFile);
|
|
const hash = createHash("sha256").update(migrationContent).digest("hex");
|
|
const existingEntry = await migrationHistoryEntryExists(
|
|
sql,
|
|
qualifiedTable,
|
|
columnNames,
|
|
migrationFile,
|
|
hash,
|
|
);
|
|
if (existingEntry) continue;
|
|
|
|
const applyMigration = async () => {
|
|
for (const statement of splitMigrationStatements(migrationContent)) {
|
|
if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
|
|
// A cancelled concurrent build leaves an invalid index. IF NOT
|
|
// EXISTS alone would skip it on retry and journal an incomplete index.
|
|
const index = statement.replace(/^\s*--.*$/gm, "").trim()
|
|
.match(/^CREATE (?:UNIQUE )?INDEX CONCURRENTLY IF NOT EXISTS "([A-Za-z_][A-Za-z0-9_]*)"/i);
|
|
if (index) {
|
|
const invalid = await sql<{ invalid: boolean }[]>`
|
|
SELECT NOT i.indisvalid AS invalid FROM pg_index i
|
|
JOIN pg_class c ON c.oid = i.indexrelid
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public' AND c.relname = ${index[1]}`;
|
|
if (invalid[0]?.invalid) {
|
|
await sql.unsafe(`DROP INDEX CONCURRENTLY ${quoteIdentifier("public")}.${quoteIdentifier(index[1])}`);
|
|
}
|
|
}
|
|
}
|
|
// Older dev schemas can lack a replaced constraint. Preserve the
|
|
// published SQL/hash while allowing the harmless drop to proceed.
|
|
const executable = statement.replace(
|
|
/^(\s*ALTER TABLE "[^"]+" DROP CONSTRAINT) (?!IF EXISTS\b)/i,
|
|
"$1 IF EXISTS ",
|
|
);
|
|
await sql.unsafe(executable);
|
|
}
|
|
|
|
await recordMigrationHistoryEntry(
|
|
sql,
|
|
qualifiedTable,
|
|
columnNames,
|
|
migrationFile,
|
|
hash,
|
|
folderMillisByFileName.get(migrationFile) ?? Date.now(),
|
|
);
|
|
};
|
|
if (migrationFile === PROJECT_ENVIRONMENT_OVERLAP.dropFile) {
|
|
const repairFile = PROJECT_ENVIRONMENT_OVERLAP.repairFile;
|
|
const repairContent = await readMigrationFileContent(repairFile);
|
|
const repairHash = createHash("sha256").update(repairContent).digest("hex");
|
|
const dropIndex = orderedPendingMigrations.indexOf(migrationFile);
|
|
if (hash !== PROJECT_ENVIRONMENT_OVERLAP.dropSha256
|
|
|| repairHash !== PROJECT_ENVIRONMENT_OVERLAP.repairSha256
|
|
|| orderedPendingMigrations[dropIndex + 1] !== repairFile
|
|
|| await migrationHistoryEntryExists(sql, qualifiedTable, columnNames, repairFile, repairHash)) {
|
|
throw new Error("Project environment migration pair does not match the qualified overlap repair");
|
|
}
|
|
await runInTransaction(sql, async () => {
|
|
await sql.unsafe('LOCK TABLE "public"."projects" IN ACCESS EXCLUSIVE MODE');
|
|
await sql.unsafe(`CREATE TEMP TABLE paperclip_project_environment_overlap ON COMMIT DROP AS
|
|
SELECT id, environment_id FROM "public"."projects" WHERE environment_id IS NOT NULL`);
|
|
await applyMigration();
|
|
for (const statement of splitMigrationStatements(repairContent)) await sql.unsafe(statement);
|
|
await sql.unsafe(`UPDATE "public"."projects" AS project SET environment_id = legacy.environment_id
|
|
FROM pg_temp.paperclip_project_environment_overlap AS legacy WHERE project.id = legacy.id`);
|
|
await recordMigrationHistoryEntry(sql, qualifiedTable, columnNames, repairFile, repairHash,
|
|
folderMillisByFileName.get(repairFile) ?? Date.now());
|
|
});
|
|
} else if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
|
|
await applyMigration();
|
|
} else {
|
|
await runInTransaction(sql, applyMigration);
|
|
}
|
|
}
|
|
} finally {
|
|
try {
|
|
await sql`SELECT pg_advisory_unlock(hashtextextended('paperclip:migrations', 0))`;
|
|
} catch {
|
|
// Preserve the migration error if the session died; PostgreSQL releases
|
|
// its lock on disconnect, and pool.end() closes any remaining connection.
|
|
}
|
|
sql.release();
|
|
}
|
|
} finally {
|
|
await pool.end();
|
|
}
|
|
}
|
|
|
|
async function mapHashesToMigrationFiles(migrationFiles: string[]): Promise<Map<string, string>> {
|
|
const mapped = new Map<string, string>();
|
|
|
|
await Promise.all(
|
|
migrationFiles.map(async (migrationFile) => {
|
|
const content = await readMigrationFileContent(migrationFile);
|
|
const hash = createHash("sha256").update(content).digest("hex");
|
|
mapped.set(hash, migrationFile);
|
|
if (migrationFile === "0355_agent_tool_definition_hooks.sql") {
|
|
mapped.set("7ac76adac0220da522c78e51da20490aeb0f90512b4370a05af9f8dd0f75eda6", migrationFile);
|
|
}
|
|
}),
|
|
);
|
|
|
|
return mapped;
|
|
}
|
|
|
|
async function getMigrationTableColumnNames(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationTableSchema: string,
|
|
): Promise<Set<string>> {
|
|
const columns = await sql.unsafe<{ column_name: string }[]>(
|
|
`
|
|
SELECT column_name
|
|
FROM information_schema.columns
|
|
WHERE table_schema = ${quoteLiteral(migrationTableSchema)}
|
|
AND table_name = ${quoteLiteral(DRIZZLE_MIGRATIONS_TABLE)}
|
|
`,
|
|
);
|
|
return new Set(columns.map((column) => column.column_name));
|
|
}
|
|
|
|
async function tableExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM information_schema.tables
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function columnExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
columnName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM information_schema.columns
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
AND column_name = ${columnName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function columnHasDataType(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
columnName: string,
|
|
dataType: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ dataType: string; udtName: string }[]>`
|
|
SELECT data_type AS "dataType", udt_name AS "udtName"
|
|
FROM information_schema.columns
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
AND column_name = ${columnName}
|
|
`;
|
|
const expected = dataType.toLowerCase();
|
|
return rows.some((row) => (
|
|
row.dataType.toLowerCase() === expected || row.udtName.toLowerCase() === expected
|
|
));
|
|
}
|
|
|
|
async function indexExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
indexName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_class c
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public'
|
|
AND c.relkind = 'i'
|
|
AND c.relname = ${indexName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function constraintExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
constraintName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_constraint c
|
|
JOIN pg_namespace n ON n.oid = c.connamespace
|
|
WHERE n.nspname = 'public'
|
|
AND c.conname = ${constraintName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function functionExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
functionName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_proc p
|
|
JOIN pg_namespace n ON n.oid = p.pronamespace
|
|
WHERE n.nspname = 'public'
|
|
AND p.proname = ${functionName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function triggerExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
triggerName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_trigger t
|
|
JOIN pg_class c ON c.oid = t.tgrelid
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public'
|
|
AND t.tgname = ${triggerName}
|
|
AND NOT t.tgisinternal
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function heartbeatEventSequencesAreUnique(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ unique: boolean }[]>`
|
|
SELECT NOT EXISTS (
|
|
SELECT 1
|
|
FROM heartbeat_run_events
|
|
GROUP BY run_id, seq
|
|
HAVING count(*) > 1
|
|
) AS unique
|
|
`;
|
|
return rows[0]?.unique ?? false;
|
|
}
|
|
|
|
async function heartbeatNextEventSequencesAreCurrent(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ current: boolean }[]>`
|
|
SELECT NOT EXISTS (
|
|
SELECT 1
|
|
FROM heartbeat_runs run
|
|
WHERE run.next_event_seq IS DISTINCT FROM COALESCE((
|
|
SELECT max(event.seq) + 1
|
|
FROM heartbeat_run_events event
|
|
WHERE event.run_id = run.id
|
|
), 1)
|
|
) AS current
|
|
`;
|
|
return rows[0]?.current ?? false;
|
|
}
|
|
|
|
async function migrationStatementAlreadyApplied(
|
|
sql: ReturnType<typeof postgres>,
|
|
statement: string,
|
|
): Promise<boolean> {
|
|
const normalized = statement
|
|
.replace(/^\s*--.*$/gm, "")
|
|
.replace(/\s+/g, " ")
|
|
.trim();
|
|
|
|
const createTableMatch = normalized.match(/^CREATE TABLE(?: IF NOT EXISTS)? "([^"]+)"/i);
|
|
if (createTableMatch) {
|
|
return tableExists(sql, createTableMatch[1]);
|
|
}
|
|
|
|
const addColumnMatch = normalized.match(
|
|
/^ALTER TABLE "([^"]+)" ADD COLUMN(?: IF NOT EXISTS)? "([^"]+)"/i,
|
|
);
|
|
if (addColumnMatch) {
|
|
return columnExists(sql, addColumnMatch[1], addColumnMatch[2]);
|
|
}
|
|
|
|
const alterColumnTypeMatch = normalized.match(
|
|
/^ALTER TABLE "([^"]+)" ALTER COLUMN "([^"]+)" SET DATA TYPE ([A-Za-z0-9_]+)/i,
|
|
);
|
|
if (alterColumnTypeMatch) {
|
|
return columnHasDataType(
|
|
sql,
|
|
alterColumnTypeMatch[1],
|
|
alterColumnTypeMatch[2],
|
|
alterColumnTypeMatch[3],
|
|
);
|
|
}
|
|
|
|
const createIndexMatch = normalized.match(/^CREATE (?:UNIQUE )?INDEX(?: IF NOT EXISTS)? "([^"]+)"/i);
|
|
if (createIndexMatch) {
|
|
return indexExists(sql, createIndexMatch[1]);
|
|
}
|
|
|
|
const addConstraintMatch = normalized.match(/^ALTER TABLE "([^"]+)" ADD CONSTRAINT "([^"]+)"/i);
|
|
if (addConstraintMatch) {
|
|
return constraintExists(sql, addConstraintMatch[2]);
|
|
}
|
|
|
|
const createFunctionMatch = normalized.match(
|
|
/^CREATE OR REPLACE FUNCTION "?([A-Za-z_][A-Za-z0-9_]*)"?\s*\(/i,
|
|
);
|
|
if (createFunctionMatch) {
|
|
return functionExists(sql, createFunctionMatch[1]);
|
|
}
|
|
|
|
const createTriggerMatch = normalized.match(
|
|
/^CREATE TRIGGER "?([A-Za-z_][A-Za-z0-9_]*)"?/i,
|
|
);
|
|
if (createTriggerMatch) {
|
|
return triggerExists(sql, createTriggerMatch[1]);
|
|
}
|
|
|
|
// These native-runner repairs have persistent postconditions. Verify them
|
|
// instead of replaying them when a restored database is missing only the
|
|
// migration-history row.
|
|
if (
|
|
normalized.startsWith("WITH ranked AS (")
|
|
&& normalized.includes('UPDATE "heartbeat_run_events" AS event')
|
|
) {
|
|
return heartbeatEventSequencesAreUnique(sql);
|
|
}
|
|
if (
|
|
normalized.startsWith('UPDATE "heartbeat_runs" AS run')
|
|
&& normalized.includes('SET "next_event_seq" = COALESCE')
|
|
) {
|
|
return heartbeatNextEventSequencesAreCurrent(sql);
|
|
}
|
|
|
|
// If we cannot reason about a statement safely, require manual migration.
|
|
return false;
|
|
}
|
|
|
|
async function migrationContentAlreadyApplied(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationContent: string,
|
|
): Promise<boolean> {
|
|
const statements = splitMigrationStatements(migrationContent);
|
|
if (statements.length === 0) return false;
|
|
|
|
for (const statement of statements) {
|
|
const applied = await migrationStatementAlreadyApplied(sql, statement);
|
|
if (!applied) return false;
|
|
}
|
|
|
|
return true;
|
|
}
|
|
|
|
async function loadAppliedMigrations(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationTableSchema: string,
|
|
availableMigrations: string[],
|
|
): Promise<string[]> {
|
|
const quotedSchema = quoteIdentifier(migrationTableSchema);
|
|
const qualifiedTable = `${quotedSchema}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
|
|
if (columnNames.has("name")) {
|
|
const rows = await sql.unsafe<{ name: string }[]>(`SELECT name FROM ${qualifiedTable} ORDER BY id`);
|
|
return rows.map((row) => row.name).filter((name): name is string => Boolean(name));
|
|
}
|
|
|
|
if (columnNames.has("hash")) {
|
|
const rows = await sql.unsafe<{ hash: string }[]>(`SELECT hash FROM ${qualifiedTable} ORDER BY id`);
|
|
const hashesToMigrationFiles = await mapHashesToMigrationFiles(availableMigrations);
|
|
const appliedFromHashes = rows
|
|
.map((row) => hashesToMigrationFiles.get(row.hash))
|
|
.filter((name): name is string => Boolean(name));
|
|
|
|
if (appliedFromHashes.length > 0) {
|
|
// Best-effort: when all hashes resolve, this is authoritative.
|
|
if (appliedFromHashes.length === rows.length) return appliedFromHashes;
|
|
|
|
// Partial hash resolution can happen when files have changed; return what we can trust.
|
|
return appliedFromHashes;
|
|
}
|
|
|
|
// Fallback only when hashes are unavailable/unresolved.
|
|
if (columnNames.has("created_at")) {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
if (journalEntries.length > 0) {
|
|
const lastDbRows = await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC LIMIT 1`,
|
|
);
|
|
const lastCreatedAt = Number(lastDbRows[0]?.created_at ?? -1);
|
|
if (Number.isFinite(lastCreatedAt) && lastCreatedAt >= 0) {
|
|
return journalEntries
|
|
.filter((entry) => availableMigrations.includes(entry.fileName))
|
|
.filter((entry) => entry.folderMillis <= lastCreatedAt)
|
|
.map((entry) => entry.fileName)
|
|
.slice(0, rows.length);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
const rows = await sql.unsafe<{ id: number }[]>(`SELECT id FROM ${qualifiedTable} ORDER BY id`);
|
|
const journalMigrationFiles = await listJournalMigrationFiles();
|
|
const appliedFromIds = rows
|
|
.map((row) => journalMigrationFiles[row.id - 1])
|
|
.filter((name): name is string => Boolean(name));
|
|
if (appliedFromIds.length > 0) return appliedFromIds;
|
|
|
|
return availableMigrations.slice(0, Math.max(0, rows.length));
|
|
}
|
|
|
|
export type MigrationHistoryReconcileResult = {
|
|
repairedMigrations: string[];
|
|
remainingMigrations: string[];
|
|
};
|
|
|
|
export async function reconcilePendingMigrationHistory(
|
|
url: string,
|
|
): Promise<MigrationHistoryReconcileResult> {
|
|
const state = await inspectMigrations(url);
|
|
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
|
|
return { repairedMigrations: [], remainingMigrations: [] };
|
|
}
|
|
|
|
const sql = createUtilitySql(url);
|
|
const repairedMigrations: string[] = [];
|
|
|
|
try {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const folderMillisByFile = new Map(journalEntries.map((entry) => [entry.fileName, entry.folderMillis]));
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
return { repairedMigrations, remainingMigrations: state.pendingMigrations };
|
|
}
|
|
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
|
|
for (const migrationFile of state.pendingMigrations) {
|
|
const migrationContent = await readMigrationFileContent(migrationFile);
|
|
const alreadyApplied = await migrationContentAlreadyApplied(sql, migrationContent);
|
|
if (!alreadyApplied) break;
|
|
|
|
const hash = createHash("sha256").update(migrationContent).digest("hex");
|
|
const folderMillis = folderMillisByFile.get(migrationFile) ?? Date.now();
|
|
const existingByHash = columnNames.has("hash")
|
|
? await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} WHERE hash = ${quoteLiteral(hash)} ORDER BY created_at DESC LIMIT 1`,
|
|
)
|
|
: [];
|
|
const existingByName = columnNames.has("name")
|
|
? await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} WHERE name = ${quoteLiteral(migrationFile)} ORDER BY created_at DESC LIMIT 1`,
|
|
)
|
|
: [];
|
|
if (existingByHash.length > 0 || existingByName.length > 0) {
|
|
if (columnNames.has("created_at")) {
|
|
const existingHashCreatedAt = Number(existingByHash[0]?.created_at ?? -1);
|
|
if (existingByHash.length > 0 && Number.isFinite(existingHashCreatedAt) && existingHashCreatedAt < folderMillis) {
|
|
await sql.unsafe(
|
|
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE hash = ${quoteLiteral(hash)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
|
|
);
|
|
}
|
|
|
|
const existingNameCreatedAt = Number(existingByName[0]?.created_at ?? -1);
|
|
if (existingByName.length > 0 && Number.isFinite(existingNameCreatedAt) && existingNameCreatedAt < folderMillis) {
|
|
await sql.unsafe(
|
|
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE name = ${quoteLiteral(migrationFile)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
repairedMigrations.push(migrationFile);
|
|
continue;
|
|
}
|
|
|
|
const insertColumns: string[] = [];
|
|
const insertValues: string[] = [];
|
|
|
|
if (columnNames.has("hash")) {
|
|
insertColumns.push(quoteIdentifier("hash"));
|
|
insertValues.push(quoteLiteral(hash));
|
|
}
|
|
if (columnNames.has("name")) {
|
|
insertColumns.push(quoteIdentifier("name"));
|
|
insertValues.push(quoteLiteral(migrationFile));
|
|
}
|
|
if (columnNames.has("created_at")) {
|
|
insertColumns.push(quoteIdentifier("created_at"));
|
|
insertValues.push(quoteLiteral(String(folderMillis)));
|
|
}
|
|
|
|
if (insertColumns.length === 0) break;
|
|
|
|
await sql.unsafe(
|
|
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
|
|
);
|
|
repairedMigrations.push(migrationFile);
|
|
}
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
|
|
const refreshed = await inspectMigrations(url);
|
|
return {
|
|
repairedMigrations,
|
|
remainingMigrations:
|
|
refreshed.status === "needsMigrations" ? refreshed.pendingMigrations : [],
|
|
};
|
|
}
|
|
|
|
async function discoverMigrationTableSchema(sql: ReturnType<typeof postgres>): Promise<string | null> {
|
|
const rows = await sql<{ schemaName: string }[]>`
|
|
SELECT n.nspname AS "schemaName"
|
|
FROM pg_class c
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE c.relname = ${DRIZZLE_MIGRATIONS_TABLE} AND c.relkind = 'r'
|
|
`;
|
|
|
|
if (rows.length === 0) return null;
|
|
|
|
const drizzleSchema = rows.find(({ schemaName }) => schemaName === "drizzle");
|
|
if (drizzleSchema) return drizzleSchema.schemaName;
|
|
|
|
const publicSchema = rows.find(({ schemaName }) => schemaName === "public");
|
|
if (publicSchema) return publicSchema.schemaName;
|
|
|
|
return rows[0]?.schemaName ?? null;
|
|
}
|
|
|
|
export async function inspectMigrations(url: string): Promise<MigrationState> {
|
|
const sql = createUtilitySql(url);
|
|
|
|
try {
|
|
const availableMigrations = await listMigrationFiles();
|
|
const tableCountResult = await sql<{ count: number }[]>`
|
|
select count(*)::int as count
|
|
from information_schema.tables
|
|
where table_schema = 'public'
|
|
and table_type = 'BASE TABLE'
|
|
`;
|
|
const tableCount = tableCountResult[0]?.count ?? 0;
|
|
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
if (tableCount > 0) {
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations: [],
|
|
pendingMigrations: availableMigrations,
|
|
journalEntryCount: 0,
|
|
reason: "no-migration-journal-non-empty-db",
|
|
};
|
|
}
|
|
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations: [],
|
|
pendingMigrations: availableMigrations,
|
|
journalEntryCount: 0,
|
|
reason: "no-migration-journal-empty-db",
|
|
};
|
|
}
|
|
|
|
const qualifiedMigrationTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
const journalCountRows = await sql.unsafe<{ count: number }[]>(
|
|
`SELECT count(*)::int AS count FROM ${qualifiedMigrationTable}`,
|
|
);
|
|
const journalEntryCount = journalCountRows[0]?.count ?? 0;
|
|
const appliedMigrations = await loadAppliedMigrations(sql, migrationTableSchema, availableMigrations);
|
|
const pendingMigrations = availableMigrations.filter((name) => !appliedMigrations.includes(name));
|
|
if (pendingMigrations.length === 0) {
|
|
return {
|
|
status: "upToDate",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations,
|
|
journalEntryCount,
|
|
};
|
|
}
|
|
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations,
|
|
pendingMigrations,
|
|
journalEntryCount,
|
|
reason: "pending-migrations",
|
|
};
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function applyPendingMigrations(url: string): Promise<void> {
|
|
const initialState = await inspectMigrations(url);
|
|
if (initialState.status === "upToDate") return;
|
|
|
|
if (initialState.reason === "no-migration-journal-empty-db") {
|
|
// Use the same per-file executor for bootstrap and upgrades: Drizzle wraps
|
|
// the whole journal in one transaction and cannot run committing backfills.
|
|
await applyPendingMigrationsManually(url, initialState.pendingMigrations);
|
|
const bootstrappedState = await inspectMigrations(url);
|
|
if (bootstrappedState.status === "upToDate") return;
|
|
throw new Error(
|
|
`Failed to bootstrap migrations: ${bootstrappedState.pendingMigrations.join(", ")}`,
|
|
);
|
|
}
|
|
|
|
if (initialState.reason === "no-migration-journal-non-empty-db") {
|
|
throw new Error(
|
|
"Database has tables but no migration journal; automatic migration is unsafe. Initialize migration history manually.",
|
|
);
|
|
}
|
|
|
|
let state = await inspectMigrations(url);
|
|
if (state.status === "upToDate") return;
|
|
|
|
const repair = await reconcilePendingMigrationHistory(url);
|
|
if (repair.repairedMigrations.length > 0) {
|
|
state = await inspectMigrations(url);
|
|
if (state.status === "upToDate") return;
|
|
}
|
|
|
|
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
|
|
throw new Error("Migrations are still pending after migration-history reconciliation; run inspectMigrations for details.");
|
|
}
|
|
|
|
await applyPendingMigrationsManually(url, state.pendingMigrations);
|
|
|
|
const finalState = await inspectMigrations(url);
|
|
if (finalState.status !== "upToDate") {
|
|
throw new Error(
|
|
`Failed to apply pending migrations: ${finalState.pendingMigrations.join(", ")}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
export type MigrationBootstrapResult =
|
|
| { migrated: true; reason: "migrated-empty-db"; tableCount: 0 }
|
|
| { migrated: false; reason: "already-migrated"; tableCount: number }
|
|
| { migrated: false; reason: "not-empty-no-migration-journal"; tableCount: number };
|
|
|
|
export async function migratePostgresIfEmpty(url: string): Promise<MigrationBootstrapResult> {
|
|
const sql = createUtilitySql(url);
|
|
|
|
try {
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
|
|
const tableCountResult = await sql<{ count: number }[]>`
|
|
select count(*)::int as count
|
|
from information_schema.tables
|
|
where table_schema = 'public'
|
|
and table_type = 'BASE TABLE'
|
|
`;
|
|
|
|
const tableCount = tableCountResult[0]?.count ?? 0;
|
|
|
|
if (migrationTableSchema) {
|
|
return { migrated: false, reason: "already-migrated", tableCount };
|
|
}
|
|
|
|
if (tableCount > 0) {
|
|
return { migrated: false, reason: "not-empty-no-migration-journal", tableCount };
|
|
}
|
|
|
|
await applyPendingMigrationsManually(url, await listMigrationFiles());
|
|
|
|
return { migrated: true, reason: "migrated-empty-db", tableCount: 0 };
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function ensurePostgresDatabase(
|
|
url: string,
|
|
databaseName: string,
|
|
): Promise<"created" | "exists"> {
|
|
if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(databaseName)) {
|
|
throw new Error(`Unsafe database name: ${databaseName}`);
|
|
}
|
|
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
const existing = await sql<{ one: number }[]>`
|
|
select 1 as one from pg_database where datname = ${databaseName} limit 1
|
|
`;
|
|
if (existing.length > 0) return "exists";
|
|
|
|
await sql.unsafe(`create database "${databaseName}" encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
|
|
return "created";
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function resetPostgresDatabase(
|
|
url: string,
|
|
databaseName: string,
|
|
): Promise<"reset"> {
|
|
const quotedDatabaseName = quoteIdentifier(databaseName);
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
await sql`
|
|
select pg_terminate_backend(pid)
|
|
from pg_stat_activity
|
|
where datname = ${databaseName}
|
|
and pid <> pg_backend_pid()
|
|
`;
|
|
await sql.unsafe(`drop database if exists ${quotedDatabaseName}`);
|
|
await sql.unsafe(`create database ${quotedDatabaseName} encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
|
|
return "reset";
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export type Db = ReturnType<typeof createDb>;
|