mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server runs every database statement from one Node main thread.
With `prepare: true`, postgres.js sends a statement in one round trip
instead of two, and the CPU cost of each statement is about half
(benchmark from #21707).
> - `DATABASE_PREPARED_STATEMENTS=true` turns on preparation for ORM
queries (#17673). Hosted instances connect through a transaction-mode
pooler (PgBouncer 1.22 or later, Neon `-pooler` hosts), which supports
protocol-level prepared statements.
> - I tested the flag locally through PgBouncer 1.26.1 in transaction
mode. Evictions, server connection churn, `RECONNECT`, transactions,
pipelining, `SET LOCAL`, advisory transaction locks and savepoints all
work. One failure is not acceptable: after a migration changes a column
type, every prepared statement that reads that column fails, for every
client and for every new process.
> - PgBouncer shares one server-side prepared statement by SQL text.
PostgreSQL never repairs a named statement whose result type changed, so
it fails with `cached plan must not change result type` until PgBouncer
drops it. The postgres.js retry prepares the same text again, and
PgBouncer gives it the same broken statement. postgres.js also caches
the failed statement without columns, so the connection throws
`TypeError: Cannot read properties of undefined (reading 'length')` for
that text even after the schema is correct again.
> - This pull request makes the client recover. When a prepared
statement fails with a prepared-statement error, the client stops
preparing that text, retries once without preparation where that is
safe, and counts it. It also caps the number of prepared texts.
> - The benefit is that the flag can be turned on behind a pooler
without a deploy migration that breaks reads. Behavior with the flag
unset or `false` does not change.
## Linked Issues or Issue Description
Refs #17673 (the opt-in), #10795 (the flag), #21707 (the CPU benchmark).
**What happened?**
With `DATABASE_PREPARED_STATEMENTS=true` behind PgBouncer 1.26.1 in
transaction mode (`max_prepared_statements=5`, local PostgreSQL 17), a
change of a column type made every later execution of an affected
statement fail. A new client process failed in the same way. Measured
with the local probe, 20 executions after the change and 10 from a new
process:
| Change | Before: failures | After: failures |
| --- | --- | --- |
| `int` to `bigint`, ORM read outside a transaction | 20/20, new process
10/10 (0A000) | 0/20, 0/10 |
| `varchar` length change, ORM read | 20/20, 10/10 | 0/20, 0/10 |
| Add column, `select *` | 20/20, 10/10 | 0/20, 0/10 |
| `int` to `bigint`, first statement of a transaction | 20/20, 10/10 |
0/20, 0/10 |
| `int` to `bigint`, second statement of a transaction | 20/20, 10/10 |
1/20, 1/10 |
| Table recreated with its original shape, same client | 10/10
(TypeError in `DataRow`) | 0/10 |
| Add column, explicit-column ORM read | 0/20 | 0/20 |
**Expected behavior**
A statement whose pooled prepared statement is stale or lost runs
anyway, and the failure does not reach the caller.
**Steps to reproduce**
1. Run PostgreSQL behind PgBouncer 1.22 or later with `pool_mode =
transaction` and `max_prepared_statements` above 0.
2. Start Paperclip with `DATABASE_PREPARED_STATEMENTS=true` and run a
read of a table a few times.
3. Change the type of a column that the read returns, for example `alter
table ... alter column n type bigint`.
4. Run the read again from the same process or from a new process: it
fails with `cached plan must not change result type`.
## What Changed
- `packages/db/src/prepared-queries.ts`:
- The prepare decision for each `unsafe` query is made when the driver
builds the query, not when the caller creates it, so a query that waits
on a busy connection sees a text that was demoted after it was queued.
- A failure with 26000 (`prepared statement does not exist`), 42P05
(duplicate statement name), or 0A000 `cached plan must not change result
type` demotes the text: the client does not prepare it again in that
process.
- A pooled statement that fails in that way is retried once with
`prepare: false`. Statements in a reserved connection, cursors,
`forEach`, `describe` and queries with `onexecute` are not retried.
- When the failed statement is the first statement that the transaction
ran (only `SET TRANSACTION` before it), the client restarts the
transaction on the same connection: `ROLLBACK`, `BEGIN` with the same
options, the same `SET TRANSACTION` statements, then the statement
unprepared. The callback runs once and receives the result. Tagged
statements and statements in savepoints count toward the execution
order. The client does not restart when another statement of the
transaction is queued, and it never restarts inside a savepoint.
- At most 1,000 distinct texts are prepared for each client and its
private pools. Texts longer than 8 KB are not prepared. Explicit
per-query `prepare` options, simple queries and the flag-off paths are
unchanged.
- `packages/db/src/client.ts`, `packages/db/src/index.ts`: a client and
its dedicated, liveness, checkpoint, file-storage and listener pools
share one state. `preparedStatementStats(db)` returns the prepared,
unprepared, demoted and retried counts.
- `packages/db/src/prepared-queries-postgres.test.ts`: nine tests
against embedded PostgreSQL: restart after a stale plan and after a lost
statement (the callback runs once and its later work commits),
`repeatable read read only` kept after a restart for driver and ORM
transactions, no restart after a later statement, after a tagged first
statement, or for an out-of-order await, a savepoint whose first
statement fails while the outer transaction continues once, the pooled
retry with `.values()`, and the cap.
- `doc/DATABASE.md`, `docs/deploy/database.md`: the cap and the
recovery.
## Verification
- Red/green, `pnpm --filter @paperclipai/db exec vitest run
src/prepared-queries-postgres.test.ts`. With the old
`prepared-queries.ts` (stub stats exports only), the five new tests
fail: two transactions fail with the stale-plan and the
missing-statement errors, the ORM read fails with 42P05, the invalidated
text is prepared again, and 5 texts are prepared where the cap is 2.
With this change, 23/23 pass.
- `pnpm --filter @paperclipai/db exec vitest run` on
`prepared-queries-postgres`, `client-options`, `client`,
`dedicated-connection-pool`, `dedicated-connection-admission`,
`response-capture-postgres`, `lazy-transaction-queries`,
`client-disconnect-replay`: 91/91 pass on the final revision. `tsc
--noEmit -p packages/db` passes.
- 31 server test files with `DATABASE_PREPARED_STATEMENTS=true`
(services, routes, inbox, recovery, heartbeat events, agent auth,
prepared-statement helpers): 829/830 pass. The one failure
(`execution-workspaces-service`, a git-scan timing test) passes alone
with the flag on and off. The machine load average was 9 to 20.
- Local PgBouncer probe (not in CI, because CI has no PgBouncer):
PgBouncer 1.26.1, PostgreSQL 17, postgres.js 3.4.9 (patched), Node 26.4.
The results in the table above, plus 0 failures for 4,800 statements
over 60 distinct texts with `max_prepared_statements=5`, 0 failures with
`pg_terminate_backend` and `RECONNECT` every 25 statements, and 0
failures for transactions with `SET LOCAL`, pipelines of 20 statements,
`pg_advisory_xact_lock` and savepoint rollback.
- Through PgBouncer 1.26.1 (transaction mode,
`max_prepared_statements=5`, 4 server connections) and PostgreSQL 17, at
head 84732c8f67, compared with the master `prepared-queries.ts`. Each
row ran 10 times on its own client, warmed before the migration. "Type"
is `int` to `bigint` under an ORM read; "star" is an added column under
`select *`. Failures before → after:
| Row | Type change | Added column, `select *` |
| --- | --- | --- |
| (a) outside a transaction, same process | 10 → 0 | 10 → 0 |
| (b) first statement of an ORM transaction (callback runs once) | 10 →
0 | 10 → 0 |
| (b) first statement, `begin isolation level repeatable read read only`
| 0 → 0 (the statement was evicted and prepared again) | 10 → 0 |
| (b) first statement, ORM `repeatable read`, `read only` (`SET
TRANSACTION` repeated) | 10 → 0 | 10 → 0 |
| (c) later statement of a transaction, same process | 10 → 1 | 10 → 1 |
| (d) new process after the migration: (a) / (b) / (b) driver / (b) ORM
/ (c) | 10/10/0/0/0 → 0/0/0/0/1 | 10/10/10/10/10 → 0/0/0/0/1 |
| (e) same client after the table is restored: (a) / (b) / (b) driver /
(b) ORM / (c) | 10/10/0/10/10 → 0 | 10 TypeError each → 0 |
Every restarted transaction checked `transaction_isolation = repeatable
read` and `transaction_read_only = on` after the restart. The PostgreSQL
log shows each restart as `ROLLBACK`, `BEGIN` with the same options,
then the statement as an unnamed statement. With 12 clients restarting
at once over 2 server connections and 6 clients of background load,
12/12 transactions succeeded. In the 15 restarts I paired by
`application_name`, the second `BEGIN` ran on the same server connection
as the `ROLLBACK`, because both are sent together. A different server
connection is also safe: only `SET TRANSACTION` ran before the restart
point, and the client repeats it after the new `BEGIN`.
- Silent rollback check through PgBouncer with
`max_prepared_statements=2`: 3 clients ran 600 transactions, each with
an insert and 6 statements over 40 distinct texts, and more distinct
statements between transactions. PgBouncer parsed 6,409 statements on
the server for 1,089 client parses, so it evicted and prepared again the
shared `commit` (`PGBOUNCER_15`) many times. All 600 `COMMIT`s
succeeded, 600/600 rows were present, and no 26000 error was logged. The
silent rollback needs a `DEALLOCATE` on the session, which PgBouncer
does not allow (it closes the client) and Paperclip does not send.
## Risks
- A transaction restart happens below the callback. Code in the callback
before its first statement runs once. The restarted transaction has a
new transaction id.
- A postgres.js behavior that this PR does not change: with driver-level
preparation (the flag `true`, or unset for tagged SQL), the tagged
`commit` of `sql.begin` is a prepared statement. If something
deallocates it on the session, the driver retries `COMMIT` in the
aborted transaction, which PostgreSQL turns into a rollback without an
error. Through PgBouncer this did not happen in 600 transactions with
the `commit` statement evicted repeatedly (see Verification). It needs a
manual `DEALLOCATE` on a direct session.
- A later statement in a transaction still fails once for each affected
text in each process after a migration that changes a column type. The
text then runs unprepared.
- A demoted text stays unprepared until the process restarts. The cap
admits texts in first-use order.
- A client that sends `DEALLOCATE ALL` through PgBouncer is disconnected
by PgBouncer. Paperclip does not send it.
- With the flag unset or `false`, the code path is the same as before.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, local PostgreSQL and PgBouncer in a throwaway
container, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1637 lines
63 KiB
TypeScript
1637 lines
63 KiB
TypeScript
import { AsyncLocalStorage } from "node:async_hooks";
|
|
import { createHash } from "node:crypto";
|
|
import { readFile, readdir } from "node:fs/promises";
|
|
import { fileURLToPath } from "node:url";
|
|
import postgres from "postgres";
|
|
import { installDatabaseWorkSignals } from "./work-signals.js";
|
|
import { installDatabaseResponseCapture } from "./response-capture.js";
|
|
import { createPreparedQueryState, installPreparedQueries, preparedQueryStats, type PreparedQueryState, type PreparedQueryStats } from "./prepared-queries.js";
|
|
import { drizzleWithLazyTransactionQueries } from "./lazy-transaction-queries.js";
|
|
import * as schema from "./schema/index.js";
|
|
|
|
const MIGRATIONS_FOLDER = fileURLToPath(new URL("./migrations", import.meta.url));
|
|
const DRIZZLE_MIGRATIONS_TABLE = "__drizzle_migrations";
|
|
const MIGRATIONS_JOURNAL_JSON = fileURLToPath(new URL("./migrations/meta/_journal.json", import.meta.url));
|
|
const PROJECT_ENVIRONMENT_OVERLAP = {
|
|
dropFile: "0466_drop_project_environment.sql",
|
|
dropSha256: "07ca1c666c6471fdb9ce7a0a5fb31bf1a074138466fa8daae50b53199b8e0d7c",
|
|
repairFile: "0467_fair_human_torch.sql",
|
|
repairSha256: "50ce0eb2567570b098bea1de92fe6a220d139e4ba83438195fbbad1be51e31a1",
|
|
} as const;
|
|
|
|
function createUtilitySql(url: string) {
|
|
return postgres(url, { max: 1, onnotice: () => {} });
|
|
}
|
|
|
|
type RegisteredPostgresClient = ReturnType<typeof postgres>;
|
|
|
|
/**
|
|
* Derives a registry key from a connection URL's host and port only. We must
|
|
* not retain or log the full URL, because it carries credentials.
|
|
*/
|
|
function hostPortKey(url: string): string {
|
|
const parsed = new URL(url);
|
|
return `${parsed.hostname}:${parsed.port || "5432"}`;
|
|
}
|
|
|
|
/**
|
|
* Same as `hostPortKey`, but returns `null` instead of throwing when the URL
|
|
* does not parse. `postgres(url)` tolerates a value `new URL()` rejects (an
|
|
* empty string falls back to the `PG*` environment variables), so `createDb`
|
|
* must tolerate it too: skip the registry entry and let the driver decide
|
|
* the outcome, instead of throwing an error the driver itself would not.
|
|
*/
|
|
function hostPortKeyOrNull(url: string): string | null {
|
|
try {
|
|
return hostPortKey(url);
|
|
} catch (error) {
|
|
if (error instanceof TypeError && (error as NodeJS.ErrnoException).code === "ERR_INVALID_URL") return null;
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
// Tracks every client `createDb` hands out, keyed by host and port, so a test
|
|
// fixture can end them before it stops the Postgres cluster they point at. A
|
|
// `WeakRef` plus `FinalizationRegistry` means a long-lived process (a real
|
|
// server) retains nothing extra: an unreferenced client is pruned on its own.
|
|
const clientsByHostPort = new Map<string, Set<WeakRef<RegisteredPostgresClient>>>();
|
|
const clientFinalizer = new FinalizationRegistry<{ hostPortKey: string; ref: WeakRef<RegisteredPostgresClient> }>(
|
|
({ hostPortKey, ref }) => {
|
|
const refs = clientsByHostPort.get(hostPortKey);
|
|
if (!refs) return;
|
|
refs.delete(ref);
|
|
if (refs.size === 0) clientsByHostPort.delete(hostPortKey);
|
|
},
|
|
);
|
|
|
|
function registerClient(key: string, client: RegisteredPostgresClient): void {
|
|
const ref = new WeakRef(client);
|
|
let refs = clientsByHostPort.get(key);
|
|
if (!refs) {
|
|
refs = new Set();
|
|
clientsByHostPort.set(key, refs);
|
|
}
|
|
refs.add(ref);
|
|
clientFinalizer.register(client, { hostPortKey: key, ref }, ref);
|
|
}
|
|
|
|
/**
|
|
* Ends every live client `createDb` handed out for the given URL's host and
|
|
* port, then forgets them. Call this before stopping a Postgres cluster: a
|
|
* client that outlives the cluster it points at can crash the process (a
|
|
* reserved connection's deferred write firing after the socket is gone).
|
|
* Swallows individual `end()` errors so one bad client cannot block the rest.
|
|
* An owned database on a shared test host must select database scope instead;
|
|
* its teardown cannot close another fixture's clients on that host.
|
|
*/
|
|
export async function closeRegisteredClients(url: string, options?: { scope: "database" }): Promise<void> {
|
|
const key = hostPortKeyOrNull(url);
|
|
if (key === null) return;
|
|
const database = options?.scope === "database" ? decodeURIComponent(new URL(url).pathname.slice(1)) : undefined;
|
|
if (database === "") throw new Error("Database-scoped teardown requires an explicit database name");
|
|
const refs = clientsByHostPort.get(key);
|
|
if (!refs) return;
|
|
|
|
const clients: RegisteredPostgresClient[] = [];
|
|
for (const ref of refs) {
|
|
const client = ref.deref();
|
|
if (client && database !== undefined && client.options.database !== database) continue;
|
|
refs.delete(ref);
|
|
clientFinalizer.unregister(ref);
|
|
if (client) clients.push(client);
|
|
}
|
|
if (refs.size === 0) clientsByHostPort.delete(key);
|
|
|
|
await Promise.all(clients.map((client) => client.end({ timeout: 1 }).catch(() => {})));
|
|
}
|
|
|
|
function isSafeIdentifier(value: string): boolean {
|
|
return /^[A-Za-z_][A-Za-z0-9_]*$/.test(value);
|
|
}
|
|
|
|
function quoteIdentifier(value: string): string {
|
|
if (!isSafeIdentifier(value)) throw new Error(`Unsafe SQL identifier: ${value}`);
|
|
return `"${value.replaceAll("\"", "\"\"")}"`;
|
|
}
|
|
|
|
function quoteLiteral(value: string): string {
|
|
return `'${value.replaceAll("'", "''")}'`;
|
|
}
|
|
|
|
function splitMigrationStatements(content: string): string[] {
|
|
return content
|
|
.split("--> statement-breakpoint")
|
|
.map((statement) => statement.trim())
|
|
.filter((statement) => statement.length > 0);
|
|
}
|
|
|
|
export type MigrationState =
|
|
| {
|
|
status: "upToDate";
|
|
tableCount: number;
|
|
availableMigrations: string[];
|
|
appliedMigrations: string[];
|
|
journalEntryCount: number;
|
|
}
|
|
| {
|
|
status: "needsMigrations";
|
|
tableCount: number;
|
|
availableMigrations: string[];
|
|
appliedMigrations: string[];
|
|
pendingMigrations: string[];
|
|
journalEntryCount: number;
|
|
reason: "no-migration-journal-empty-db" | "no-migration-journal-non-empty-db" | "pending-migrations";
|
|
};
|
|
|
|
export interface DatabaseClientOptions {
|
|
prepare?: boolean;
|
|
/** postgres.js `max` — connection pool size (driver default: 10). */
|
|
maxConnections?: number;
|
|
/** postgres.js `idle_timeout` in seconds (driver default: disabled). */
|
|
idleTimeoutSeconds?: number;
|
|
/** postgres.js `connect_timeout` in seconds (driver default: 30). */
|
|
connectTimeoutSeconds?: number;
|
|
/**
|
|
* postgres.js `max_lifetime` in seconds. Bounds how long one pooled
|
|
* connection is reused before the client replaces it (driver default: a
|
|
* random value between 30 and 60 minutes).
|
|
*/
|
|
maxLifetimeSeconds?: number;
|
|
/**
|
|
* postgres.js `connection.application_name`, shown in
|
|
* `pg_stat_activity.application_name`. Lets an operator tell Paperclip's
|
|
* pool apart from other clients of the same database (driver default:
|
|
* `postgres.js`).
|
|
*/
|
|
applicationName?: string;
|
|
/**
|
|
* Connection-level `statement_timeout` in milliseconds, sent as a startup
|
|
* parameter (no extra round trip). For dedicated single-purpose pools only.
|
|
*/
|
|
statementTimeoutMs?: number;
|
|
dedicatedConnectionMax?: number;
|
|
dedicatedConnectionWaitMs?: number;
|
|
}
|
|
|
|
/**
|
|
* Idle pooled connections close after this many seconds unless
|
|
* `DATABASE_IDLE_TIMEOUT_SECONDS` says otherwise. The driver default keeps an
|
|
* idle connection open forever, so a process that stops issuing queries still
|
|
* holds every backend it ever opened. Set `DATABASE_IDLE_TIMEOUT_SECONDS=0`
|
|
* to restore the driver default.
|
|
*/
|
|
export const DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS = 60;
|
|
/** `application_name` reported to PostgreSQL unless `DATABASE_APPLICATION_NAME` overrides it. */
|
|
export const DEFAULT_DATABASE_APPLICATION_NAME = "paperclip";
|
|
|
|
function envBoolean(env: NodeJS.ProcessEnv, name: string): boolean | undefined {
|
|
const value = env[name]?.trim().toLowerCase();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (value === "true" || value === "1") return true;
|
|
if (value === "false" || value === "0") return false;
|
|
throw new Error(`${name} must be "true" or "false", got: ${env[name]}`);
|
|
}
|
|
|
|
function envPositiveInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (!/^[1-9]\d*$/.test(value)) {
|
|
throw new Error(`${name} must be a positive integer, got: ${env[name]}`);
|
|
}
|
|
return Number.parseInt(value, 10);
|
|
}
|
|
|
|
function envNonNegativeInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (!/^(?:0|[1-9]\d*)$/.test(value)) {
|
|
throw new Error(`${name} must be a non-negative integer, got: ${env[name]}`);
|
|
}
|
|
return Number.parseInt(value, 10);
|
|
}
|
|
|
|
function envNonEmptyString(env: NodeJS.ProcessEnv, name: string): string | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
return value;
|
|
}
|
|
|
|
/**
|
|
* Database client tuning from the environment, so hosted deployments can
|
|
* adapt to their connection topology (pooled endpoints, network latency)
|
|
* without editing source. Every variable is optional. This function returns
|
|
* only the values the environment sets; `resolveDatabaseClientOptions` adds
|
|
* Paperclip's own defaults on top, and the driver defaults apply to the rest
|
|
* — self-hosted setups need none of these.
|
|
*/
|
|
export function databaseClientOptionsFromEnv(env: NodeJS.ProcessEnv = process.env): DatabaseClientOptions {
|
|
const options: DatabaseClientOptions = {};
|
|
const prepare = envBoolean(env, "DATABASE_PREPARED_STATEMENTS");
|
|
if (prepare !== undefined) options.prepare = prepare;
|
|
const maxConnections = envPositiveInteger(env, "DATABASE_POOL_MAX");
|
|
if (maxConnections !== undefined) options.maxConnections = maxConnections;
|
|
// `0` is allowed here: it disables idle reaping (the driver default).
|
|
const idleTimeoutSeconds = envNonNegativeInteger(env, "DATABASE_IDLE_TIMEOUT_SECONDS");
|
|
if (idleTimeoutSeconds !== undefined) options.idleTimeoutSeconds = idleTimeoutSeconds;
|
|
const connectTimeoutSeconds = envPositiveInteger(env, "DATABASE_CONNECT_TIMEOUT_SECONDS");
|
|
if (connectTimeoutSeconds !== undefined) options.connectTimeoutSeconds = connectTimeoutSeconds;
|
|
const maxLifetimeSeconds = envPositiveInteger(env, "DATABASE_MAX_LIFETIME_SECONDS");
|
|
if (maxLifetimeSeconds !== undefined) options.maxLifetimeSeconds = maxLifetimeSeconds;
|
|
const applicationName = envNonEmptyString(env, "DATABASE_APPLICATION_NAME");
|
|
if (applicationName !== undefined) options.applicationName = applicationName;
|
|
const dedicatedConnectionMax = envPositiveInteger(env, "DATABASE_DEDICATED_CONNECTION_MAX");
|
|
if (dedicatedConnectionMax !== undefined) options.dedicatedConnectionMax = dedicatedConnectionMax;
|
|
const dedicatedConnectionWaitMs = envPositiveInteger(env, "DATABASE_DEDICATED_CONNECTION_WAIT_MS");
|
|
if (dedicatedConnectionWaitMs !== undefined) options.dedicatedConnectionWaitMs = dedicatedConnectionWaitMs;
|
|
return options;
|
|
}
|
|
|
|
/**
|
|
* Fills in Paperclip's defaults for the options the caller left unset: idle
|
|
* connections are reaped after `DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS`, and the
|
|
* pool identifies itself as `DEFAULT_DATABASE_APPLICATION_NAME`. Everything
|
|
* else stays at the driver default. An explicit value (including
|
|
* `idleTimeoutSeconds: 0`) always wins over the default.
|
|
*/
|
|
export function resolveDatabaseClientOptions(options: DatabaseClientOptions): DatabaseClientOptions {
|
|
return {
|
|
...options,
|
|
idleTimeoutSeconds: options.idleTimeoutSeconds ?? DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS,
|
|
applicationName: options.applicationName ?? DEFAULT_DATABASE_APPLICATION_NAME,
|
|
};
|
|
}
|
|
|
|
export function postgresJsOptions(options: DatabaseClientOptions): Record<string, unknown> {
|
|
const driverOptions: Record<string, unknown> = {};
|
|
if (options.prepare !== undefined) driverOptions.prepare = options.prepare;
|
|
if (options.maxConnections !== undefined) driverOptions.max = options.maxConnections;
|
|
if (options.idleTimeoutSeconds !== undefined) driverOptions.idle_timeout = options.idleTimeoutSeconds;
|
|
if (options.connectTimeoutSeconds !== undefined) driverOptions.connect_timeout = options.connectTimeoutSeconds;
|
|
if (options.maxLifetimeSeconds !== undefined) driverOptions.max_lifetime = options.maxLifetimeSeconds;
|
|
const connection: Record<string, string> = {};
|
|
if (options.applicationName !== undefined) connection.application_name = options.applicationName;
|
|
if (options.statementTimeoutMs !== undefined) connection.statement_timeout = String(options.statementTimeoutMs);
|
|
if (Object.keys(connection).length) driverOptions.connection = connection;
|
|
return driverOptions;
|
|
}
|
|
|
|
// A long advisory-lock transaction must not borrow the normal pool that its
|
|
// work needs for progress writes. Keep connection configuration private to the
|
|
// originating Db lifetime; callers receive neither URLs nor credentials.
|
|
const ownedDatabaseListenerFactories = new WeakMap<object, () => Db>();
|
|
|
|
export const DEDICATED_CONNECTION_IDLE_MAX = 4;
|
|
export const DEFAULT_DEDICATED_CONNECTION_MAX = 24;
|
|
export const DEFAULT_DEDICATED_CONNECTION_WAIT_MS = 30_000;
|
|
|
|
type DedicatedConnectionWaiter = { grant(): void; timer: ReturnType<typeof setTimeout> };
|
|
type DedicatedConnectionSet = {
|
|
create(): Db;
|
|
idle: Db[];
|
|
ended: WeakSet<Db>;
|
|
closed: boolean;
|
|
limit: number;
|
|
waitMs: number;
|
|
inUse: number;
|
|
waiters: DedicatedConnectionWaiter[];
|
|
stats: DedicatedDbConnectionStats;
|
|
};
|
|
export type DedicatedDbConnectionStats = { limit: number; inUse: number; peakInUse: number; waiting: number;
|
|
queued: number; overLimit: number; nestedOverLimit: number };
|
|
const dedicatedConnectionSets = new WeakMap<object, DedicatedConnectionSet>();
|
|
type DedicatedConnectionHold = { set: DedicatedConnectionSet; active: boolean };
|
|
const heldDedicatedConnections = new AsyncLocalStorage<readonly DedicatedConnectionHold[]>();
|
|
const openTransactions = new AsyncLocalStorage<{ active: boolean }>();
|
|
|
|
function dedicatedConnectionSet(create: DedicatedConnectionSet["create"], resolved: DatabaseClientOptions): DedicatedConnectionSet {
|
|
const limit = resolved.dedicatedConnectionMax ?? DEFAULT_DEDICATED_CONNECTION_MAX;
|
|
return { create, idle: [], ended: new WeakSet(), closed: false, limit, inUse: 0, waiters: [],
|
|
waitMs: resolved.dedicatedConnectionWaitMs ?? DEFAULT_DEDICATED_CONNECTION_WAIT_MS,
|
|
stats: { limit, inUse: 0, peakInUse: 0, waiting: 0, queued: 0, overLimit: 0, nestedOverLimit: 0 } };
|
|
}
|
|
|
|
function takeDedicatedSlot(set: DedicatedConnectionSet) {
|
|
set.inUse++;
|
|
set.stats.inUse = set.inUse;
|
|
set.stats.peakInUse = Math.max(set.stats.peakInUse, set.inUse);
|
|
}
|
|
|
|
function releaseDedicatedSlot(set: DedicatedConnectionSet) {
|
|
set.inUse--;
|
|
set.stats.inUse = set.inUse;
|
|
while (set.waiters.length && set.inUse < set.limit) set.waiters.shift()!.grant();
|
|
set.stats.waiting = set.waiters.length;
|
|
}
|
|
|
|
function holdsLiveDedicatedOrTransaction(set: DedicatedConnectionSet): boolean {
|
|
return Boolean(heldDedicatedConnections.getStore()?.some(hold => hold.active && hold.set === set)
|
|
|| openTransactions.getStore()?.active);
|
|
}
|
|
|
|
async function admitDedicatedConnection(set: DedicatedConnectionSet, nested: boolean): Promise<void> {
|
|
if (nested || (set.inUse < set.limit && set.waiters.length === 0)) {
|
|
if (nested && set.inUse >= set.limit) set.stats.nestedOverLimit++;
|
|
takeDedicatedSlot(set);
|
|
return;
|
|
}
|
|
set.stats.queued++;
|
|
await new Promise<void>(resolve => {
|
|
const waiter: DedicatedConnectionWaiter = {
|
|
grant: () => { clearTimeout(waiter.timer); takeDedicatedSlot(set); resolve(); },
|
|
timer: setTimeout(() => {
|
|
const index = set.waiters.indexOf(waiter);
|
|
if (index < 0) return;
|
|
set.waiters.splice(index, 1);
|
|
set.stats.waiting = set.waiters.length;
|
|
set.stats.overLimit++;
|
|
takeDedicatedSlot(set);
|
|
resolve();
|
|
}, set.waitMs),
|
|
};
|
|
set.waiters.push(waiter);
|
|
set.stats.waiting = set.waiters.length;
|
|
});
|
|
}
|
|
|
|
function markOpenTransactions(db: Db) {
|
|
const transaction = db.transaction.bind(db) as (...args: unknown[]) => Promise<unknown>;
|
|
db.transaction = ((fn: (tx: unknown) => Promise<unknown>, ...rest: unknown[]) => transaction(async (tx: unknown) => {
|
|
const open = { active: true };
|
|
try { return await openTransactions.run(open, () => fn(tx)); }
|
|
finally { open.active = false; }
|
|
}, ...rest)) as Db["transaction"];
|
|
}
|
|
|
|
export const DRIVER_DEFAULT_POOL_MAX_CONNECTIONS = 10;
|
|
export const PRIVATE_POOL_MAX_CONNECTIONS = {
|
|
liveness: 2, legacyControllerLease: 1, controllerInstanceLease: 1, checkpoint: 3, fileStorage: 2,
|
|
} as const;
|
|
export const DATABASE_LISTENER_CONNECTIONS = 2;
|
|
|
|
export function databaseConnectionBudget(options: DatabaseClientOptions = {}) {
|
|
const main = options.maxConnections ?? DRIVER_DEFAULT_POOL_MAX_CONNECTIONS;
|
|
const privatePools = Object.values(PRIVATE_POOL_MAX_CONNECTIONS).reduce((sum, value) => sum + value, 0);
|
|
const dedicatedLimit = options.dedicatedConnectionMax ?? DEFAULT_DEDICATED_CONNECTION_MAX;
|
|
const dedicated = 2 * dedicatedLimit + DEDICATED_CONNECTION_IDLE_MAX;
|
|
return { main, privatePools, listeners: DATABASE_LISTENER_CONNECTIONS, dedicatedLimit, dedicated,
|
|
total: main + privatePools + DATABASE_LISTENER_CONNECTIONS + dedicated };
|
|
}
|
|
|
|
const preparedQueryStates = new WeakMap<object, PreparedQueryState>();
|
|
|
|
export function preparedStatementStats(db: Db): PreparedQueryStats | null {
|
|
const state = preparedQueryStates.get(db);
|
|
return state ? preparedQueryStats(state) : null;
|
|
}
|
|
|
|
export function dedicatedDbConnectionStats(db: Db): DedicatedDbConnectionStats | null {
|
|
const set = dedicatedConnectionSets.get(db);
|
|
return set ? { ...set.stats } : null;
|
|
}
|
|
|
|
function trackClientEnd(client: Db, onEnd: (timeoutSeconds: number | undefined) => Promise<void> | void): void {
|
|
const sql = client.$client;
|
|
const end = sql.end;
|
|
sql.end = async (options?: { timeout?: number }) => {
|
|
await Promise.all([onEnd(options?.timeout), end(options)]);
|
|
};
|
|
}
|
|
|
|
async function endDedicatedConnections(clients: Db[], timeoutSeconds: number): Promise<void> {
|
|
await Promise.all(clients.map(client => client.$client.end({ timeout: timeoutSeconds }).catch(() => undefined)));
|
|
}
|
|
|
|
export async function closeDedicatedDbConnections(db: Db, timeoutSeconds: number): Promise<void> {
|
|
const set = dedicatedConnectionSets.get(db);
|
|
if (!set) return;
|
|
set.closed = true;
|
|
await endDedicatedConnections(set.idle.splice(0), timeoutSeconds);
|
|
}
|
|
|
|
export interface OwnedDatabaseListener {
|
|
listen: Db["$client"]["listen"];
|
|
close(): Promise<void>;
|
|
}
|
|
|
|
export function createOwnedDatabaseListener(db: Db): OwnedDatabaseListener {
|
|
const factory = ownedDatabaseListenerFactories.get(db);
|
|
if (!factory) throw new Error("owned_listener_requires_create_db");
|
|
const owned = factory();
|
|
return {
|
|
listen: owned.$client.listen.bind(owned.$client),
|
|
close: () => owned.$client.end({ timeout: 1 }),
|
|
};
|
|
}
|
|
|
|
export type DedicatedDbConnectionOptions = { admitWithoutWaiting?: boolean };
|
|
|
|
export async function withDedicatedDbConnection<T>(db: Db, action: (dedicated: Db) => Promise<T>,
|
|
options: DedicatedDbConnectionOptions = {}): Promise<T> {
|
|
const set = dedicatedConnectionSets.get(db);
|
|
if (!set) throw new Error("dedicated_connection_requires_create_db");
|
|
await admitDedicatedConnection(set, options.admitWithoutWaiting === true || holdsLiveDedicatedOrTransaction(set));
|
|
const hold: DedicatedConnectionHold = { set, active: true };
|
|
let dedicated: Db | undefined;
|
|
let completed = false;
|
|
try {
|
|
while (!dedicated && set.idle.length) {
|
|
const candidate = set.idle.pop()!;
|
|
if (!set.ended.has(candidate)) dedicated = candidate;
|
|
}
|
|
dedicated ??= set.create();
|
|
const connection = dedicated;
|
|
const result = await heldDedicatedConnections.run([...(heldDedicatedConnections.getStore() ?? []), hold], () => action(connection));
|
|
completed = true;
|
|
return result;
|
|
} finally {
|
|
hold.active = false;
|
|
try {
|
|
if (dedicated && completed && !set.closed && !set.ended.has(dedicated) && set.idle.length < DEDICATED_CONNECTION_IDLE_MAX) {
|
|
set.idle.push(dedicated);
|
|
} else if (dedicated && !set.ended.has(dedicated)) {
|
|
await dedicated.$client.end({ timeout: 1 });
|
|
}
|
|
} finally {
|
|
releaseDedicatedSlot(set);
|
|
}
|
|
}
|
|
}
|
|
|
|
// postgres.js pipelines a query onto a busy connection when none is idle, so
|
|
// a short liveness write can wait behind an unrelated slow statement with no
|
|
// client-side bound. Liveness writes (controller-lease renewal) get a small
|
|
// private pool, created on first use and shared for the owner's lifetime.
|
|
const livenessDbFactories = new WeakMap<object, () => Db>();
|
|
const livenessDbs = new WeakMap<object, Db>();
|
|
|
|
/** The private liveness pool for a `createDb` client; other clients (test
|
|
* doubles, transactions) are returned unchanged. */
|
|
export function livenessDb(db: Db): Db {
|
|
const existing = livenessDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = livenessDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory();
|
|
livenessDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
const controllerLeaseDbFactories = new WeakMap<object, () => Db>();
|
|
const controllerLeaseDbs = new WeakMap<object, Db>();
|
|
|
|
export function controllerLeaseDb(db: Db): Db {
|
|
const existing = controllerLeaseDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = controllerLeaseDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory();
|
|
controllerLeaseDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
const controllerInstanceLeaseDbFactories = new WeakMap<object, (statementTimeoutMs: number | null) => Db>();
|
|
const controllerInstanceLeaseDbs = new WeakMap<object, Db>();
|
|
|
|
/** One private connection for the server's controller-instance lease, so its
|
|
* renewal never waits behind request or worker traffic on the main pool.
|
|
* `statementTimeoutMs` (first call wins) is a connection-level
|
|
* `statement_timeout`: the server cancels a slow renewal, so it never holds the
|
|
* single connection that the next renewal needs, at no extra round trip. */
|
|
export function controllerInstanceLeaseDb(db: Db, options: { statementTimeoutMs?: number | null } = {}): Db {
|
|
const existing = controllerInstanceLeaseDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = controllerInstanceLeaseDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory(options.statementTimeoutMs ?? null);
|
|
controllerInstanceLeaseDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
/** Replaces the lease connection with one that sends no `statement_timeout`
|
|
* startup parameter, for a connection pooler that refuses it. */
|
|
export function controllerInstanceLeaseDbWithoutStatementTimeout(db: Db): Db {
|
|
const factory = controllerInstanceLeaseDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const previous = controllerInstanceLeaseDbs.get(db);
|
|
const created = factory(null);
|
|
controllerInstanceLeaseDbs.set(db, created);
|
|
void previous?.$client.end({ timeout: 0 }).catch(() => undefined);
|
|
return created;
|
|
}
|
|
|
|
/** Ends the liveness pool if one was created. The ended pool stays cached, so
|
|
* a late renewal fails like a query on the ended primary pool. */
|
|
export async function closeLivenessDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
await Promise.all([livenessDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
|
|
controllerLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
|
|
controllerInstanceLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds })]);
|
|
}
|
|
|
|
// Control checkpoints must not occupy the request or lease-renewal pools
|
|
// during an aligned burst. Share at most three lazy connections per owner.
|
|
const checkpointDbFactories = new WeakMap<object, () => Db>();
|
|
const checkpointDbs = new WeakMap<object, Db>();
|
|
const closedCheckpointOwners = new WeakSet<object>();
|
|
|
|
/** The bounded controller checkpoint pool; transactions and test doubles keep
|
|
* their existing database when they have no owning client factory. */
|
|
export function checkpointDb(db: Db): Db {
|
|
const existing = checkpointDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = checkpointDbFactories.get(db);
|
|
if (!factory) return db;
|
|
if (closedCheckpointOwners.has(db)) throw new Error("checkpoint_pool_closed");
|
|
const created = factory();
|
|
checkpointDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
/** Fence lazy creation immediately, and keep any ended pool cached. */
|
|
export async function closeCheckpointDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
closedCheckpointOwners.add(db);
|
|
await checkpointDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
|
|
}
|
|
|
|
// Managed file publication holds an agent lock while transferring bytes. A
|
|
// stalled object store must not consume request or controller-renewal capacity.
|
|
const fileStorageDbFactories = new WeakMap<object, () => Db>();
|
|
const fileStorageDbs = new WeakMap<object, Db>();
|
|
const closedFileStorageOwners = new WeakSet<object>();
|
|
export function fileStorageDb(db: Db): Db {
|
|
const existing = fileStorageDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = fileStorageDbFactories.get(db);
|
|
if (!factory) return db; // Preserve the caller's transaction/savepoint.
|
|
if (closedFileStorageOwners.has(db)) throw new Error("file_storage_pool_closed");
|
|
const created = factory();
|
|
fileStorageDbs.set(db, created);
|
|
fileStorageDbs.set(created, created);
|
|
return created;
|
|
}
|
|
export async function closeFileStorageDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
closedFileStorageOwners.add(db);
|
|
await fileStorageDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
|
|
}
|
|
|
|
export function createDb(url: string, options?: DatabaseClientOptions) {
|
|
const resolved = resolveDatabaseClientOptions(options ?? databaseClientOptionsFromEnv());
|
|
return createDbClient(url, resolved);
|
|
}
|
|
|
|
function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: object, dedicatedSet?: DedicatedConnectionSet) {
|
|
const sql = postgres(url, postgresJsOptions(resolved));
|
|
const preparedState = resolved.prepare === true
|
|
? (owner && preparedQueryStates.get(owner)) || createPreparedQueryState()
|
|
: undefined;
|
|
if (preparedState) installPreparedQueries(sql, preparedState);
|
|
const key = hostPortKeyOrNull(url);
|
|
if (key) registerClient(key, sql);
|
|
// A disconnect can lose the response after a statement has committed.
|
|
// postgres.js calls that error "write CONNECTION_CLOSED" too, so the
|
|
// message cannot establish that replay is safe. Leave retries to callers
|
|
// that know the complete operation is idempotent.
|
|
const db = installDatabaseWorkSignals(drizzleWithLazyTransactionQueries(sql, schema), owner);
|
|
installDatabaseResponseCapture(db, schema);
|
|
markOpenTransactions(db);
|
|
if (preparedState) preparedQueryStates.set(db, preparedState);
|
|
const ownerSet = owner ? dedicatedConnectionSets.get(owner) : undefined;
|
|
if (dedicatedSet) {
|
|
trackClientEnd(db, () => { dedicatedSet.ended.add(db); });
|
|
dedicatedConnectionSets.set(db, dedicatedSet);
|
|
} else if (ownerSet) {
|
|
dedicatedConnectionSets.set(db, ownerSet);
|
|
} else {
|
|
const set: DedicatedConnectionSet = dedicatedConnectionSet(() => createDbClient(url, {
|
|
...resolved, maxConnections: 1, applicationName: "paperclip-workspace-finalization-lock",
|
|
}, db, set), resolved);
|
|
dedicatedConnectionSets.set(db, set);
|
|
trackClientEnd(db, timeoutSeconds => closeDedicatedDbConnections(db, timeoutSeconds ?? 1));
|
|
}
|
|
ownedDatabaseListenerFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: 1,
|
|
}, db));
|
|
livenessDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.liveness, applicationName: "paperclip-controller-lease",
|
|
}, db));
|
|
controllerLeaseDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.legacyControllerLease, applicationName: "paperclip-legacy-controller-lease",
|
|
}, db));
|
|
controllerInstanceLeaseDbFactories.set(db, statementTimeoutMs => createDbClient(url, {
|
|
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.controllerInstanceLease, applicationName: "paperclip-controller-instance-lease",
|
|
...(statementTimeoutMs !== null ? { statementTimeoutMs } : {}),
|
|
}, db));
|
|
checkpointDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.checkpoint, applicationName: "paperclip-runner-checkpoint",
|
|
}, db));
|
|
fileStorageDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: PRIVATE_POOL_MAX_CONNECTIONS.fileStorage, applicationName: "paperclip-file-storage",
|
|
}, db));
|
|
return db;
|
|
}
|
|
|
|
export async function getPostgresDataDirectory(url: string): Promise<string | null> {
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
const rows = await sql<{ data_directory: string | null }[]>`
|
|
SELECT current_setting('data_directory', true) AS data_directory
|
|
`;
|
|
const actual = rows[0]?.data_directory;
|
|
return typeof actual === "string" && actual.length > 0 ? actual : null;
|
|
} catch {
|
|
return null;
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
async function listMigrationFiles(): Promise<string[]> {
|
|
const entries = await readdir(MIGRATIONS_FOLDER, { withFileTypes: true });
|
|
return entries
|
|
.filter((entry) => entry.isFile() && entry.name.endsWith(".sql"))
|
|
.map((entry) => entry.name)
|
|
.sort((a, b) => a.localeCompare(b));
|
|
}
|
|
|
|
type MigrationJournalFile = {
|
|
entries?: Array<{ idx?: number; tag?: string; when?: number }>;
|
|
};
|
|
|
|
type JournalMigrationEntry = {
|
|
fileName: string;
|
|
folderMillis: number;
|
|
order: number;
|
|
};
|
|
|
|
async function listJournalMigrationEntries(): Promise<JournalMigrationEntry[]> {
|
|
try {
|
|
const raw = await readFile(MIGRATIONS_JOURNAL_JSON, "utf8");
|
|
const parsed = JSON.parse(raw) as MigrationJournalFile;
|
|
if (!Array.isArray(parsed.entries)) return [];
|
|
return parsed.entries
|
|
.map((entry, entryIndex) => {
|
|
if (typeof entry?.tag !== "string") return null;
|
|
if (typeof entry?.when !== "number" || !Number.isFinite(entry.when)) return null;
|
|
const order = Number.isInteger(entry.idx) ? Number(entry.idx) : entryIndex;
|
|
return { fileName: `${entry.tag}.sql`, folderMillis: entry.when, order };
|
|
})
|
|
.filter((entry): entry is JournalMigrationEntry => entry !== null);
|
|
} catch {
|
|
return [];
|
|
}
|
|
}
|
|
|
|
async function listJournalMigrationFiles(): Promise<string[]> {
|
|
const entries = await listJournalMigrationEntries();
|
|
return entries.map((entry) => entry.fileName);
|
|
}
|
|
|
|
async function readMigrationFileContent(migrationFile: string): Promise<string> {
|
|
return readFile(new URL(`./migrations/${migrationFile}`, import.meta.url), "utf8");
|
|
}
|
|
|
|
async function orderMigrationsByJournal(migrationFiles: string[]): Promise<string[]> {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const orderByFileName = new Map(journalEntries.map((entry) => [entry.fileName, entry.order]));
|
|
return [...migrationFiles].sort((left, right) => {
|
|
const leftOrder = orderByFileName.get(left);
|
|
const rightOrder = orderByFileName.get(right);
|
|
if (leftOrder === undefined && rightOrder === undefined) return left.localeCompare(right);
|
|
if (leftOrder === undefined) return 1;
|
|
if (rightOrder === undefined) return -1;
|
|
if (leftOrder === rightOrder) return left.localeCompare(right);
|
|
return leftOrder - rightOrder;
|
|
});
|
|
}
|
|
|
|
type SqlExecutor = Pick<ReturnType<typeof postgres>, "unsafe">;
|
|
|
|
async function runInTransaction(sql: SqlExecutor, action: () => Promise<void>): Promise<void> {
|
|
await sql.unsafe("BEGIN");
|
|
try {
|
|
await action();
|
|
await sql.unsafe("COMMIT");
|
|
} catch (error) {
|
|
try {
|
|
await sql.unsafe("ROLLBACK");
|
|
} catch {
|
|
// Ignore rollback failures and surface the original error.
|
|
}
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
async function latestMigrationCreatedAt(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
): Promise<number | null> {
|
|
const rows = await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC NULLS LAST LIMIT 1`,
|
|
);
|
|
const value = Number(rows[0]?.created_at ?? Number.NaN);
|
|
return Number.isFinite(value) ? value : null;
|
|
}
|
|
|
|
function normalizeFolderMillis(value: number | null | undefined): number {
|
|
if (typeof value === "number" && Number.isFinite(value) && value >= 0) {
|
|
return Math.trunc(value);
|
|
}
|
|
return Date.now();
|
|
}
|
|
|
|
async function ensureMigrationJournalTable(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<{ migrationTableSchema: string; columnNames: Set<string> }> {
|
|
let migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
const drizzleSchema = quoteIdentifier("drizzle");
|
|
const migrationTable = quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE);
|
|
await sql.unsafe(`CREATE SCHEMA IF NOT EXISTS ${drizzleSchema}`);
|
|
await sql.unsafe(
|
|
`CREATE TABLE IF NOT EXISTS ${drizzleSchema}.${migrationTable} (id SERIAL PRIMARY KEY, hash text NOT NULL, created_at bigint)`,
|
|
);
|
|
migrationTableSchema = (await discoverMigrationTableSchema(sql)) ?? "drizzle";
|
|
}
|
|
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
return { migrationTableSchema, columnNames };
|
|
}
|
|
|
|
async function migrationHistoryEntryExists(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
columnNames: Set<string>,
|
|
migrationFile: string,
|
|
hash: string,
|
|
): Promise<boolean> {
|
|
const predicates: string[] = [];
|
|
if (columnNames.has("hash")) predicates.push(`hash = ${quoteLiteral(hash)}`);
|
|
if (columnNames.has("name")) predicates.push(`name = ${quoteLiteral(migrationFile)}`);
|
|
if (predicates.length === 0) return false;
|
|
|
|
const rows = await sql.unsafe<{ one: number }[]>(
|
|
`SELECT 1 AS one FROM ${qualifiedTable} WHERE ${predicates.join(" OR ")} LIMIT 1`,
|
|
);
|
|
return rows.length > 0;
|
|
}
|
|
|
|
async function recordMigrationHistoryEntry(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
columnNames: Set<string>,
|
|
migrationFile: string,
|
|
hash: string,
|
|
folderMillis: number,
|
|
): Promise<void> {
|
|
const insertColumns: string[] = [];
|
|
const insertValues: string[] = [];
|
|
|
|
if (columnNames.has("hash")) {
|
|
insertColumns.push(quoteIdentifier("hash"));
|
|
insertValues.push(quoteLiteral(hash));
|
|
}
|
|
if (columnNames.has("name")) {
|
|
insertColumns.push(quoteIdentifier("name"));
|
|
insertValues.push(quoteLiteral(migrationFile));
|
|
}
|
|
if (columnNames.has("created_at")) {
|
|
const latestCreatedAt = await latestMigrationCreatedAt(sql, qualifiedTable);
|
|
const createdAt = latestCreatedAt === null
|
|
? normalizeFolderMillis(folderMillis)
|
|
: Math.max(latestCreatedAt + 1, normalizeFolderMillis(folderMillis));
|
|
insertColumns.push(quoteIdentifier("created_at"));
|
|
insertValues.push(quoteLiteral(String(createdAt)));
|
|
}
|
|
|
|
if (insertColumns.length === 0) return;
|
|
|
|
await sql.unsafe(
|
|
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
|
|
);
|
|
}
|
|
|
|
// These idempotent migrations require top-level statements for batch commits
|
|
// or concurrent index builds. History is recorded only after completion.
|
|
const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
|
|
"0313_private_task_access.sql",
|
|
"0314_private_task_draft_assets.sql",
|
|
"0332_little_night_nurse.sql",
|
|
"0345_luxuriant_carmella_unuscione.sql",
|
|
"0369_late_logan.sql",
|
|
"0381_mean_scream.sql",
|
|
"0382_narrow_the_executioner.sql",
|
|
"0385_open_madame_web.sql",
|
|
"0386_steady_epoch.sql",
|
|
"0387_boring_professor_monster.sql",
|
|
"0388_ambitious_micromax.sql",
|
|
"0389_brave_hulk.sql",
|
|
"0390_salty_killraven.sql",
|
|
"0391_third_roulette.sql",
|
|
"0392_small_frightful_four.sql",
|
|
"0393_open_grandmaster.sql",
|
|
"0395_tranquil_firestar.sql",
|
|
"0396_tan_paladin.sql",
|
|
"0397_known_madrox.sql",
|
|
"0398_yielding_random.sql",
|
|
"0399_ambitious_onslaught.sql",
|
|
"0400_absent_phantom_reporter.sql",
|
|
"0401_yummy_ultimates.sql",
|
|
"0402_ancient_malice.sql",
|
|
"0403_natural_cerise.sql",
|
|
"0406_zippy_war_machine.sql",
|
|
"0407_greedy_star_brand.sql",
|
|
"0408_short_red_skull.sql",
|
|
"0409_uneven_zuras.sql",
|
|
"0410_freezing_kinsey_walden.sql",
|
|
"0411_real_scrambler.sql",
|
|
"0412_tearful_squadron_supreme.sql",
|
|
"0413_tired_kronos.sql",
|
|
"0414_closed_edwin_jarvis.sql",
|
|
"0419_slippery_impossible_man.sql",
|
|
"0432_strong_mole_man.sql",
|
|
"0435_premium_puff_adder.sql",
|
|
"0436_legal_reavers.sql",
|
|
"0447_legacy_controller_process.sql",
|
|
"0450_petite_frightful_four.sql",
|
|
"0453_curious_ben_parker.sql",
|
|
"0454_lumpy_ronan.sql",
|
|
"0455_pending_budget_cancellation_index.sql",
|
|
"0456_chat_review_lookup_indexes.sql",
|
|
"0457_native_session_run_index.sql",
|
|
"0458_chat_admission_issue_index.sql",
|
|
"0459_environment_lease_backup_scope_index.sql",
|
|
"0460_workspace_activity_context_index.sql",
|
|
"0461_created_from_legacy_origin_run_index.sql",
|
|
"0470_heartbeat_runs_queued_created_index.sql",
|
|
"0472_runner_api_capture_pending_deletes.sql",
|
|
"0476_wakeup_request_run_index.sql",
|
|
"0486_heartbeat_runs_company_finished_index.sql",
|
|
"0487_heartbeat_runs_settled_process_indexes.sql",
|
|
"0469_zippy_photon.sql",
|
|
|
|
|
|
"0384_funny_adam_warlock.sql",
|
|
"0383_tool_review_company_expiry.sql",
|
|
]);
|
|
|
|
async function applyPendingMigrationsManually(
|
|
url: string,
|
|
pendingMigrations: string[],
|
|
): Promise<void> {
|
|
if (pendingMigrations.length === 0) return;
|
|
|
|
const orderedPendingMigrations = await orderMigrationsByJournal(pendingMigrations);
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const folderMillisByFileName = new Map(
|
|
journalEntries.map((entry) => [entry.fileName, normalizeFolderMillis(entry.folderMillis)]),
|
|
);
|
|
|
|
const pool = createUtilitySql(url);
|
|
try {
|
|
const sql = await pool.reserve();
|
|
try {
|
|
// A session lock survives each batch COMMIT. Reserve the connection so
|
|
// pool rotation cannot release it while another migrator is waiting.
|
|
// A blocking lock SELECT retains its statement snapshot while waiting.
|
|
// A partial concurrent index build can wait for that snapshot, forming a
|
|
// deadlock with this lock. Each failed try ends its snapshot before sleep.
|
|
while (!(await sql<{ locked: boolean }[]>`
|
|
SELECT pg_try_advisory_lock(hashtextextended('paperclip:migrations', 0)) AS locked`)[0]?.locked) {
|
|
await new Promise((resolve) => setTimeout(resolve, 100));
|
|
}
|
|
const { migrationTableSchema, columnNames } = await ensureMigrationJournalTable(sql);
|
|
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
|
|
for (const migrationFile of orderedPendingMigrations) {
|
|
const migrationContent = await readMigrationFileContent(migrationFile);
|
|
const hash = createHash("sha256").update(migrationContent).digest("hex");
|
|
const existingEntry = await migrationHistoryEntryExists(
|
|
sql,
|
|
qualifiedTable,
|
|
columnNames,
|
|
migrationFile,
|
|
hash,
|
|
);
|
|
if (existingEntry) continue;
|
|
|
|
const applyMigration = async () => {
|
|
for (const statement of splitMigrationStatements(migrationContent)) {
|
|
if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
|
|
// A cancelled concurrent build leaves an invalid index. IF NOT
|
|
// EXISTS alone would skip it on retry and journal an incomplete index.
|
|
const index = statement.replace(/^\s*--.*$/gm, "").trim()
|
|
.match(/^CREATE (?:UNIQUE )?INDEX CONCURRENTLY IF NOT EXISTS "([A-Za-z_][A-Za-z0-9_]*)"/i);
|
|
if (index) {
|
|
const invalid = await sql<{ invalid: boolean }[]>`
|
|
SELECT NOT i.indisvalid AS invalid FROM pg_index i
|
|
JOIN pg_class c ON c.oid = i.indexrelid
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public' AND c.relname = ${index[1]}`;
|
|
if (invalid[0]?.invalid) {
|
|
await sql.unsafe(`DROP INDEX CONCURRENTLY ${quoteIdentifier("public")}.${quoteIdentifier(index[1])}`);
|
|
}
|
|
}
|
|
}
|
|
// Older dev schemas can lack a replaced constraint. Preserve the
|
|
// published SQL/hash while allowing the harmless drop to proceed.
|
|
const executable = statement.replace(
|
|
/^(\s*ALTER TABLE "[^"]+" DROP CONSTRAINT) (?!IF EXISTS\b)/i,
|
|
"$1 IF EXISTS ",
|
|
);
|
|
await sql.unsafe(executable);
|
|
}
|
|
|
|
await recordMigrationHistoryEntry(
|
|
sql,
|
|
qualifiedTable,
|
|
columnNames,
|
|
migrationFile,
|
|
hash,
|
|
folderMillisByFileName.get(migrationFile) ?? Date.now(),
|
|
);
|
|
};
|
|
if (migrationFile === PROJECT_ENVIRONMENT_OVERLAP.dropFile) {
|
|
const repairFile = PROJECT_ENVIRONMENT_OVERLAP.repairFile;
|
|
const repairContent = await readMigrationFileContent(repairFile);
|
|
const repairHash = createHash("sha256").update(repairContent).digest("hex");
|
|
const dropIndex = orderedPendingMigrations.indexOf(migrationFile);
|
|
if (hash !== PROJECT_ENVIRONMENT_OVERLAP.dropSha256
|
|
|| repairHash !== PROJECT_ENVIRONMENT_OVERLAP.repairSha256
|
|
|| orderedPendingMigrations[dropIndex + 1] !== repairFile
|
|
|| await migrationHistoryEntryExists(sql, qualifiedTable, columnNames, repairFile, repairHash)) {
|
|
throw new Error("Project environment migration pair does not match the qualified overlap repair");
|
|
}
|
|
await runInTransaction(sql, async () => {
|
|
await sql.unsafe('LOCK TABLE "public"."projects" IN ACCESS EXCLUSIVE MODE');
|
|
await sql.unsafe(`CREATE TEMP TABLE paperclip_project_environment_overlap ON COMMIT DROP AS
|
|
SELECT id, environment_id FROM "public"."projects" WHERE environment_id IS NOT NULL`);
|
|
await applyMigration();
|
|
for (const statement of splitMigrationStatements(repairContent)) await sql.unsafe(statement);
|
|
await sql.unsafe(`UPDATE "public"."projects" AS project SET environment_id = legacy.environment_id
|
|
FROM pg_temp.paperclip_project_environment_overlap AS legacy WHERE project.id = legacy.id`);
|
|
await recordMigrationHistoryEntry(sql, qualifiedTable, columnNames, repairFile, repairHash,
|
|
folderMillisByFileName.get(repairFile) ?? Date.now());
|
|
});
|
|
} else if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
|
|
await applyMigration();
|
|
} else {
|
|
await runInTransaction(sql, applyMigration);
|
|
}
|
|
}
|
|
} finally {
|
|
try {
|
|
await sql`SELECT pg_advisory_unlock(hashtextextended('paperclip:migrations', 0))`;
|
|
} catch {
|
|
// Preserve the migration error if the session died; PostgreSQL releases
|
|
// its lock on disconnect, and pool.end() closes any remaining connection.
|
|
}
|
|
sql.release();
|
|
}
|
|
} finally {
|
|
await pool.end();
|
|
}
|
|
}
|
|
|
|
async function mapHashesToMigrationFiles(migrationFiles: string[]): Promise<Map<string, string>> {
|
|
const mapped = new Map<string, string>();
|
|
|
|
await Promise.all(
|
|
migrationFiles.map(async (migrationFile) => {
|
|
const content = await readMigrationFileContent(migrationFile);
|
|
const hash = createHash("sha256").update(content).digest("hex");
|
|
mapped.set(hash, migrationFile);
|
|
if (migrationFile === "0355_agent_tool_definition_hooks.sql") {
|
|
mapped.set("7ac76adac0220da522c78e51da20490aeb0f90512b4370a05af9f8dd0f75eda6", migrationFile);
|
|
}
|
|
}),
|
|
);
|
|
|
|
return mapped;
|
|
}
|
|
|
|
async function getMigrationTableColumnNames(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationTableSchema: string,
|
|
): Promise<Set<string>> {
|
|
const columns = await sql.unsafe<{ column_name: string }[]>(
|
|
`
|
|
SELECT column_name
|
|
FROM information_schema.columns
|
|
WHERE table_schema = ${quoteLiteral(migrationTableSchema)}
|
|
AND table_name = ${quoteLiteral(DRIZZLE_MIGRATIONS_TABLE)}
|
|
`,
|
|
);
|
|
return new Set(columns.map((column) => column.column_name));
|
|
}
|
|
|
|
async function tableExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM information_schema.tables
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function columnExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
columnName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM information_schema.columns
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
AND column_name = ${columnName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function columnHasDataType(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
columnName: string,
|
|
dataType: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ dataType: string; udtName: string }[]>`
|
|
SELECT data_type AS "dataType", udt_name AS "udtName"
|
|
FROM information_schema.columns
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
AND column_name = ${columnName}
|
|
`;
|
|
const expected = dataType.toLowerCase();
|
|
return rows.some((row) => (
|
|
row.dataType.toLowerCase() === expected || row.udtName.toLowerCase() === expected
|
|
));
|
|
}
|
|
|
|
async function indexExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
indexName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_class c
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public'
|
|
AND c.relkind = 'i'
|
|
AND c.relname = ${indexName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function constraintExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
constraintName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_constraint c
|
|
JOIN pg_namespace n ON n.oid = c.connamespace
|
|
WHERE n.nspname = 'public'
|
|
AND c.conname = ${constraintName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function functionExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
functionName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_proc p
|
|
JOIN pg_namespace n ON n.oid = p.pronamespace
|
|
WHERE n.nspname = 'public'
|
|
AND p.proname = ${functionName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function triggerExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
triggerName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_trigger t
|
|
JOIN pg_class c ON c.oid = t.tgrelid
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public'
|
|
AND t.tgname = ${triggerName}
|
|
AND NOT t.tgisinternal
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function heartbeatEventSequencesAreUnique(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ unique: boolean }[]>`
|
|
SELECT NOT EXISTS (
|
|
SELECT 1
|
|
FROM heartbeat_run_events
|
|
GROUP BY run_id, seq
|
|
HAVING count(*) > 1
|
|
) AS unique
|
|
`;
|
|
return rows[0]?.unique ?? false;
|
|
}
|
|
|
|
async function heartbeatNextEventSequencesAreCurrent(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ current: boolean }[]>`
|
|
SELECT NOT EXISTS (
|
|
SELECT 1
|
|
FROM heartbeat_runs run
|
|
WHERE run.next_event_seq IS DISTINCT FROM COALESCE((
|
|
SELECT max(event.seq) + 1
|
|
FROM heartbeat_run_events event
|
|
WHERE event.run_id = run.id
|
|
), 1)
|
|
) AS current
|
|
`;
|
|
return rows[0]?.current ?? false;
|
|
}
|
|
|
|
async function migrationStatementAlreadyApplied(
|
|
sql: ReturnType<typeof postgres>,
|
|
statement: string,
|
|
): Promise<boolean> {
|
|
const normalized = statement
|
|
.replace(/^\s*--.*$/gm, "")
|
|
.replace(/\s+/g, " ")
|
|
.trim();
|
|
|
|
const createTableMatch = normalized.match(/^CREATE TABLE(?: IF NOT EXISTS)? "([^"]+)"/i);
|
|
if (createTableMatch) {
|
|
return tableExists(sql, createTableMatch[1]);
|
|
}
|
|
|
|
const addColumnMatch = normalized.match(
|
|
/^ALTER TABLE "([^"]+)" ADD COLUMN(?: IF NOT EXISTS)? "([^"]+)"/i,
|
|
);
|
|
if (addColumnMatch) {
|
|
return columnExists(sql, addColumnMatch[1], addColumnMatch[2]);
|
|
}
|
|
|
|
const alterColumnTypeMatch = normalized.match(
|
|
/^ALTER TABLE "([^"]+)" ALTER COLUMN "([^"]+)" SET DATA TYPE ([A-Za-z0-9_]+)/i,
|
|
);
|
|
if (alterColumnTypeMatch) {
|
|
return columnHasDataType(
|
|
sql,
|
|
alterColumnTypeMatch[1],
|
|
alterColumnTypeMatch[2],
|
|
alterColumnTypeMatch[3],
|
|
);
|
|
}
|
|
|
|
const createIndexMatch = normalized.match(/^CREATE (?:UNIQUE )?INDEX(?: IF NOT EXISTS)? "([^"]+)"/i);
|
|
if (createIndexMatch) {
|
|
return indexExists(sql, createIndexMatch[1]);
|
|
}
|
|
|
|
const addConstraintMatch = normalized.match(/^ALTER TABLE "([^"]+)" ADD CONSTRAINT "([^"]+)"/i);
|
|
if (addConstraintMatch) {
|
|
return constraintExists(sql, addConstraintMatch[2]);
|
|
}
|
|
|
|
const createFunctionMatch = normalized.match(
|
|
/^CREATE OR REPLACE FUNCTION "?([A-Za-z_][A-Za-z0-9_]*)"?\s*\(/i,
|
|
);
|
|
if (createFunctionMatch) {
|
|
return functionExists(sql, createFunctionMatch[1]);
|
|
}
|
|
|
|
const createTriggerMatch = normalized.match(
|
|
/^CREATE TRIGGER "?([A-Za-z_][A-Za-z0-9_]*)"?/i,
|
|
);
|
|
if (createTriggerMatch) {
|
|
return triggerExists(sql, createTriggerMatch[1]);
|
|
}
|
|
|
|
// These native-runner repairs have persistent postconditions. Verify them
|
|
// instead of replaying them when a restored database is missing only the
|
|
// migration-history row.
|
|
if (
|
|
normalized.startsWith("WITH ranked AS (")
|
|
&& normalized.includes('UPDATE "heartbeat_run_events" AS event')
|
|
) {
|
|
return heartbeatEventSequencesAreUnique(sql);
|
|
}
|
|
if (
|
|
normalized.startsWith('UPDATE "heartbeat_runs" AS run')
|
|
&& normalized.includes('SET "next_event_seq" = COALESCE')
|
|
) {
|
|
return heartbeatNextEventSequencesAreCurrent(sql);
|
|
}
|
|
|
|
// If we cannot reason about a statement safely, require manual migration.
|
|
return false;
|
|
}
|
|
|
|
async function migrationContentAlreadyApplied(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationContent: string,
|
|
): Promise<boolean> {
|
|
const statements = splitMigrationStatements(migrationContent);
|
|
if (statements.length === 0) return false;
|
|
|
|
for (const statement of statements) {
|
|
const applied = await migrationStatementAlreadyApplied(sql, statement);
|
|
if (!applied) return false;
|
|
}
|
|
|
|
return true;
|
|
}
|
|
|
|
async function loadAppliedMigrations(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationTableSchema: string,
|
|
availableMigrations: string[],
|
|
): Promise<string[]> {
|
|
const quotedSchema = quoteIdentifier(migrationTableSchema);
|
|
const qualifiedTable = `${quotedSchema}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
|
|
if (columnNames.has("name")) {
|
|
const rows = await sql.unsafe<{ name: string }[]>(`SELECT name FROM ${qualifiedTable} ORDER BY id`);
|
|
return rows.map((row) => row.name).filter((name): name is string => Boolean(name));
|
|
}
|
|
|
|
if (columnNames.has("hash")) {
|
|
const rows = await sql.unsafe<{ hash: string }[]>(`SELECT hash FROM ${qualifiedTable} ORDER BY id`);
|
|
const hashesToMigrationFiles = await mapHashesToMigrationFiles(availableMigrations);
|
|
const appliedFromHashes = rows
|
|
.map((row) => hashesToMigrationFiles.get(row.hash))
|
|
.filter((name): name is string => Boolean(name));
|
|
|
|
if (appliedFromHashes.length > 0) {
|
|
// Best-effort: when all hashes resolve, this is authoritative.
|
|
if (appliedFromHashes.length === rows.length) return appliedFromHashes;
|
|
|
|
// Partial hash resolution can happen when files have changed; return what we can trust.
|
|
return appliedFromHashes;
|
|
}
|
|
|
|
// Fallback only when hashes are unavailable/unresolved.
|
|
if (columnNames.has("created_at")) {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
if (journalEntries.length > 0) {
|
|
const lastDbRows = await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC LIMIT 1`,
|
|
);
|
|
const lastCreatedAt = Number(lastDbRows[0]?.created_at ?? -1);
|
|
if (Number.isFinite(lastCreatedAt) && lastCreatedAt >= 0) {
|
|
return journalEntries
|
|
.filter((entry) => availableMigrations.includes(entry.fileName))
|
|
.filter((entry) => entry.folderMillis <= lastCreatedAt)
|
|
.map((entry) => entry.fileName)
|
|
.slice(0, rows.length);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
const rows = await sql.unsafe<{ id: number }[]>(`SELECT id FROM ${qualifiedTable} ORDER BY id`);
|
|
const journalMigrationFiles = await listJournalMigrationFiles();
|
|
const appliedFromIds = rows
|
|
.map((row) => journalMigrationFiles[row.id - 1])
|
|
.filter((name): name is string => Boolean(name));
|
|
if (appliedFromIds.length > 0) return appliedFromIds;
|
|
|
|
return availableMigrations.slice(0, Math.max(0, rows.length));
|
|
}
|
|
|
|
export type MigrationHistoryReconcileResult = {
|
|
repairedMigrations: string[];
|
|
remainingMigrations: string[];
|
|
};
|
|
|
|
export async function reconcilePendingMigrationHistory(
|
|
url: string,
|
|
): Promise<MigrationHistoryReconcileResult> {
|
|
const state = await inspectMigrations(url);
|
|
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
|
|
return { repairedMigrations: [], remainingMigrations: [] };
|
|
}
|
|
|
|
const sql = createUtilitySql(url);
|
|
const repairedMigrations: string[] = [];
|
|
|
|
try {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const folderMillisByFile = new Map(journalEntries.map((entry) => [entry.fileName, entry.folderMillis]));
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
return { repairedMigrations, remainingMigrations: state.pendingMigrations };
|
|
}
|
|
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
|
|
for (const migrationFile of state.pendingMigrations) {
|
|
const migrationContent = await readMigrationFileContent(migrationFile);
|
|
const alreadyApplied = await migrationContentAlreadyApplied(sql, migrationContent);
|
|
if (!alreadyApplied) break;
|
|
|
|
const hash = createHash("sha256").update(migrationContent).digest("hex");
|
|
const folderMillis = folderMillisByFile.get(migrationFile) ?? Date.now();
|
|
const existingByHash = columnNames.has("hash")
|
|
? await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} WHERE hash = ${quoteLiteral(hash)} ORDER BY created_at DESC LIMIT 1`,
|
|
)
|
|
: [];
|
|
const existingByName = columnNames.has("name")
|
|
? await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} WHERE name = ${quoteLiteral(migrationFile)} ORDER BY created_at DESC LIMIT 1`,
|
|
)
|
|
: [];
|
|
if (existingByHash.length > 0 || existingByName.length > 0) {
|
|
if (columnNames.has("created_at")) {
|
|
const existingHashCreatedAt = Number(existingByHash[0]?.created_at ?? -1);
|
|
if (existingByHash.length > 0 && Number.isFinite(existingHashCreatedAt) && existingHashCreatedAt < folderMillis) {
|
|
await sql.unsafe(
|
|
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE hash = ${quoteLiteral(hash)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
|
|
);
|
|
}
|
|
|
|
const existingNameCreatedAt = Number(existingByName[0]?.created_at ?? -1);
|
|
if (existingByName.length > 0 && Number.isFinite(existingNameCreatedAt) && existingNameCreatedAt < folderMillis) {
|
|
await sql.unsafe(
|
|
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE name = ${quoteLiteral(migrationFile)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
repairedMigrations.push(migrationFile);
|
|
continue;
|
|
}
|
|
|
|
const insertColumns: string[] = [];
|
|
const insertValues: string[] = [];
|
|
|
|
if (columnNames.has("hash")) {
|
|
insertColumns.push(quoteIdentifier("hash"));
|
|
insertValues.push(quoteLiteral(hash));
|
|
}
|
|
if (columnNames.has("name")) {
|
|
insertColumns.push(quoteIdentifier("name"));
|
|
insertValues.push(quoteLiteral(migrationFile));
|
|
}
|
|
if (columnNames.has("created_at")) {
|
|
insertColumns.push(quoteIdentifier("created_at"));
|
|
insertValues.push(quoteLiteral(String(folderMillis)));
|
|
}
|
|
|
|
if (insertColumns.length === 0) break;
|
|
|
|
await sql.unsafe(
|
|
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
|
|
);
|
|
repairedMigrations.push(migrationFile);
|
|
}
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
|
|
const refreshed = await inspectMigrations(url);
|
|
return {
|
|
repairedMigrations,
|
|
remainingMigrations:
|
|
refreshed.status === "needsMigrations" ? refreshed.pendingMigrations : [],
|
|
};
|
|
}
|
|
|
|
async function discoverMigrationTableSchema(sql: ReturnType<typeof postgres>): Promise<string | null> {
|
|
const rows = await sql<{ schemaName: string }[]>`
|
|
SELECT n.nspname AS "schemaName"
|
|
FROM pg_class c
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE c.relname = ${DRIZZLE_MIGRATIONS_TABLE} AND c.relkind = 'r'
|
|
`;
|
|
|
|
if (rows.length === 0) return null;
|
|
|
|
const drizzleSchema = rows.find(({ schemaName }) => schemaName === "drizzle");
|
|
if (drizzleSchema) return drizzleSchema.schemaName;
|
|
|
|
const publicSchema = rows.find(({ schemaName }) => schemaName === "public");
|
|
if (publicSchema) return publicSchema.schemaName;
|
|
|
|
return rows[0]?.schemaName ?? null;
|
|
}
|
|
|
|
export async function inspectMigrations(url: string): Promise<MigrationState> {
|
|
const sql = createUtilitySql(url);
|
|
|
|
try {
|
|
const availableMigrations = await listMigrationFiles();
|
|
const tableCountResult = await sql<{ count: number }[]>`
|
|
select count(*)::int as count
|
|
from information_schema.tables
|
|
where table_schema = 'public'
|
|
and table_type = 'BASE TABLE'
|
|
`;
|
|
const tableCount = tableCountResult[0]?.count ?? 0;
|
|
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
if (tableCount > 0) {
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations: [],
|
|
pendingMigrations: availableMigrations,
|
|
journalEntryCount: 0,
|
|
reason: "no-migration-journal-non-empty-db",
|
|
};
|
|
}
|
|
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations: [],
|
|
pendingMigrations: availableMigrations,
|
|
journalEntryCount: 0,
|
|
reason: "no-migration-journal-empty-db",
|
|
};
|
|
}
|
|
|
|
const qualifiedMigrationTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
const journalCountRows = await sql.unsafe<{ count: number }[]>(
|
|
`SELECT count(*)::int AS count FROM ${qualifiedMigrationTable}`,
|
|
);
|
|
const journalEntryCount = journalCountRows[0]?.count ?? 0;
|
|
const appliedMigrations = await loadAppliedMigrations(sql, migrationTableSchema, availableMigrations);
|
|
const pendingMigrations = availableMigrations.filter((name) => !appliedMigrations.includes(name));
|
|
if (pendingMigrations.length === 0) {
|
|
return {
|
|
status: "upToDate",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations,
|
|
journalEntryCount,
|
|
};
|
|
}
|
|
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations,
|
|
pendingMigrations,
|
|
journalEntryCount,
|
|
reason: "pending-migrations",
|
|
};
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function applyPendingMigrations(url: string): Promise<void> {
|
|
const initialState = await inspectMigrations(url);
|
|
if (initialState.status === "upToDate") return;
|
|
|
|
if (initialState.reason === "no-migration-journal-empty-db") {
|
|
// Use the same per-file executor for bootstrap and upgrades: Drizzle wraps
|
|
// the whole journal in one transaction and cannot run committing backfills.
|
|
await applyPendingMigrationsManually(url, initialState.pendingMigrations);
|
|
const bootstrappedState = await inspectMigrations(url);
|
|
if (bootstrappedState.status === "upToDate") return;
|
|
throw new Error(
|
|
`Failed to bootstrap migrations: ${bootstrappedState.pendingMigrations.join(", ")}`,
|
|
);
|
|
}
|
|
|
|
if (initialState.reason === "no-migration-journal-non-empty-db") {
|
|
throw new Error(
|
|
"Database has tables but no migration journal; automatic migration is unsafe. Initialize migration history manually.",
|
|
);
|
|
}
|
|
|
|
let state = await inspectMigrations(url);
|
|
if (state.status === "upToDate") return;
|
|
|
|
const repair = await reconcilePendingMigrationHistory(url);
|
|
if (repair.repairedMigrations.length > 0) {
|
|
state = await inspectMigrations(url);
|
|
if (state.status === "upToDate") return;
|
|
}
|
|
|
|
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
|
|
throw new Error("Migrations are still pending after migration-history reconciliation; run inspectMigrations for details.");
|
|
}
|
|
|
|
await applyPendingMigrationsManually(url, state.pendingMigrations);
|
|
|
|
const finalState = await inspectMigrations(url);
|
|
if (finalState.status !== "upToDate") {
|
|
throw new Error(
|
|
`Failed to apply pending migrations: ${finalState.pendingMigrations.join(", ")}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
export type MigrationBootstrapResult =
|
|
| { migrated: true; reason: "migrated-empty-db"; tableCount: 0 }
|
|
| { migrated: false; reason: "already-migrated"; tableCount: number }
|
|
| { migrated: false; reason: "not-empty-no-migration-journal"; tableCount: number };
|
|
|
|
export async function migratePostgresIfEmpty(url: string): Promise<MigrationBootstrapResult> {
|
|
const sql = createUtilitySql(url);
|
|
|
|
try {
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
|
|
const tableCountResult = await sql<{ count: number }[]>`
|
|
select count(*)::int as count
|
|
from information_schema.tables
|
|
where table_schema = 'public'
|
|
and table_type = 'BASE TABLE'
|
|
`;
|
|
|
|
const tableCount = tableCountResult[0]?.count ?? 0;
|
|
|
|
if (migrationTableSchema) {
|
|
return { migrated: false, reason: "already-migrated", tableCount };
|
|
}
|
|
|
|
if (tableCount > 0) {
|
|
return { migrated: false, reason: "not-empty-no-migration-journal", tableCount };
|
|
}
|
|
|
|
await applyPendingMigrationsManually(url, await listMigrationFiles());
|
|
|
|
return { migrated: true, reason: "migrated-empty-db", tableCount: 0 };
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function ensurePostgresDatabase(
|
|
url: string,
|
|
databaseName: string,
|
|
): Promise<"created" | "exists"> {
|
|
if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(databaseName)) {
|
|
throw new Error(`Unsafe database name: ${databaseName}`);
|
|
}
|
|
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
const existing = await sql<{ one: number }[]>`
|
|
select 1 as one from pg_database where datname = ${databaseName} limit 1
|
|
`;
|
|
if (existing.length > 0) return "exists";
|
|
|
|
await sql.unsafe(`create database "${databaseName}" encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
|
|
return "created";
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function resetPostgresDatabase(
|
|
url: string,
|
|
databaseName: string,
|
|
): Promise<"reset"> {
|
|
const quotedDatabaseName = quoteIdentifier(databaseName);
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
await sql`
|
|
select pg_terminate_backend(pid)
|
|
from pg_stat_activity
|
|
where datname = ${databaseName}
|
|
and pid <> pg_backend_pid()
|
|
`;
|
|
await sql.unsafe(`drop database if exists ${quotedDatabaseName}`);
|
|
await sql.unsafe(`create database ${quotedDatabaseName} encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
|
|
return "reset";
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export type Db = ReturnType<typeof createDb>;
|