mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts each agent's queued runs, up to the agent's `maxConcurrentRuns` > - #21234 starts person-initiated runs before system recovery runs, and it reads "person-initiated" from the run's own wake request > - When a person comments on a task that already has a queued system run, the comment joins that run. The run keeps its system wake, so dispatch ranks the person's message as system work > - Dispatch is event driven. The queued-run re-drive backstop covers only agents with no live run, so an agent with several slots keeps a free slot empty until one of its runs finishes, and runs can take hours > - This pull request makes dispatch read the person wakes that joined each queued run, and makes the re-drive backstop cover an agent that still has a free slot > - The benefit is that a person's comment starts first, and free slots fill within about a minute instead of waiting for an unrelated run to finish ## Linked Issues or Issue Description Refs #21234, #20986 **What happened?** On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76 queued runs at 03:12Z. No run started on that agent from 03:01:33Z to 03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least 2 slots were free. Other agents in the same company started runs during that time. Runs started again only after the next deploy restarted the server. Five queued runs came from a person: comments at 02:43Z, 02:44Z and 02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had the wake reasons `issue_reopened_via_comment` and `issue_commented`, the invocation source `automation`, and no person on the run's own wake request. **Expected behavior** A run that carries a person's comment ranks as person-initiated, also when the comment joined an existing queued run. When an agent has a free slot and a queued run is ready, the run starts promptly, without waiting for another run on the agent to finish. **Steps to reproduce** 1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it. 2. Queue a third run while no dispatch signal reaches the queue (for example, a claim declines it while a gate is still closed, and the gate then clears). 3. See that `listUndispatchedQueuedRuns` returns nothing, because the agent has a live run. The run stays queued until one of the 2 running runs finishes. 4. Separately, queue a system run on a task, then post a person's comment on the task so that the comment wake coalesces into that queued run. Queue older system runs on other tasks of the same single-slot agent. 5. See that dispatch starts the older system runs first. **Root cause** 1. `coalesceIntoActiveExecutionRun` (`server/src/modules/wake-queue/adapters/postgres.ts`) merges the person's comment context into the queued run and records the person's wake as a separate `coalesced` row with `runId` set to that run. The run's own `wakeupRequestId` still points to the system wake. `queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`) only read the wake behind `wakeupRequestId`. The task routes do record the comment author as the requester (`requestedByActorType: "user"`). The person was lost only at dispatch time. 2. `listUndispatchedQueuedRunPage` (`server/src/services/queued-run-dispatch-liveness.ts`) skipped every agent with any `running` or `scheduled_retry` run. That rule assumes one slot per agent. For an agent with several slots, a lost or declined dispatch is not re-driven until a run on that agent finishes. This change is not in `claimQueuedRun`, the fresh-isolated preflight, the wake-queue promoter or `fresh-continuation-fallback.ts`. **Paperclip version or commit** Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is based on master `6cb0dd0c6b`. **Deployment mode** Authenticated, public, Paperclip Cloud managed staging. **Agent adapter(s) involved** `paperclip_runner` (Codex native runtime). The fix is adapter-independent. ## What Changed - `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is a queued run of the agent (by `run_id IN (queued run ids)`, so an earlier saved comment that a run adopted is included). A run with such a wake ranks as person-initiated and is never counted as system recovery. - `listUndispatchedQueuedRuns` now returns a queued run when its agent has fewer `running` plus `scheduled_retry` runs than its `maxConcurrentRuns`, and its task has no `running` or `scheduled_retry` run. The re-drive backstop and the ownership I7 check use this query. - The query counts each agent's live runs once, in one grouped subquery that is joined to the queued runs. - Migration 0476 adds `agent_wakeup_requests_company_run_idx` (`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as `online_compatible`. `doc/DATABASE.md` describes it. - Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same limit as `parseHeartbeatPolicy`, including the single-run runner providers, which now share one constant. ## Verification - Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a queued run that a person's comment joined ahead of older system work". It fails on master (`expected null not to be null`: the older system run started first) and passes with this change. - Red/green: `server/src/__tests__/ownership-reconciliation-report.test.ts` "re-drives a queued run while its agent still has a free slot, and not once every slot is running". It fails on master (`expected [] to deeply equal [ Array(1) ]`) and passes with this change. - Red/green: "starts a queued run that adopted a person's comment saved before every queued run". It fails with a `requestedAt` bound and passes with the `run_id` lookup. - Red/green: "counts an agent's live runs in one grouped aggregate, not once per queued run" runs EXPLAIN on the page query. It fails with a per-row correlated count and passes with the grouped subquery. - New: "uses the agent's single-run limit for a runner provider that runs one turn at a time". - Two existing tests now set `maxConcurrentRuns: 1` so that they keep testing "the agent's only slot is busy". - `npx vitest run src/__tests__/recovery-wake-herd.test.ts src/services/heartbeat/queued-run-dispatch-order.test.ts src/__tests__/ownership-reconciliation-report.test.ts src/services/queued-run-dispatch-liveness.test.ts src/services/heartbeat/dispatch-work.test.ts`: 71 passed. - `npx tsc --noEmit -p server/tsconfig.json`: no errors. ## Risks - The re-drive backstop now also covers multi-slot agents with live runs. Each queued run is re-driven at most on its existing backoff (1 minute, then doubling to 15 minutes). A queued run whose task already has a live run is not re-driven. - The ownership I7 report can now list a queued run on a multi-slot agent that has a free slot. This is the stall that this change targets. - One more indexed read per dispatch pass (`agent_wakeup_requests` by company and run id). The new index is built concurrently, so the previous release keeps writing during the build. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, read-only staging API reads in a browser tab). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1462 lines
55 KiB
TypeScript
1462 lines
55 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { readFile, readdir } from "node:fs/promises";
|
|
import { fileURLToPath } from "node:url";
|
|
import postgres from "postgres";
|
|
import { installDatabaseWorkSignals } from "./work-signals.js";
|
|
import { installDatabaseResponseCapture } from "./response-capture.js";
|
|
import { installPreparedQueries } from "./prepared-queries.js";
|
|
import { drizzleWithLazyTransactionQueries } from "./lazy-transaction-queries.js";
|
|
import * as schema from "./schema/index.js";
|
|
|
|
const MIGRATIONS_FOLDER = fileURLToPath(new URL("./migrations", import.meta.url));
|
|
const DRIZZLE_MIGRATIONS_TABLE = "__drizzle_migrations";
|
|
const MIGRATIONS_JOURNAL_JSON = fileURLToPath(new URL("./migrations/meta/_journal.json", import.meta.url));
|
|
const PROJECT_ENVIRONMENT_OVERLAP = {
|
|
dropFile: "0466_drop_project_environment.sql",
|
|
dropSha256: "07ca1c666c6471fdb9ce7a0a5fb31bf1a074138466fa8daae50b53199b8e0d7c",
|
|
repairFile: "0467_fair_human_torch.sql",
|
|
repairSha256: "50ce0eb2567570b098bea1de92fe6a220d139e4ba83438195fbbad1be51e31a1",
|
|
} as const;
|
|
|
|
function createUtilitySql(url: string) {
|
|
return postgres(url, { max: 1, onnotice: () => {} });
|
|
}
|
|
|
|
type RegisteredPostgresClient = ReturnType<typeof postgres>;
|
|
|
|
/**
|
|
* Derives a registry key from a connection URL's host and port only. We must
|
|
* not retain or log the full URL, because it carries credentials.
|
|
*/
|
|
function hostPortKey(url: string): string {
|
|
const parsed = new URL(url);
|
|
return `${parsed.hostname}:${parsed.port || "5432"}`;
|
|
}
|
|
|
|
/**
|
|
* Same as `hostPortKey`, but returns `null` instead of throwing when the URL
|
|
* does not parse. `postgres(url)` tolerates a value `new URL()` rejects (an
|
|
* empty string falls back to the `PG*` environment variables), so `createDb`
|
|
* must tolerate it too: skip the registry entry and let the driver decide
|
|
* the outcome, instead of throwing an error the driver itself would not.
|
|
*/
|
|
function hostPortKeyOrNull(url: string): string | null {
|
|
try {
|
|
return hostPortKey(url);
|
|
} catch (error) {
|
|
if (error instanceof TypeError && (error as NodeJS.ErrnoException).code === "ERR_INVALID_URL") return null;
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
// Tracks every client `createDb` hands out, keyed by host and port, so a test
|
|
// fixture can end them before it stops the Postgres cluster they point at. A
|
|
// `WeakRef` plus `FinalizationRegistry` means a long-lived process (a real
|
|
// server) retains nothing extra: an unreferenced client is pruned on its own.
|
|
const clientsByHostPort = new Map<string, Set<WeakRef<RegisteredPostgresClient>>>();
|
|
const clientFinalizer = new FinalizationRegistry<{ hostPortKey: string; ref: WeakRef<RegisteredPostgresClient> }>(
|
|
({ hostPortKey, ref }) => {
|
|
const refs = clientsByHostPort.get(hostPortKey);
|
|
if (!refs) return;
|
|
refs.delete(ref);
|
|
if (refs.size === 0) clientsByHostPort.delete(hostPortKey);
|
|
},
|
|
);
|
|
|
|
function registerClient(key: string, client: RegisteredPostgresClient): void {
|
|
const ref = new WeakRef(client);
|
|
let refs = clientsByHostPort.get(key);
|
|
if (!refs) {
|
|
refs = new Set();
|
|
clientsByHostPort.set(key, refs);
|
|
}
|
|
refs.add(ref);
|
|
clientFinalizer.register(client, { hostPortKey: key, ref }, ref);
|
|
}
|
|
|
|
/**
|
|
* Ends every live client `createDb` handed out for the given URL's host and
|
|
* port, then forgets them. Call this before stopping a Postgres cluster: a
|
|
* client that outlives the cluster it points at can crash the process (a
|
|
* reserved connection's deferred write firing after the socket is gone).
|
|
* Swallows individual `end()` errors so one bad client cannot block the rest.
|
|
* An owned database on a shared test host must select database scope instead;
|
|
* its teardown cannot close another fixture's clients on that host.
|
|
*/
|
|
export async function closeRegisteredClients(url: string, options?: { scope: "database" }): Promise<void> {
|
|
const key = hostPortKeyOrNull(url);
|
|
if (key === null) return;
|
|
const database = options?.scope === "database" ? decodeURIComponent(new URL(url).pathname.slice(1)) : undefined;
|
|
if (database === "") throw new Error("Database-scoped teardown requires an explicit database name");
|
|
const refs = clientsByHostPort.get(key);
|
|
if (!refs) return;
|
|
|
|
const clients: RegisteredPostgresClient[] = [];
|
|
for (const ref of refs) {
|
|
const client = ref.deref();
|
|
if (client && database !== undefined && client.options.database !== database) continue;
|
|
refs.delete(ref);
|
|
clientFinalizer.unregister(ref);
|
|
if (client) clients.push(client);
|
|
}
|
|
if (refs.size === 0) clientsByHostPort.delete(key);
|
|
|
|
await Promise.all(clients.map((client) => client.end({ timeout: 1 }).catch(() => {})));
|
|
}
|
|
|
|
function isSafeIdentifier(value: string): boolean {
|
|
return /^[A-Za-z_][A-Za-z0-9_]*$/.test(value);
|
|
}
|
|
|
|
function quoteIdentifier(value: string): string {
|
|
if (!isSafeIdentifier(value)) throw new Error(`Unsafe SQL identifier: ${value}`);
|
|
return `"${value.replaceAll("\"", "\"\"")}"`;
|
|
}
|
|
|
|
function quoteLiteral(value: string): string {
|
|
return `'${value.replaceAll("'", "''")}'`;
|
|
}
|
|
|
|
function splitMigrationStatements(content: string): string[] {
|
|
return content
|
|
.split("--> statement-breakpoint")
|
|
.map((statement) => statement.trim())
|
|
.filter((statement) => statement.length > 0);
|
|
}
|
|
|
|
export type MigrationState =
|
|
| {
|
|
status: "upToDate";
|
|
tableCount: number;
|
|
availableMigrations: string[];
|
|
appliedMigrations: string[];
|
|
journalEntryCount: number;
|
|
}
|
|
| {
|
|
status: "needsMigrations";
|
|
tableCount: number;
|
|
availableMigrations: string[];
|
|
appliedMigrations: string[];
|
|
pendingMigrations: string[];
|
|
journalEntryCount: number;
|
|
reason: "no-migration-journal-empty-db" | "no-migration-journal-non-empty-db" | "pending-migrations";
|
|
};
|
|
|
|
export interface DatabaseClientOptions {
|
|
prepare?: boolean;
|
|
/** postgres.js `max` — connection pool size (driver default: 10). */
|
|
maxConnections?: number;
|
|
/** postgres.js `idle_timeout` in seconds (driver default: disabled). */
|
|
idleTimeoutSeconds?: number;
|
|
/** postgres.js `connect_timeout` in seconds (driver default: 30). */
|
|
connectTimeoutSeconds?: number;
|
|
/**
|
|
* postgres.js `max_lifetime` in seconds. Bounds how long one pooled
|
|
* connection is reused before the client replaces it (driver default: a
|
|
* random value between 30 and 60 minutes).
|
|
*/
|
|
maxLifetimeSeconds?: number;
|
|
/**
|
|
* postgres.js `connection.application_name`, shown in
|
|
* `pg_stat_activity.application_name`. Lets an operator tell Paperclip's
|
|
* pool apart from other clients of the same database (driver default:
|
|
* `postgres.js`).
|
|
*/
|
|
applicationName?: string;
|
|
/**
|
|
* Connection-level `statement_timeout` in milliseconds, sent as a startup
|
|
* parameter (no extra round trip). For dedicated single-purpose pools only.
|
|
*/
|
|
statementTimeoutMs?: number;
|
|
}
|
|
|
|
/**
|
|
* Idle pooled connections close after this many seconds unless
|
|
* `DATABASE_IDLE_TIMEOUT_SECONDS` says otherwise. The driver default keeps an
|
|
* idle connection open forever, so a process that stops issuing queries still
|
|
* holds every backend it ever opened. Set `DATABASE_IDLE_TIMEOUT_SECONDS=0`
|
|
* to restore the driver default.
|
|
*/
|
|
export const DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS = 60;
|
|
/** `application_name` reported to PostgreSQL unless `DATABASE_APPLICATION_NAME` overrides it. */
|
|
export const DEFAULT_DATABASE_APPLICATION_NAME = "paperclip";
|
|
|
|
function envBoolean(env: NodeJS.ProcessEnv, name: string): boolean | undefined {
|
|
const value = env[name]?.trim().toLowerCase();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (value === "true" || value === "1") return true;
|
|
if (value === "false" || value === "0") return false;
|
|
throw new Error(`${name} must be "true" or "false", got: ${env[name]}`);
|
|
}
|
|
|
|
function envPositiveInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (!/^[1-9]\d*$/.test(value)) {
|
|
throw new Error(`${name} must be a positive integer, got: ${env[name]}`);
|
|
}
|
|
return Number.parseInt(value, 10);
|
|
}
|
|
|
|
function envNonNegativeInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (!/^(?:0|[1-9]\d*)$/.test(value)) {
|
|
throw new Error(`${name} must be a non-negative integer, got: ${env[name]}`);
|
|
}
|
|
return Number.parseInt(value, 10);
|
|
}
|
|
|
|
function envNonEmptyString(env: NodeJS.ProcessEnv, name: string): string | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
return value;
|
|
}
|
|
|
|
/**
|
|
* Database client tuning from the environment, so hosted deployments can
|
|
* adapt to their connection topology (pooled endpoints, network latency)
|
|
* without editing source. Every variable is optional. This function returns
|
|
* only the values the environment sets; `resolveDatabaseClientOptions` adds
|
|
* Paperclip's own defaults on top, and the driver defaults apply to the rest
|
|
* — self-hosted setups need none of these.
|
|
*/
|
|
export function databaseClientOptionsFromEnv(env: NodeJS.ProcessEnv = process.env): DatabaseClientOptions {
|
|
const options: DatabaseClientOptions = {};
|
|
const prepare = envBoolean(env, "DATABASE_PREPARED_STATEMENTS");
|
|
if (prepare !== undefined) options.prepare = prepare;
|
|
const maxConnections = envPositiveInteger(env, "DATABASE_POOL_MAX");
|
|
if (maxConnections !== undefined) options.maxConnections = maxConnections;
|
|
// `0` is allowed here: it disables idle reaping (the driver default).
|
|
const idleTimeoutSeconds = envNonNegativeInteger(env, "DATABASE_IDLE_TIMEOUT_SECONDS");
|
|
if (idleTimeoutSeconds !== undefined) options.idleTimeoutSeconds = idleTimeoutSeconds;
|
|
const connectTimeoutSeconds = envPositiveInteger(env, "DATABASE_CONNECT_TIMEOUT_SECONDS");
|
|
if (connectTimeoutSeconds !== undefined) options.connectTimeoutSeconds = connectTimeoutSeconds;
|
|
const maxLifetimeSeconds = envPositiveInteger(env, "DATABASE_MAX_LIFETIME_SECONDS");
|
|
if (maxLifetimeSeconds !== undefined) options.maxLifetimeSeconds = maxLifetimeSeconds;
|
|
const applicationName = envNonEmptyString(env, "DATABASE_APPLICATION_NAME");
|
|
if (applicationName !== undefined) options.applicationName = applicationName;
|
|
return options;
|
|
}
|
|
|
|
/**
|
|
* Fills in Paperclip's defaults for the options the caller left unset: idle
|
|
* connections are reaped after `DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS`, and the
|
|
* pool identifies itself as `DEFAULT_DATABASE_APPLICATION_NAME`. Everything
|
|
* else stays at the driver default. An explicit value (including
|
|
* `idleTimeoutSeconds: 0`) always wins over the default.
|
|
*/
|
|
export function resolveDatabaseClientOptions(options: DatabaseClientOptions): DatabaseClientOptions {
|
|
return {
|
|
...options,
|
|
idleTimeoutSeconds: options.idleTimeoutSeconds ?? DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS,
|
|
applicationName: options.applicationName ?? DEFAULT_DATABASE_APPLICATION_NAME,
|
|
};
|
|
}
|
|
|
|
export function postgresJsOptions(options: DatabaseClientOptions): Record<string, unknown> {
|
|
const driverOptions: Record<string, unknown> = {};
|
|
if (options.prepare !== undefined) driverOptions.prepare = options.prepare;
|
|
if (options.maxConnections !== undefined) driverOptions.max = options.maxConnections;
|
|
if (options.idleTimeoutSeconds !== undefined) driverOptions.idle_timeout = options.idleTimeoutSeconds;
|
|
if (options.connectTimeoutSeconds !== undefined) driverOptions.connect_timeout = options.connectTimeoutSeconds;
|
|
if (options.maxLifetimeSeconds !== undefined) driverOptions.max_lifetime = options.maxLifetimeSeconds;
|
|
const connection: Record<string, string> = {};
|
|
if (options.applicationName !== undefined) connection.application_name = options.applicationName;
|
|
if (options.statementTimeoutMs !== undefined) connection.statement_timeout = String(options.statementTimeoutMs);
|
|
if (Object.keys(connection).length) driverOptions.connection = connection;
|
|
return driverOptions;
|
|
}
|
|
|
|
// A long advisory-lock transaction must not borrow the normal pool that its
|
|
// work needs for progress writes. Keep connection configuration private to the
|
|
// originating Db lifetime; callers receive neither URLs nor credentials.
|
|
const dedicatedDbFactories = new WeakMap<object, () => Db>();
|
|
const ownedDatabaseListenerFactories = new WeakMap<object, () => Db>();
|
|
|
|
export interface OwnedDatabaseListener {
|
|
listen: Db["$client"]["listen"];
|
|
close(): Promise<void>;
|
|
}
|
|
|
|
export function createOwnedDatabaseListener(db: Db): OwnedDatabaseListener {
|
|
const factory = ownedDatabaseListenerFactories.get(db);
|
|
if (!factory) throw new Error("owned_listener_requires_create_db");
|
|
const owned = factory();
|
|
return {
|
|
listen: owned.$client.listen.bind(owned.$client),
|
|
close: () => owned.$client.end({ timeout: 1 }),
|
|
};
|
|
}
|
|
|
|
export async function withDedicatedDbConnection<T>(db: Db, action: (dedicated: Db) => Promise<T>): Promise<T> {
|
|
const factory = dedicatedDbFactories.get(db);
|
|
if (!factory) throw new Error("dedicated_connection_requires_create_db");
|
|
const dedicated = factory();
|
|
try { return await action(dedicated); }
|
|
finally { await dedicated.$client.end({ timeout: 1 }); }
|
|
}
|
|
|
|
// postgres.js pipelines a query onto a busy connection when none is idle, so
|
|
// a short liveness write can wait behind an unrelated slow statement with no
|
|
// client-side bound. Liveness writes (controller-lease renewal) get a small
|
|
// private pool, created on first use and shared for the owner's lifetime.
|
|
const livenessDbFactories = new WeakMap<object, () => Db>();
|
|
const livenessDbs = new WeakMap<object, Db>();
|
|
const LIVENESS_POOL_MAX_CONNECTIONS = 2;
|
|
|
|
/** The private liveness pool for a `createDb` client; other clients (test
|
|
* doubles, transactions) are returned unchanged. */
|
|
export function livenessDb(db: Db): Db {
|
|
const existing = livenessDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = livenessDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory();
|
|
livenessDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
const controllerLeaseDbFactories = new WeakMap<object, () => Db>();
|
|
const controllerLeaseDbs = new WeakMap<object, Db>();
|
|
|
|
export function controllerLeaseDb(db: Db): Db {
|
|
const existing = controllerLeaseDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = controllerLeaseDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory();
|
|
controllerLeaseDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
const controllerInstanceLeaseDbFactories = new WeakMap<object, (statementTimeoutMs: number | null) => Db>();
|
|
const controllerInstanceLeaseDbs = new WeakMap<object, Db>();
|
|
|
|
/** One private connection for the server's controller-instance lease, so its
|
|
* renewal never waits behind request or worker traffic on the main pool.
|
|
* `statementTimeoutMs` (first call wins) is a connection-level
|
|
* `statement_timeout`: the server cancels a slow renewal, so it never holds the
|
|
* single connection that the next renewal needs, at no extra round trip. */
|
|
export function controllerInstanceLeaseDb(db: Db, options: { statementTimeoutMs?: number | null } = {}): Db {
|
|
const existing = controllerInstanceLeaseDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = controllerInstanceLeaseDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory(options.statementTimeoutMs ?? null);
|
|
controllerInstanceLeaseDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
/** Replaces the lease connection with one that sends no `statement_timeout`
|
|
* startup parameter, for a connection pooler that refuses it. */
|
|
export function controllerInstanceLeaseDbWithoutStatementTimeout(db: Db): Db {
|
|
const factory = controllerInstanceLeaseDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const previous = controllerInstanceLeaseDbs.get(db);
|
|
const created = factory(null);
|
|
controllerInstanceLeaseDbs.set(db, created);
|
|
void previous?.$client.end({ timeout: 0 }).catch(() => undefined);
|
|
return created;
|
|
}
|
|
|
|
/** Ends the liveness pool if one was created. The ended pool stays cached, so
|
|
* a late renewal fails like a query on the ended primary pool. */
|
|
export async function closeLivenessDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
await Promise.all([livenessDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
|
|
controllerLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
|
|
controllerInstanceLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds })]);
|
|
}
|
|
|
|
// Control checkpoints must not occupy the request or lease-renewal pools
|
|
// during an aligned burst. Share at most three lazy connections per owner.
|
|
const checkpointDbFactories = new WeakMap<object, () => Db>();
|
|
const checkpointDbs = new WeakMap<object, Db>();
|
|
const CHECKPOINT_POOL_MAX_CONNECTIONS = 3;
|
|
const closedCheckpointOwners = new WeakSet<object>();
|
|
|
|
/** The bounded controller checkpoint pool; transactions and test doubles keep
|
|
* their existing database when they have no owning client factory. */
|
|
export function checkpointDb(db: Db): Db {
|
|
const existing = checkpointDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = checkpointDbFactories.get(db);
|
|
if (!factory) return db;
|
|
if (closedCheckpointOwners.has(db)) throw new Error("checkpoint_pool_closed");
|
|
const created = factory();
|
|
checkpointDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
/** Fence lazy creation immediately, and keep any ended pool cached. */
|
|
export async function closeCheckpointDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
closedCheckpointOwners.add(db);
|
|
await checkpointDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
|
|
}
|
|
|
|
// Managed file publication holds an agent lock while transferring bytes. A
|
|
// stalled object store must not consume request or controller-renewal capacity.
|
|
const fileStorageDbFactories = new WeakMap<object, () => Db>();
|
|
const fileStorageDbs = new WeakMap<object, Db>();
|
|
const closedFileStorageOwners = new WeakSet<object>();
|
|
export function fileStorageDb(db: Db): Db {
|
|
const existing = fileStorageDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = fileStorageDbFactories.get(db);
|
|
if (!factory) return db; // Preserve the caller's transaction/savepoint.
|
|
if (closedFileStorageOwners.has(db)) throw new Error("file_storage_pool_closed");
|
|
const created = factory();
|
|
fileStorageDbs.set(db, created);
|
|
fileStorageDbs.set(created, created);
|
|
return created;
|
|
}
|
|
export async function closeFileStorageDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
closedFileStorageOwners.add(db);
|
|
await fileStorageDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
|
|
}
|
|
|
|
export function createDb(url: string, options?: DatabaseClientOptions) {
|
|
const resolved = resolveDatabaseClientOptions(options ?? databaseClientOptionsFromEnv());
|
|
return createDbClient(url, resolved);
|
|
}
|
|
|
|
function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: object) {
|
|
const sql = postgres(url, postgresJsOptions(resolved));
|
|
if (resolved.prepare === true) installPreparedQueries(sql);
|
|
const key = hostPortKeyOrNull(url);
|
|
if (key) registerClient(key, sql);
|
|
// A disconnect can lose the response after a statement has committed.
|
|
// postgres.js calls that error "write CONNECTION_CLOSED" too, so the
|
|
// message cannot establish that replay is safe. Leave retries to callers
|
|
// that know the complete operation is idempotent.
|
|
const db = installDatabaseWorkSignals(drizzleWithLazyTransactionQueries(sql, schema), owner);
|
|
installDatabaseResponseCapture(db, schema);
|
|
dedicatedDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: 1, applicationName: "paperclip-workspace-finalization-lock",
|
|
}, db));
|
|
ownedDatabaseListenerFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: 1,
|
|
}, db));
|
|
livenessDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: LIVENESS_POOL_MAX_CONNECTIONS, applicationName: "paperclip-controller-lease",
|
|
}, db));
|
|
controllerLeaseDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: 1, applicationName: "paperclip-legacy-controller-lease",
|
|
}, db));
|
|
controllerInstanceLeaseDbFactories.set(db, statementTimeoutMs => createDbClient(url, {
|
|
...resolved, maxConnections: 1, applicationName: "paperclip-controller-instance-lease",
|
|
...(statementTimeoutMs !== null ? { statementTimeoutMs } : {}),
|
|
}, db));
|
|
checkpointDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: CHECKPOINT_POOL_MAX_CONNECTIONS, applicationName: "paperclip-runner-checkpoint",
|
|
}, db));
|
|
fileStorageDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: 2, applicationName: "paperclip-file-storage",
|
|
}, db));
|
|
return db;
|
|
}
|
|
|
|
export async function getPostgresDataDirectory(url: string): Promise<string | null> {
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
const rows = await sql<{ data_directory: string | null }[]>`
|
|
SELECT current_setting('data_directory', true) AS data_directory
|
|
`;
|
|
const actual = rows[0]?.data_directory;
|
|
return typeof actual === "string" && actual.length > 0 ? actual : null;
|
|
} catch {
|
|
return null;
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
async function listMigrationFiles(): Promise<string[]> {
|
|
const entries = await readdir(MIGRATIONS_FOLDER, { withFileTypes: true });
|
|
return entries
|
|
.filter((entry) => entry.isFile() && entry.name.endsWith(".sql"))
|
|
.map((entry) => entry.name)
|
|
.sort((a, b) => a.localeCompare(b));
|
|
}
|
|
|
|
type MigrationJournalFile = {
|
|
entries?: Array<{ idx?: number; tag?: string; when?: number }>;
|
|
};
|
|
|
|
type JournalMigrationEntry = {
|
|
fileName: string;
|
|
folderMillis: number;
|
|
order: number;
|
|
};
|
|
|
|
async function listJournalMigrationEntries(): Promise<JournalMigrationEntry[]> {
|
|
try {
|
|
const raw = await readFile(MIGRATIONS_JOURNAL_JSON, "utf8");
|
|
const parsed = JSON.parse(raw) as MigrationJournalFile;
|
|
if (!Array.isArray(parsed.entries)) return [];
|
|
return parsed.entries
|
|
.map((entry, entryIndex) => {
|
|
if (typeof entry?.tag !== "string") return null;
|
|
if (typeof entry?.when !== "number" || !Number.isFinite(entry.when)) return null;
|
|
const order = Number.isInteger(entry.idx) ? Number(entry.idx) : entryIndex;
|
|
return { fileName: `${entry.tag}.sql`, folderMillis: entry.when, order };
|
|
})
|
|
.filter((entry): entry is JournalMigrationEntry => entry !== null);
|
|
} catch {
|
|
return [];
|
|
}
|
|
}
|
|
|
|
async function listJournalMigrationFiles(): Promise<string[]> {
|
|
const entries = await listJournalMigrationEntries();
|
|
return entries.map((entry) => entry.fileName);
|
|
}
|
|
|
|
async function readMigrationFileContent(migrationFile: string): Promise<string> {
|
|
return readFile(new URL(`./migrations/${migrationFile}`, import.meta.url), "utf8");
|
|
}
|
|
|
|
async function orderMigrationsByJournal(migrationFiles: string[]): Promise<string[]> {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const orderByFileName = new Map(journalEntries.map((entry) => [entry.fileName, entry.order]));
|
|
return [...migrationFiles].sort((left, right) => {
|
|
const leftOrder = orderByFileName.get(left);
|
|
const rightOrder = orderByFileName.get(right);
|
|
if (leftOrder === undefined && rightOrder === undefined) return left.localeCompare(right);
|
|
if (leftOrder === undefined) return 1;
|
|
if (rightOrder === undefined) return -1;
|
|
if (leftOrder === rightOrder) return left.localeCompare(right);
|
|
return leftOrder - rightOrder;
|
|
});
|
|
}
|
|
|
|
type SqlExecutor = Pick<ReturnType<typeof postgres>, "unsafe">;
|
|
|
|
async function runInTransaction(sql: SqlExecutor, action: () => Promise<void>): Promise<void> {
|
|
await sql.unsafe("BEGIN");
|
|
try {
|
|
await action();
|
|
await sql.unsafe("COMMIT");
|
|
} catch (error) {
|
|
try {
|
|
await sql.unsafe("ROLLBACK");
|
|
} catch {
|
|
// Ignore rollback failures and surface the original error.
|
|
}
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
async function latestMigrationCreatedAt(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
): Promise<number | null> {
|
|
const rows = await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC NULLS LAST LIMIT 1`,
|
|
);
|
|
const value = Number(rows[0]?.created_at ?? Number.NaN);
|
|
return Number.isFinite(value) ? value : null;
|
|
}
|
|
|
|
function normalizeFolderMillis(value: number | null | undefined): number {
|
|
if (typeof value === "number" && Number.isFinite(value) && value >= 0) {
|
|
return Math.trunc(value);
|
|
}
|
|
return Date.now();
|
|
}
|
|
|
|
async function ensureMigrationJournalTable(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<{ migrationTableSchema: string; columnNames: Set<string> }> {
|
|
let migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
const drizzleSchema = quoteIdentifier("drizzle");
|
|
const migrationTable = quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE);
|
|
await sql.unsafe(`CREATE SCHEMA IF NOT EXISTS ${drizzleSchema}`);
|
|
await sql.unsafe(
|
|
`CREATE TABLE IF NOT EXISTS ${drizzleSchema}.${migrationTable} (id SERIAL PRIMARY KEY, hash text NOT NULL, created_at bigint)`,
|
|
);
|
|
migrationTableSchema = (await discoverMigrationTableSchema(sql)) ?? "drizzle";
|
|
}
|
|
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
return { migrationTableSchema, columnNames };
|
|
}
|
|
|
|
async function migrationHistoryEntryExists(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
columnNames: Set<string>,
|
|
migrationFile: string,
|
|
hash: string,
|
|
): Promise<boolean> {
|
|
const predicates: string[] = [];
|
|
if (columnNames.has("hash")) predicates.push(`hash = ${quoteLiteral(hash)}`);
|
|
if (columnNames.has("name")) predicates.push(`name = ${quoteLiteral(migrationFile)}`);
|
|
if (predicates.length === 0) return false;
|
|
|
|
const rows = await sql.unsafe<{ one: number }[]>(
|
|
`SELECT 1 AS one FROM ${qualifiedTable} WHERE ${predicates.join(" OR ")} LIMIT 1`,
|
|
);
|
|
return rows.length > 0;
|
|
}
|
|
|
|
async function recordMigrationHistoryEntry(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
columnNames: Set<string>,
|
|
migrationFile: string,
|
|
hash: string,
|
|
folderMillis: number,
|
|
): Promise<void> {
|
|
const insertColumns: string[] = [];
|
|
const insertValues: string[] = [];
|
|
|
|
if (columnNames.has("hash")) {
|
|
insertColumns.push(quoteIdentifier("hash"));
|
|
insertValues.push(quoteLiteral(hash));
|
|
}
|
|
if (columnNames.has("name")) {
|
|
insertColumns.push(quoteIdentifier("name"));
|
|
insertValues.push(quoteLiteral(migrationFile));
|
|
}
|
|
if (columnNames.has("created_at")) {
|
|
const latestCreatedAt = await latestMigrationCreatedAt(sql, qualifiedTable);
|
|
const createdAt = latestCreatedAt === null
|
|
? normalizeFolderMillis(folderMillis)
|
|
: Math.max(latestCreatedAt + 1, normalizeFolderMillis(folderMillis));
|
|
insertColumns.push(quoteIdentifier("created_at"));
|
|
insertValues.push(quoteLiteral(String(createdAt)));
|
|
}
|
|
|
|
if (insertColumns.length === 0) return;
|
|
|
|
await sql.unsafe(
|
|
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
|
|
);
|
|
}
|
|
|
|
// These idempotent migrations require top-level statements for batch commits
|
|
// or concurrent index builds. History is recorded only after completion.
|
|
const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
|
|
"0313_private_task_access.sql",
|
|
"0314_private_task_draft_assets.sql",
|
|
"0332_little_night_nurse.sql",
|
|
"0345_luxuriant_carmella_unuscione.sql",
|
|
"0369_late_logan.sql",
|
|
"0381_mean_scream.sql",
|
|
"0382_narrow_the_executioner.sql",
|
|
"0385_open_madame_web.sql",
|
|
"0386_steady_epoch.sql",
|
|
"0387_boring_professor_monster.sql",
|
|
"0388_ambitious_micromax.sql",
|
|
"0389_brave_hulk.sql",
|
|
"0390_salty_killraven.sql",
|
|
"0391_third_roulette.sql",
|
|
"0392_small_frightful_four.sql",
|
|
"0393_open_grandmaster.sql",
|
|
"0395_tranquil_firestar.sql",
|
|
"0396_tan_paladin.sql",
|
|
"0397_known_madrox.sql",
|
|
"0398_yielding_random.sql",
|
|
"0399_ambitious_onslaught.sql",
|
|
"0400_absent_phantom_reporter.sql",
|
|
"0401_yummy_ultimates.sql",
|
|
"0402_ancient_malice.sql",
|
|
"0403_natural_cerise.sql",
|
|
"0406_zippy_war_machine.sql",
|
|
"0407_greedy_star_brand.sql",
|
|
"0408_short_red_skull.sql",
|
|
"0409_uneven_zuras.sql",
|
|
"0410_freezing_kinsey_walden.sql",
|
|
"0411_real_scrambler.sql",
|
|
"0412_tearful_squadron_supreme.sql",
|
|
"0413_tired_kronos.sql",
|
|
"0414_closed_edwin_jarvis.sql",
|
|
"0419_slippery_impossible_man.sql",
|
|
"0432_strong_mole_man.sql",
|
|
"0435_premium_puff_adder.sql",
|
|
"0436_legal_reavers.sql",
|
|
"0447_legacy_controller_process.sql",
|
|
"0450_petite_frightful_four.sql",
|
|
"0453_curious_ben_parker.sql",
|
|
"0454_lumpy_ronan.sql",
|
|
"0455_pending_budget_cancellation_index.sql",
|
|
"0456_chat_review_lookup_indexes.sql",
|
|
"0457_native_session_run_index.sql",
|
|
"0458_chat_admission_issue_index.sql",
|
|
"0459_environment_lease_backup_scope_index.sql",
|
|
"0460_workspace_activity_context_index.sql",
|
|
"0461_created_from_legacy_origin_run_index.sql",
|
|
"0470_heartbeat_runs_queued_created_index.sql",
|
|
"0472_runner_api_capture_pending_deletes.sql",
|
|
"0476_wakeup_request_run_index.sql",
|
|
"0469_zippy_photon.sql",
|
|
|
|
|
|
"0384_funny_adam_warlock.sql",
|
|
"0383_tool_review_company_expiry.sql",
|
|
]);
|
|
|
|
async function applyPendingMigrationsManually(
|
|
url: string,
|
|
pendingMigrations: string[],
|
|
): Promise<void> {
|
|
if (pendingMigrations.length === 0) return;
|
|
|
|
const orderedPendingMigrations = await orderMigrationsByJournal(pendingMigrations);
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const folderMillisByFileName = new Map(
|
|
journalEntries.map((entry) => [entry.fileName, normalizeFolderMillis(entry.folderMillis)]),
|
|
);
|
|
|
|
const pool = createUtilitySql(url);
|
|
try {
|
|
const sql = await pool.reserve();
|
|
try {
|
|
// A session lock survives each batch COMMIT. Reserve the connection so
|
|
// pool rotation cannot release it while another migrator is waiting.
|
|
// A blocking lock SELECT retains its statement snapshot while waiting.
|
|
// A partial concurrent index build can wait for that snapshot, forming a
|
|
// deadlock with this lock. Each failed try ends its snapshot before sleep.
|
|
while (!(await sql<{ locked: boolean }[]>`
|
|
SELECT pg_try_advisory_lock(hashtextextended('paperclip:migrations', 0)) AS locked`)[0]?.locked) {
|
|
await new Promise((resolve) => setTimeout(resolve, 100));
|
|
}
|
|
const { migrationTableSchema, columnNames } = await ensureMigrationJournalTable(sql);
|
|
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
|
|
for (const migrationFile of orderedPendingMigrations) {
|
|
const migrationContent = await readMigrationFileContent(migrationFile);
|
|
const hash = createHash("sha256").update(migrationContent).digest("hex");
|
|
const existingEntry = await migrationHistoryEntryExists(
|
|
sql,
|
|
qualifiedTable,
|
|
columnNames,
|
|
migrationFile,
|
|
hash,
|
|
);
|
|
if (existingEntry) continue;
|
|
|
|
const applyMigration = async () => {
|
|
for (const statement of splitMigrationStatements(migrationContent)) {
|
|
if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
|
|
// A cancelled concurrent build leaves an invalid index. IF NOT
|
|
// EXISTS alone would skip it on retry and journal an incomplete index.
|
|
const index = statement.replace(/^\s*--.*$/gm, "").trim()
|
|
.match(/^CREATE (?:UNIQUE )?INDEX CONCURRENTLY IF NOT EXISTS "([A-Za-z_][A-Za-z0-9_]*)"/i);
|
|
if (index) {
|
|
const invalid = await sql<{ invalid: boolean }[]>`
|
|
SELECT NOT i.indisvalid AS invalid FROM pg_index i
|
|
JOIN pg_class c ON c.oid = i.indexrelid
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public' AND c.relname = ${index[1]}`;
|
|
if (invalid[0]?.invalid) {
|
|
await sql.unsafe(`DROP INDEX CONCURRENTLY ${quoteIdentifier("public")}.${quoteIdentifier(index[1])}`);
|
|
}
|
|
}
|
|
}
|
|
// Older dev schemas can lack a replaced constraint. Preserve the
|
|
// published SQL/hash while allowing the harmless drop to proceed.
|
|
const executable = statement.replace(
|
|
/^(\s*ALTER TABLE "[^"]+" DROP CONSTRAINT) (?!IF EXISTS\b)/i,
|
|
"$1 IF EXISTS ",
|
|
);
|
|
await sql.unsafe(executable);
|
|
}
|
|
|
|
await recordMigrationHistoryEntry(
|
|
sql,
|
|
qualifiedTable,
|
|
columnNames,
|
|
migrationFile,
|
|
hash,
|
|
folderMillisByFileName.get(migrationFile) ?? Date.now(),
|
|
);
|
|
};
|
|
if (migrationFile === PROJECT_ENVIRONMENT_OVERLAP.dropFile) {
|
|
const repairFile = PROJECT_ENVIRONMENT_OVERLAP.repairFile;
|
|
const repairContent = await readMigrationFileContent(repairFile);
|
|
const repairHash = createHash("sha256").update(repairContent).digest("hex");
|
|
const dropIndex = orderedPendingMigrations.indexOf(migrationFile);
|
|
if (hash !== PROJECT_ENVIRONMENT_OVERLAP.dropSha256
|
|
|| repairHash !== PROJECT_ENVIRONMENT_OVERLAP.repairSha256
|
|
|| orderedPendingMigrations[dropIndex + 1] !== repairFile
|
|
|| await migrationHistoryEntryExists(sql, qualifiedTable, columnNames, repairFile, repairHash)) {
|
|
throw new Error("Project environment migration pair does not match the qualified overlap repair");
|
|
}
|
|
await runInTransaction(sql, async () => {
|
|
await sql.unsafe('LOCK TABLE "public"."projects" IN ACCESS EXCLUSIVE MODE');
|
|
await sql.unsafe(`CREATE TEMP TABLE paperclip_project_environment_overlap ON COMMIT DROP AS
|
|
SELECT id, environment_id FROM "public"."projects" WHERE environment_id IS NOT NULL`);
|
|
await applyMigration();
|
|
for (const statement of splitMigrationStatements(repairContent)) await sql.unsafe(statement);
|
|
await sql.unsafe(`UPDATE "public"."projects" AS project SET environment_id = legacy.environment_id
|
|
FROM pg_temp.paperclip_project_environment_overlap AS legacy WHERE project.id = legacy.id`);
|
|
await recordMigrationHistoryEntry(sql, qualifiedTable, columnNames, repairFile, repairHash,
|
|
folderMillisByFileName.get(repairFile) ?? Date.now());
|
|
});
|
|
} else if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
|
|
await applyMigration();
|
|
} else {
|
|
await runInTransaction(sql, applyMigration);
|
|
}
|
|
}
|
|
} finally {
|
|
try {
|
|
await sql`SELECT pg_advisory_unlock(hashtextextended('paperclip:migrations', 0))`;
|
|
} catch {
|
|
// Preserve the migration error if the session died; PostgreSQL releases
|
|
// its lock on disconnect, and pool.end() closes any remaining connection.
|
|
}
|
|
sql.release();
|
|
}
|
|
} finally {
|
|
await pool.end();
|
|
}
|
|
}
|
|
|
|
async function mapHashesToMigrationFiles(migrationFiles: string[]): Promise<Map<string, string>> {
|
|
const mapped = new Map<string, string>();
|
|
|
|
await Promise.all(
|
|
migrationFiles.map(async (migrationFile) => {
|
|
const content = await readMigrationFileContent(migrationFile);
|
|
const hash = createHash("sha256").update(content).digest("hex");
|
|
mapped.set(hash, migrationFile);
|
|
if (migrationFile === "0355_agent_tool_definition_hooks.sql") {
|
|
mapped.set("7ac76adac0220da522c78e51da20490aeb0f90512b4370a05af9f8dd0f75eda6", migrationFile);
|
|
}
|
|
}),
|
|
);
|
|
|
|
return mapped;
|
|
}
|
|
|
|
async function getMigrationTableColumnNames(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationTableSchema: string,
|
|
): Promise<Set<string>> {
|
|
const columns = await sql.unsafe<{ column_name: string }[]>(
|
|
`
|
|
SELECT column_name
|
|
FROM information_schema.columns
|
|
WHERE table_schema = ${quoteLiteral(migrationTableSchema)}
|
|
AND table_name = ${quoteLiteral(DRIZZLE_MIGRATIONS_TABLE)}
|
|
`,
|
|
);
|
|
return new Set(columns.map((column) => column.column_name));
|
|
}
|
|
|
|
async function tableExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM information_schema.tables
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function columnExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
columnName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM information_schema.columns
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
AND column_name = ${columnName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function columnHasDataType(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
columnName: string,
|
|
dataType: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ dataType: string; udtName: string }[]>`
|
|
SELECT data_type AS "dataType", udt_name AS "udtName"
|
|
FROM information_schema.columns
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
AND column_name = ${columnName}
|
|
`;
|
|
const expected = dataType.toLowerCase();
|
|
return rows.some((row) => (
|
|
row.dataType.toLowerCase() === expected || row.udtName.toLowerCase() === expected
|
|
));
|
|
}
|
|
|
|
async function indexExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
indexName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_class c
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public'
|
|
AND c.relkind = 'i'
|
|
AND c.relname = ${indexName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function constraintExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
constraintName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_constraint c
|
|
JOIN pg_namespace n ON n.oid = c.connamespace
|
|
WHERE n.nspname = 'public'
|
|
AND c.conname = ${constraintName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function functionExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
functionName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_proc p
|
|
JOIN pg_namespace n ON n.oid = p.pronamespace
|
|
WHERE n.nspname = 'public'
|
|
AND p.proname = ${functionName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function triggerExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
triggerName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_trigger t
|
|
JOIN pg_class c ON c.oid = t.tgrelid
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public'
|
|
AND t.tgname = ${triggerName}
|
|
AND NOT t.tgisinternal
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function heartbeatEventSequencesAreUnique(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ unique: boolean }[]>`
|
|
SELECT NOT EXISTS (
|
|
SELECT 1
|
|
FROM heartbeat_run_events
|
|
GROUP BY run_id, seq
|
|
HAVING count(*) > 1
|
|
) AS unique
|
|
`;
|
|
return rows[0]?.unique ?? false;
|
|
}
|
|
|
|
async function heartbeatNextEventSequencesAreCurrent(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ current: boolean }[]>`
|
|
SELECT NOT EXISTS (
|
|
SELECT 1
|
|
FROM heartbeat_runs run
|
|
WHERE run.next_event_seq IS DISTINCT FROM COALESCE((
|
|
SELECT max(event.seq) + 1
|
|
FROM heartbeat_run_events event
|
|
WHERE event.run_id = run.id
|
|
), 1)
|
|
) AS current
|
|
`;
|
|
return rows[0]?.current ?? false;
|
|
}
|
|
|
|
async function migrationStatementAlreadyApplied(
|
|
sql: ReturnType<typeof postgres>,
|
|
statement: string,
|
|
): Promise<boolean> {
|
|
const normalized = statement
|
|
.replace(/^\s*--.*$/gm, "")
|
|
.replace(/\s+/g, " ")
|
|
.trim();
|
|
|
|
const createTableMatch = normalized.match(/^CREATE TABLE(?: IF NOT EXISTS)? "([^"]+)"/i);
|
|
if (createTableMatch) {
|
|
return tableExists(sql, createTableMatch[1]);
|
|
}
|
|
|
|
const addColumnMatch = normalized.match(
|
|
/^ALTER TABLE "([^"]+)" ADD COLUMN(?: IF NOT EXISTS)? "([^"]+)"/i,
|
|
);
|
|
if (addColumnMatch) {
|
|
return columnExists(sql, addColumnMatch[1], addColumnMatch[2]);
|
|
}
|
|
|
|
const alterColumnTypeMatch = normalized.match(
|
|
/^ALTER TABLE "([^"]+)" ALTER COLUMN "([^"]+)" SET DATA TYPE ([A-Za-z0-9_]+)/i,
|
|
);
|
|
if (alterColumnTypeMatch) {
|
|
return columnHasDataType(
|
|
sql,
|
|
alterColumnTypeMatch[1],
|
|
alterColumnTypeMatch[2],
|
|
alterColumnTypeMatch[3],
|
|
);
|
|
}
|
|
|
|
const createIndexMatch = normalized.match(/^CREATE (?:UNIQUE )?INDEX(?: IF NOT EXISTS)? "([^"]+)"/i);
|
|
if (createIndexMatch) {
|
|
return indexExists(sql, createIndexMatch[1]);
|
|
}
|
|
|
|
const addConstraintMatch = normalized.match(/^ALTER TABLE "([^"]+)" ADD CONSTRAINT "([^"]+)"/i);
|
|
if (addConstraintMatch) {
|
|
return constraintExists(sql, addConstraintMatch[2]);
|
|
}
|
|
|
|
const createFunctionMatch = normalized.match(
|
|
/^CREATE OR REPLACE FUNCTION "?([A-Za-z_][A-Za-z0-9_]*)"?\s*\(/i,
|
|
);
|
|
if (createFunctionMatch) {
|
|
return functionExists(sql, createFunctionMatch[1]);
|
|
}
|
|
|
|
const createTriggerMatch = normalized.match(
|
|
/^CREATE TRIGGER "?([A-Za-z_][A-Za-z0-9_]*)"?/i,
|
|
);
|
|
if (createTriggerMatch) {
|
|
return triggerExists(sql, createTriggerMatch[1]);
|
|
}
|
|
|
|
// These native-runner repairs have persistent postconditions. Verify them
|
|
// instead of replaying them when a restored database is missing only the
|
|
// migration-history row.
|
|
if (
|
|
normalized.startsWith("WITH ranked AS (")
|
|
&& normalized.includes('UPDATE "heartbeat_run_events" AS event')
|
|
) {
|
|
return heartbeatEventSequencesAreUnique(sql);
|
|
}
|
|
if (
|
|
normalized.startsWith('UPDATE "heartbeat_runs" AS run')
|
|
&& normalized.includes('SET "next_event_seq" = COALESCE')
|
|
) {
|
|
return heartbeatNextEventSequencesAreCurrent(sql);
|
|
}
|
|
|
|
// If we cannot reason about a statement safely, require manual migration.
|
|
return false;
|
|
}
|
|
|
|
async function migrationContentAlreadyApplied(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationContent: string,
|
|
): Promise<boolean> {
|
|
const statements = splitMigrationStatements(migrationContent);
|
|
if (statements.length === 0) return false;
|
|
|
|
for (const statement of statements) {
|
|
const applied = await migrationStatementAlreadyApplied(sql, statement);
|
|
if (!applied) return false;
|
|
}
|
|
|
|
return true;
|
|
}
|
|
|
|
async function loadAppliedMigrations(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationTableSchema: string,
|
|
availableMigrations: string[],
|
|
): Promise<string[]> {
|
|
const quotedSchema = quoteIdentifier(migrationTableSchema);
|
|
const qualifiedTable = `${quotedSchema}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
|
|
if (columnNames.has("name")) {
|
|
const rows = await sql.unsafe<{ name: string }[]>(`SELECT name FROM ${qualifiedTable} ORDER BY id`);
|
|
return rows.map((row) => row.name).filter((name): name is string => Boolean(name));
|
|
}
|
|
|
|
if (columnNames.has("hash")) {
|
|
const rows = await sql.unsafe<{ hash: string }[]>(`SELECT hash FROM ${qualifiedTable} ORDER BY id`);
|
|
const hashesToMigrationFiles = await mapHashesToMigrationFiles(availableMigrations);
|
|
const appliedFromHashes = rows
|
|
.map((row) => hashesToMigrationFiles.get(row.hash))
|
|
.filter((name): name is string => Boolean(name));
|
|
|
|
if (appliedFromHashes.length > 0) {
|
|
// Best-effort: when all hashes resolve, this is authoritative.
|
|
if (appliedFromHashes.length === rows.length) return appliedFromHashes;
|
|
|
|
// Partial hash resolution can happen when files have changed; return what we can trust.
|
|
return appliedFromHashes;
|
|
}
|
|
|
|
// Fallback only when hashes are unavailable/unresolved.
|
|
if (columnNames.has("created_at")) {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
if (journalEntries.length > 0) {
|
|
const lastDbRows = await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC LIMIT 1`,
|
|
);
|
|
const lastCreatedAt = Number(lastDbRows[0]?.created_at ?? -1);
|
|
if (Number.isFinite(lastCreatedAt) && lastCreatedAt >= 0) {
|
|
return journalEntries
|
|
.filter((entry) => availableMigrations.includes(entry.fileName))
|
|
.filter((entry) => entry.folderMillis <= lastCreatedAt)
|
|
.map((entry) => entry.fileName)
|
|
.slice(0, rows.length);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
const rows = await sql.unsafe<{ id: number }[]>(`SELECT id FROM ${qualifiedTable} ORDER BY id`);
|
|
const journalMigrationFiles = await listJournalMigrationFiles();
|
|
const appliedFromIds = rows
|
|
.map((row) => journalMigrationFiles[row.id - 1])
|
|
.filter((name): name is string => Boolean(name));
|
|
if (appliedFromIds.length > 0) return appliedFromIds;
|
|
|
|
return availableMigrations.slice(0, Math.max(0, rows.length));
|
|
}
|
|
|
|
export type MigrationHistoryReconcileResult = {
|
|
repairedMigrations: string[];
|
|
remainingMigrations: string[];
|
|
};
|
|
|
|
export async function reconcilePendingMigrationHistory(
|
|
url: string,
|
|
): Promise<MigrationHistoryReconcileResult> {
|
|
const state = await inspectMigrations(url);
|
|
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
|
|
return { repairedMigrations: [], remainingMigrations: [] };
|
|
}
|
|
|
|
const sql = createUtilitySql(url);
|
|
const repairedMigrations: string[] = [];
|
|
|
|
try {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const folderMillisByFile = new Map(journalEntries.map((entry) => [entry.fileName, entry.folderMillis]));
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
return { repairedMigrations, remainingMigrations: state.pendingMigrations };
|
|
}
|
|
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
|
|
for (const migrationFile of state.pendingMigrations) {
|
|
const migrationContent = await readMigrationFileContent(migrationFile);
|
|
const alreadyApplied = await migrationContentAlreadyApplied(sql, migrationContent);
|
|
if (!alreadyApplied) break;
|
|
|
|
const hash = createHash("sha256").update(migrationContent).digest("hex");
|
|
const folderMillis = folderMillisByFile.get(migrationFile) ?? Date.now();
|
|
const existingByHash = columnNames.has("hash")
|
|
? await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} WHERE hash = ${quoteLiteral(hash)} ORDER BY created_at DESC LIMIT 1`,
|
|
)
|
|
: [];
|
|
const existingByName = columnNames.has("name")
|
|
? await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} WHERE name = ${quoteLiteral(migrationFile)} ORDER BY created_at DESC LIMIT 1`,
|
|
)
|
|
: [];
|
|
if (existingByHash.length > 0 || existingByName.length > 0) {
|
|
if (columnNames.has("created_at")) {
|
|
const existingHashCreatedAt = Number(existingByHash[0]?.created_at ?? -1);
|
|
if (existingByHash.length > 0 && Number.isFinite(existingHashCreatedAt) && existingHashCreatedAt < folderMillis) {
|
|
await sql.unsafe(
|
|
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE hash = ${quoteLiteral(hash)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
|
|
);
|
|
}
|
|
|
|
const existingNameCreatedAt = Number(existingByName[0]?.created_at ?? -1);
|
|
if (existingByName.length > 0 && Number.isFinite(existingNameCreatedAt) && existingNameCreatedAt < folderMillis) {
|
|
await sql.unsafe(
|
|
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE name = ${quoteLiteral(migrationFile)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
repairedMigrations.push(migrationFile);
|
|
continue;
|
|
}
|
|
|
|
const insertColumns: string[] = [];
|
|
const insertValues: string[] = [];
|
|
|
|
if (columnNames.has("hash")) {
|
|
insertColumns.push(quoteIdentifier("hash"));
|
|
insertValues.push(quoteLiteral(hash));
|
|
}
|
|
if (columnNames.has("name")) {
|
|
insertColumns.push(quoteIdentifier("name"));
|
|
insertValues.push(quoteLiteral(migrationFile));
|
|
}
|
|
if (columnNames.has("created_at")) {
|
|
insertColumns.push(quoteIdentifier("created_at"));
|
|
insertValues.push(quoteLiteral(String(folderMillis)));
|
|
}
|
|
|
|
if (insertColumns.length === 0) break;
|
|
|
|
await sql.unsafe(
|
|
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
|
|
);
|
|
repairedMigrations.push(migrationFile);
|
|
}
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
|
|
const refreshed = await inspectMigrations(url);
|
|
return {
|
|
repairedMigrations,
|
|
remainingMigrations:
|
|
refreshed.status === "needsMigrations" ? refreshed.pendingMigrations : [],
|
|
};
|
|
}
|
|
|
|
async function discoverMigrationTableSchema(sql: ReturnType<typeof postgres>): Promise<string | null> {
|
|
const rows = await sql<{ schemaName: string }[]>`
|
|
SELECT n.nspname AS "schemaName"
|
|
FROM pg_class c
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE c.relname = ${DRIZZLE_MIGRATIONS_TABLE} AND c.relkind = 'r'
|
|
`;
|
|
|
|
if (rows.length === 0) return null;
|
|
|
|
const drizzleSchema = rows.find(({ schemaName }) => schemaName === "drizzle");
|
|
if (drizzleSchema) return drizzleSchema.schemaName;
|
|
|
|
const publicSchema = rows.find(({ schemaName }) => schemaName === "public");
|
|
if (publicSchema) return publicSchema.schemaName;
|
|
|
|
return rows[0]?.schemaName ?? null;
|
|
}
|
|
|
|
export async function inspectMigrations(url: string): Promise<MigrationState> {
|
|
const sql = createUtilitySql(url);
|
|
|
|
try {
|
|
const availableMigrations = await listMigrationFiles();
|
|
const tableCountResult = await sql<{ count: number }[]>`
|
|
select count(*)::int as count
|
|
from information_schema.tables
|
|
where table_schema = 'public'
|
|
and table_type = 'BASE TABLE'
|
|
`;
|
|
const tableCount = tableCountResult[0]?.count ?? 0;
|
|
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
if (tableCount > 0) {
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations: [],
|
|
pendingMigrations: availableMigrations,
|
|
journalEntryCount: 0,
|
|
reason: "no-migration-journal-non-empty-db",
|
|
};
|
|
}
|
|
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations: [],
|
|
pendingMigrations: availableMigrations,
|
|
journalEntryCount: 0,
|
|
reason: "no-migration-journal-empty-db",
|
|
};
|
|
}
|
|
|
|
const qualifiedMigrationTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
const journalCountRows = await sql.unsafe<{ count: number }[]>(
|
|
`SELECT count(*)::int AS count FROM ${qualifiedMigrationTable}`,
|
|
);
|
|
const journalEntryCount = journalCountRows[0]?.count ?? 0;
|
|
const appliedMigrations = await loadAppliedMigrations(sql, migrationTableSchema, availableMigrations);
|
|
const pendingMigrations = availableMigrations.filter((name) => !appliedMigrations.includes(name));
|
|
if (pendingMigrations.length === 0) {
|
|
return {
|
|
status: "upToDate",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations,
|
|
journalEntryCount,
|
|
};
|
|
}
|
|
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations,
|
|
pendingMigrations,
|
|
journalEntryCount,
|
|
reason: "pending-migrations",
|
|
};
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function applyPendingMigrations(url: string): Promise<void> {
|
|
const initialState = await inspectMigrations(url);
|
|
if (initialState.status === "upToDate") return;
|
|
|
|
if (initialState.reason === "no-migration-journal-empty-db") {
|
|
// Use the same per-file executor for bootstrap and upgrades: Drizzle wraps
|
|
// the whole journal in one transaction and cannot run committing backfills.
|
|
await applyPendingMigrationsManually(url, initialState.pendingMigrations);
|
|
const bootstrappedState = await inspectMigrations(url);
|
|
if (bootstrappedState.status === "upToDate") return;
|
|
throw new Error(
|
|
`Failed to bootstrap migrations: ${bootstrappedState.pendingMigrations.join(", ")}`,
|
|
);
|
|
}
|
|
|
|
if (initialState.reason === "no-migration-journal-non-empty-db") {
|
|
throw new Error(
|
|
"Database has tables but no migration journal; automatic migration is unsafe. Initialize migration history manually.",
|
|
);
|
|
}
|
|
|
|
let state = await inspectMigrations(url);
|
|
if (state.status === "upToDate") return;
|
|
|
|
const repair = await reconcilePendingMigrationHistory(url);
|
|
if (repair.repairedMigrations.length > 0) {
|
|
state = await inspectMigrations(url);
|
|
if (state.status === "upToDate") return;
|
|
}
|
|
|
|
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
|
|
throw new Error("Migrations are still pending after migration-history reconciliation; run inspectMigrations for details.");
|
|
}
|
|
|
|
await applyPendingMigrationsManually(url, state.pendingMigrations);
|
|
|
|
const finalState = await inspectMigrations(url);
|
|
if (finalState.status !== "upToDate") {
|
|
throw new Error(
|
|
`Failed to apply pending migrations: ${finalState.pendingMigrations.join(", ")}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
export type MigrationBootstrapResult =
|
|
| { migrated: true; reason: "migrated-empty-db"; tableCount: 0 }
|
|
| { migrated: false; reason: "already-migrated"; tableCount: number }
|
|
| { migrated: false; reason: "not-empty-no-migration-journal"; tableCount: number };
|
|
|
|
export async function migratePostgresIfEmpty(url: string): Promise<MigrationBootstrapResult> {
|
|
const sql = createUtilitySql(url);
|
|
|
|
try {
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
|
|
const tableCountResult = await sql<{ count: number }[]>`
|
|
select count(*)::int as count
|
|
from information_schema.tables
|
|
where table_schema = 'public'
|
|
and table_type = 'BASE TABLE'
|
|
`;
|
|
|
|
const tableCount = tableCountResult[0]?.count ?? 0;
|
|
|
|
if (migrationTableSchema) {
|
|
return { migrated: false, reason: "already-migrated", tableCount };
|
|
}
|
|
|
|
if (tableCount > 0) {
|
|
return { migrated: false, reason: "not-empty-no-migration-journal", tableCount };
|
|
}
|
|
|
|
await applyPendingMigrationsManually(url, await listMigrationFiles());
|
|
|
|
return { migrated: true, reason: "migrated-empty-db", tableCount: 0 };
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function ensurePostgresDatabase(
|
|
url: string,
|
|
databaseName: string,
|
|
): Promise<"created" | "exists"> {
|
|
if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(databaseName)) {
|
|
throw new Error(`Unsafe database name: ${databaseName}`);
|
|
}
|
|
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
const existing = await sql<{ one: number }[]>`
|
|
select 1 as one from pg_database where datname = ${databaseName} limit 1
|
|
`;
|
|
if (existing.length > 0) return "exists";
|
|
|
|
await sql.unsafe(`create database "${databaseName}" encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
|
|
return "created";
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function resetPostgresDatabase(
|
|
url: string,
|
|
databaseName: string,
|
|
): Promise<"reset"> {
|
|
const quotedDatabaseName = quoteIdentifier(databaseName);
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
await sql`
|
|
select pg_terminate_backend(pid)
|
|
from pg_stat_activity
|
|
where datname = ${databaseName}
|
|
and pid <> pg_backend_pid()
|
|
`;
|
|
await sql.unsafe(`drop database if exists ${quotedDatabaseName}`);
|
|
await sql.unsafe(`create database ${quotedDatabaseName} encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
|
|
return "reset";
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export type Db = ReturnType<typeof createDb>;
|