Files
PaperClipAI/packages/db/src/client.ts
T
DottaandClaude Opus 5.5 49f0030f06 fix(heartbeat): rank a person's joined comment first and refill free run slots (#21277)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts each agent's queued runs, up to the
agent's `maxConcurrentRuns`
> - #21234 starts person-initiated runs before system recovery runs, and
it reads "person-initiated" from the run's own wake request
> - When a person comments on a task that already has a queued system
run, the comment joins that run. The run keeps its system wake, so
dispatch ranks the person's message as system work
> - Dispatch is event driven. The queued-run re-drive backstop covers
only agents with no live run, so an agent with several slots keeps a
free slot empty until one of its runs finishes, and runs can take hours
> - This pull request makes dispatch read the person wakes that joined
each queued run, and makes the re-drive backstop cover an agent that
still has a free slot
> - The benefit is that a person's comment starts first, and free slots
fill within about a minute instead of waiting for an unrelated run to
finish

## Linked Issues or Issue Description

Refs #21234, #20986

**What happened?**

On staging, one agent had `maxConcurrentRuns` 20, 18 running runs and 76
queued runs at 03:12Z. No run started on that agent from 03:01:33Z to
03:16:57Z (15.4 minutes). From 03:11:10Z, after a run finished, at least
2 slots were free. Other agents in the same company started runs during
that time. Runs started again only after the next deploy restarted the
server.

Five queued runs came from a person: comments at 02:43Z, 02:44Z and
02:45Z, and Retries at 02:46Z, 02:49Z and 03:01Z. The comment runs had
the wake reasons `issue_reopened_via_comment` and `issue_commented`, the
invocation source `automation`, and no person on the run's own wake
request.

**Expected behavior**

A run that carries a person's comment ranks as person-initiated, also
when the comment joined an existing queued run. When an agent has a free
slot and a queued run is ready, the run starts promptly, without waiting
for another run on the agent to finish.

**Steps to reproduce**

1. Give an agent `maxConcurrentRuns` 3. Start 2 runs on it.
2. Queue a third run while no dispatch signal reaches the queue (for
example, a claim declines it while a gate is still closed, and the gate
then clears).
3. See that `listUndispatchedQueuedRuns` returns nothing, because the
agent has a live run. The run stays queued until one of the 2 running
runs finishes.
4. Separately, queue a system run on a task, then post a person's
comment on the task so that the comment wake coalesces into that queued
run. Queue older system runs on other tasks of the same single-slot
agent.
5. See that dispatch starts the older system runs first.

**Root cause**

1. `coalesceIntoActiveExecutionRun`
(`server/src/modules/wake-queue/adapters/postgres.ts`) merges the
person's comment context into the queued run and records the person's
wake as a separate `coalesced` row with `runId` set to that run. The
run's own `wakeupRequestId` still points to the system wake.
`queuedRunDispatcherForAgent` (`server/src/services/heartbeat/queue.ts`)
only read the wake behind `wakeupRequestId`. The task routes do record
the comment author as the requester (`requestedByActorType: "user"`).
The person was lost only at dispatch time.
2. `listUndispatchedQueuedRunPage`
(`server/src/services/queued-run-dispatch-liveness.ts`) skipped every
agent with any `running` or `scheduled_retry` run. That rule assumes one
slot per agent. For an agent with several slots, a lost or declined
dispatch is not re-driven until a run on that agent finishes.

This change is not in `claimQueuedRun`, the fresh-isolated preflight,
the wake-queue promoter or `fresh-continuation-fallback.ts`.

**Paperclip version or commit**

Staging Core `ddc0aacad8` (contains #21234 `ade7e0643f`). This branch is
based on master `6cb0dd0c6b`.

**Deployment mode**

Authenticated, public, Paperclip Cloud managed staging.

**Agent adapter(s) involved**

`paperclip_runner` (Codex native runtime). The fix is
adapter-independent.

## What Changed

- `queuedRunDispatcherForAgent` also reads person wakes whose `runId` is
a queued run of the agent (by `run_id IN (queued run ids)`, so an
earlier saved comment that a run adopted is included). A run with such a
wake ranks as person-initiated and is never counted as system recovery.
- `listUndispatchedQueuedRuns` now returns a queued run when its agent
has fewer `running` plus `scheduled_retry` runs than its
`maxConcurrentRuns`, and its task has no `running` or `scheduled_retry`
run. The re-drive backstop and the ownership I7 check use this query.
- The query counts each agent's live runs once, in one grouped subquery
that is joined to the queued runs.
- Migration 0476 adds `agent_wakeup_requests_company_run_idx`
(`company_id`, `run_id`, partial on a non-null run) with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`. It is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` and classifies as
`online_compatible`. `doc/DATABASE.md` describes it.
- Added `maxConcurrentRunsSql()` to `timer-policy.ts`. It reads the same
limit as `parseHeartbeatPolicy`, including the single-run runner
providers, which now share one constant.

## Verification

- Red/green: `server/src/__tests__/recovery-wake-herd.test.ts` "starts a
queued run that a person's comment joined ahead of older system work".
It fails on master (`expected null not to be null`: the older system run
started first) and passes with this change.
- Red/green:
`server/src/__tests__/ownership-reconciliation-report.test.ts`
"re-drives a queued run while its agent still has a free slot, and not
once every slot is running". It fails on master (`expected [] to deeply
equal [ Array(1) ]`) and passes with this change.
- Red/green: "starts a queued run that adopted a person's comment saved
before every queued run". It fails with a `requestedAt` bound and passes
with the `run_id` lookup.
- Red/green: "counts an agent's live runs in one grouped aggregate, not
once per queued run" runs EXPLAIN on the page query. It fails with a
per-row correlated count and passes with the grouped subquery.
- New: "uses the agent's single-run limit for a runner provider that
runs one turn at a time".
- Two existing tests now set `maxConcurrentRuns: 1` so that they keep
testing "the agent's only slot is busy".
- `npx vitest run src/__tests__/recovery-wake-herd.test.ts
src/services/heartbeat/queued-run-dispatch-order.test.ts
src/__tests__/ownership-reconciliation-report.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/services/heartbeat/dispatch-work.test.ts`: 71 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: no errors.

## Risks

- The re-drive backstop now also covers multi-slot agents with live
runs. Each queued run is re-driven at most on its existing backoff (1
minute, then doubling to 15 minutes). A queued run whose task already
has a live run is not re-driven.
- The ownership I7 report can now list a queued run on a multi-slot
agent that has a free slot. This is the stall that this change targets.
- One more indexed read per dispatch pass (`agent_wakeup_requests` by
company and run id). The new index is built concurrently, so the
previous release keeps writing during the build.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, read-only staging API reads in a
browser tab).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 01:17:57 -06:00

1462 lines
55 KiB
TypeScript

import { createHash } from "node:crypto";
import { readFile, readdir } from "node:fs/promises";
import { fileURLToPath } from "node:url";
import postgres from "postgres";
import { installDatabaseWorkSignals } from "./work-signals.js";
import { installDatabaseResponseCapture } from "./response-capture.js";
import { installPreparedQueries } from "./prepared-queries.js";
import { drizzleWithLazyTransactionQueries } from "./lazy-transaction-queries.js";
import * as schema from "./schema/index.js";
const MIGRATIONS_FOLDER = fileURLToPath(new URL("./migrations", import.meta.url));
const DRIZZLE_MIGRATIONS_TABLE = "__drizzle_migrations";
const MIGRATIONS_JOURNAL_JSON = fileURLToPath(new URL("./migrations/meta/_journal.json", import.meta.url));
const PROJECT_ENVIRONMENT_OVERLAP = {
dropFile: "0466_drop_project_environment.sql",
dropSha256: "07ca1c666c6471fdb9ce7a0a5fb31bf1a074138466fa8daae50b53199b8e0d7c",
repairFile: "0467_fair_human_torch.sql",
repairSha256: "50ce0eb2567570b098bea1de92fe6a220d139e4ba83438195fbbad1be51e31a1",
} as const;
function createUtilitySql(url: string) {
return postgres(url, { max: 1, onnotice: () => {} });
}
type RegisteredPostgresClient = ReturnType<typeof postgres>;
/**
* Derives a registry key from a connection URL's host and port only. We must
* not retain or log the full URL, because it carries credentials.
*/
function hostPortKey(url: string): string {
const parsed = new URL(url);
return `${parsed.hostname}:${parsed.port || "5432"}`;
}
/**
* Same as `hostPortKey`, but returns `null` instead of throwing when the URL
* does not parse. `postgres(url)` tolerates a value `new URL()` rejects (an
* empty string falls back to the `PG*` environment variables), so `createDb`
* must tolerate it too: skip the registry entry and let the driver decide
* the outcome, instead of throwing an error the driver itself would not.
*/
function hostPortKeyOrNull(url: string): string | null {
try {
return hostPortKey(url);
} catch (error) {
if (error instanceof TypeError && (error as NodeJS.ErrnoException).code === "ERR_INVALID_URL") return null;
throw error;
}
}
// Tracks every client `createDb` hands out, keyed by host and port, so a test
// fixture can end them before it stops the Postgres cluster they point at. A
// `WeakRef` plus `FinalizationRegistry` means a long-lived process (a real
// server) retains nothing extra: an unreferenced client is pruned on its own.
const clientsByHostPort = new Map<string, Set<WeakRef<RegisteredPostgresClient>>>();
const clientFinalizer = new FinalizationRegistry<{ hostPortKey: string; ref: WeakRef<RegisteredPostgresClient> }>(
({ hostPortKey, ref }) => {
const refs = clientsByHostPort.get(hostPortKey);
if (!refs) return;
refs.delete(ref);
if (refs.size === 0) clientsByHostPort.delete(hostPortKey);
},
);
function registerClient(key: string, client: RegisteredPostgresClient): void {
const ref = new WeakRef(client);
let refs = clientsByHostPort.get(key);
if (!refs) {
refs = new Set();
clientsByHostPort.set(key, refs);
}
refs.add(ref);
clientFinalizer.register(client, { hostPortKey: key, ref }, ref);
}
/**
* Ends every live client `createDb` handed out for the given URL's host and
* port, then forgets them. Call this before stopping a Postgres cluster: a
* client that outlives the cluster it points at can crash the process (a
* reserved connection's deferred write firing after the socket is gone).
* Swallows individual `end()` errors so one bad client cannot block the rest.
* An owned database on a shared test host must select database scope instead;
* its teardown cannot close another fixture's clients on that host.
*/
export async function closeRegisteredClients(url: string, options?: { scope: "database" }): Promise<void> {
const key = hostPortKeyOrNull(url);
if (key === null) return;
const database = options?.scope === "database" ? decodeURIComponent(new URL(url).pathname.slice(1)) : undefined;
if (database === "") throw new Error("Database-scoped teardown requires an explicit database name");
const refs = clientsByHostPort.get(key);
if (!refs) return;
const clients: RegisteredPostgresClient[] = [];
for (const ref of refs) {
const client = ref.deref();
if (client && database !== undefined && client.options.database !== database) continue;
refs.delete(ref);
clientFinalizer.unregister(ref);
if (client) clients.push(client);
}
if (refs.size === 0) clientsByHostPort.delete(key);
await Promise.all(clients.map((client) => client.end({ timeout: 1 }).catch(() => {})));
}
function isSafeIdentifier(value: string): boolean {
return /^[A-Za-z_][A-Za-z0-9_]*$/.test(value);
}
function quoteIdentifier(value: string): string {
if (!isSafeIdentifier(value)) throw new Error(`Unsafe SQL identifier: ${value}`);
return `"${value.replaceAll("\"", "\"\"")}"`;
}
function quoteLiteral(value: string): string {
return `'${value.replaceAll("'", "''")}'`;
}
function splitMigrationStatements(content: string): string[] {
return content
.split("--> statement-breakpoint")
.map((statement) => statement.trim())
.filter((statement) => statement.length > 0);
}
export type MigrationState =
| {
status: "upToDate";
tableCount: number;
availableMigrations: string[];
appliedMigrations: string[];
journalEntryCount: number;
}
| {
status: "needsMigrations";
tableCount: number;
availableMigrations: string[];
appliedMigrations: string[];
pendingMigrations: string[];
journalEntryCount: number;
reason: "no-migration-journal-empty-db" | "no-migration-journal-non-empty-db" | "pending-migrations";
};
export interface DatabaseClientOptions {
prepare?: boolean;
/** postgres.js `max` — connection pool size (driver default: 10). */
maxConnections?: number;
/** postgres.js `idle_timeout` in seconds (driver default: disabled). */
idleTimeoutSeconds?: number;
/** postgres.js `connect_timeout` in seconds (driver default: 30). */
connectTimeoutSeconds?: number;
/**
* postgres.js `max_lifetime` in seconds. Bounds how long one pooled
* connection is reused before the client replaces it (driver default: a
* random value between 30 and 60 minutes).
*/
maxLifetimeSeconds?: number;
/**
* postgres.js `connection.application_name`, shown in
* `pg_stat_activity.application_name`. Lets an operator tell Paperclip's
* pool apart from other clients of the same database (driver default:
* `postgres.js`).
*/
applicationName?: string;
/**
* Connection-level `statement_timeout` in milliseconds, sent as a startup
* parameter (no extra round trip). For dedicated single-purpose pools only.
*/
statementTimeoutMs?: number;
}
/**
* Idle pooled connections close after this many seconds unless
* `DATABASE_IDLE_TIMEOUT_SECONDS` says otherwise. The driver default keeps an
* idle connection open forever, so a process that stops issuing queries still
* holds every backend it ever opened. Set `DATABASE_IDLE_TIMEOUT_SECONDS=0`
* to restore the driver default.
*/
export const DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS = 60;
/** `application_name` reported to PostgreSQL unless `DATABASE_APPLICATION_NAME` overrides it. */
export const DEFAULT_DATABASE_APPLICATION_NAME = "paperclip";
function envBoolean(env: NodeJS.ProcessEnv, name: string): boolean | undefined {
const value = env[name]?.trim().toLowerCase();
if (value === undefined || value === "") return undefined;
if (value === "true" || value === "1") return true;
if (value === "false" || value === "0") return false;
throw new Error(`${name} must be "true" or "false", got: ${env[name]}`);
}
function envPositiveInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
const value = env[name]?.trim();
if (value === undefined || value === "") return undefined;
if (!/^[1-9]\d*$/.test(value)) {
throw new Error(`${name} must be a positive integer, got: ${env[name]}`);
}
return Number.parseInt(value, 10);
}
function envNonNegativeInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
const value = env[name]?.trim();
if (value === undefined || value === "") return undefined;
if (!/^(?:0|[1-9]\d*)$/.test(value)) {
throw new Error(`${name} must be a non-negative integer, got: ${env[name]}`);
}
return Number.parseInt(value, 10);
}
function envNonEmptyString(env: NodeJS.ProcessEnv, name: string): string | undefined {
const value = env[name]?.trim();
if (value === undefined || value === "") return undefined;
return value;
}
/**
* Database client tuning from the environment, so hosted deployments can
* adapt to their connection topology (pooled endpoints, network latency)
* without editing source. Every variable is optional. This function returns
* only the values the environment sets; `resolveDatabaseClientOptions` adds
* Paperclip's own defaults on top, and the driver defaults apply to the rest
* — self-hosted setups need none of these.
*/
export function databaseClientOptionsFromEnv(env: NodeJS.ProcessEnv = process.env): DatabaseClientOptions {
const options: DatabaseClientOptions = {};
const prepare = envBoolean(env, "DATABASE_PREPARED_STATEMENTS");
if (prepare !== undefined) options.prepare = prepare;
const maxConnections = envPositiveInteger(env, "DATABASE_POOL_MAX");
if (maxConnections !== undefined) options.maxConnections = maxConnections;
// `0` is allowed here: it disables idle reaping (the driver default).
const idleTimeoutSeconds = envNonNegativeInteger(env, "DATABASE_IDLE_TIMEOUT_SECONDS");
if (idleTimeoutSeconds !== undefined) options.idleTimeoutSeconds = idleTimeoutSeconds;
const connectTimeoutSeconds = envPositiveInteger(env, "DATABASE_CONNECT_TIMEOUT_SECONDS");
if (connectTimeoutSeconds !== undefined) options.connectTimeoutSeconds = connectTimeoutSeconds;
const maxLifetimeSeconds = envPositiveInteger(env, "DATABASE_MAX_LIFETIME_SECONDS");
if (maxLifetimeSeconds !== undefined) options.maxLifetimeSeconds = maxLifetimeSeconds;
const applicationName = envNonEmptyString(env, "DATABASE_APPLICATION_NAME");
if (applicationName !== undefined) options.applicationName = applicationName;
return options;
}
/**
* Fills in Paperclip's defaults for the options the caller left unset: idle
* connections are reaped after `DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS`, and the
* pool identifies itself as `DEFAULT_DATABASE_APPLICATION_NAME`. Everything
* else stays at the driver default. An explicit value (including
* `idleTimeoutSeconds: 0`) always wins over the default.
*/
export function resolveDatabaseClientOptions(options: DatabaseClientOptions): DatabaseClientOptions {
return {
...options,
idleTimeoutSeconds: options.idleTimeoutSeconds ?? DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS,
applicationName: options.applicationName ?? DEFAULT_DATABASE_APPLICATION_NAME,
};
}
export function postgresJsOptions(options: DatabaseClientOptions): Record<string, unknown> {
const driverOptions: Record<string, unknown> = {};
if (options.prepare !== undefined) driverOptions.prepare = options.prepare;
if (options.maxConnections !== undefined) driverOptions.max = options.maxConnections;
if (options.idleTimeoutSeconds !== undefined) driverOptions.idle_timeout = options.idleTimeoutSeconds;
if (options.connectTimeoutSeconds !== undefined) driverOptions.connect_timeout = options.connectTimeoutSeconds;
if (options.maxLifetimeSeconds !== undefined) driverOptions.max_lifetime = options.maxLifetimeSeconds;
const connection: Record<string, string> = {};
if (options.applicationName !== undefined) connection.application_name = options.applicationName;
if (options.statementTimeoutMs !== undefined) connection.statement_timeout = String(options.statementTimeoutMs);
if (Object.keys(connection).length) driverOptions.connection = connection;
return driverOptions;
}
// A long advisory-lock transaction must not borrow the normal pool that its
// work needs for progress writes. Keep connection configuration private to the
// originating Db lifetime; callers receive neither URLs nor credentials.
const dedicatedDbFactories = new WeakMap<object, () => Db>();
const ownedDatabaseListenerFactories = new WeakMap<object, () => Db>();
export interface OwnedDatabaseListener {
listen: Db["$client"]["listen"];
close(): Promise<void>;
}
export function createOwnedDatabaseListener(db: Db): OwnedDatabaseListener {
const factory = ownedDatabaseListenerFactories.get(db);
if (!factory) throw new Error("owned_listener_requires_create_db");
const owned = factory();
return {
listen: owned.$client.listen.bind(owned.$client),
close: () => owned.$client.end({ timeout: 1 }),
};
}
export async function withDedicatedDbConnection<T>(db: Db, action: (dedicated: Db) => Promise<T>): Promise<T> {
const factory = dedicatedDbFactories.get(db);
if (!factory) throw new Error("dedicated_connection_requires_create_db");
const dedicated = factory();
try { return await action(dedicated); }
finally { await dedicated.$client.end({ timeout: 1 }); }
}
// postgres.js pipelines a query onto a busy connection when none is idle, so
// a short liveness write can wait behind an unrelated slow statement with no
// client-side bound. Liveness writes (controller-lease renewal) get a small
// private pool, created on first use and shared for the owner's lifetime.
const livenessDbFactories = new WeakMap<object, () => Db>();
const livenessDbs = new WeakMap<object, Db>();
const LIVENESS_POOL_MAX_CONNECTIONS = 2;
/** The private liveness pool for a `createDb` client; other clients (test
* doubles, transactions) are returned unchanged. */
export function livenessDb(db: Db): Db {
const existing = livenessDbs.get(db);
if (existing) return existing;
const factory = livenessDbFactories.get(db);
if (!factory) return db;
const created = factory();
livenessDbs.set(db, created);
return created;
}
const controllerLeaseDbFactories = new WeakMap<object, () => Db>();
const controllerLeaseDbs = new WeakMap<object, Db>();
export function controllerLeaseDb(db: Db): Db {
const existing = controllerLeaseDbs.get(db);
if (existing) return existing;
const factory = controllerLeaseDbFactories.get(db);
if (!factory) return db;
const created = factory();
controllerLeaseDbs.set(db, created);
return created;
}
const controllerInstanceLeaseDbFactories = new WeakMap<object, (statementTimeoutMs: number | null) => Db>();
const controllerInstanceLeaseDbs = new WeakMap<object, Db>();
/** One private connection for the server's controller-instance lease, so its
* renewal never waits behind request or worker traffic on the main pool.
* `statementTimeoutMs` (first call wins) is a connection-level
* `statement_timeout`: the server cancels a slow renewal, so it never holds the
* single connection that the next renewal needs, at no extra round trip. */
export function controllerInstanceLeaseDb(db: Db, options: { statementTimeoutMs?: number | null } = {}): Db {
const existing = controllerInstanceLeaseDbs.get(db);
if (existing) return existing;
const factory = controllerInstanceLeaseDbFactories.get(db);
if (!factory) return db;
const created = factory(options.statementTimeoutMs ?? null);
controllerInstanceLeaseDbs.set(db, created);
return created;
}
/** Replaces the lease connection with one that sends no `statement_timeout`
* startup parameter, for a connection pooler that refuses it. */
export function controllerInstanceLeaseDbWithoutStatementTimeout(db: Db): Db {
const factory = controllerInstanceLeaseDbFactories.get(db);
if (!factory) return db;
const previous = controllerInstanceLeaseDbs.get(db);
const created = factory(null);
controllerInstanceLeaseDbs.set(db, created);
void previous?.$client.end({ timeout: 0 }).catch(() => undefined);
return created;
}
/** Ends the liveness pool if one was created. The ended pool stays cached, so
* a late renewal fails like a query on the ended primary pool. */
export async function closeLivenessDb(db: Db, timeoutSeconds: number): Promise<void> {
await Promise.all([livenessDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
controllerLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
controllerInstanceLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds })]);
}
// Control checkpoints must not occupy the request or lease-renewal pools
// during an aligned burst. Share at most three lazy connections per owner.
const checkpointDbFactories = new WeakMap<object, () => Db>();
const checkpointDbs = new WeakMap<object, Db>();
const CHECKPOINT_POOL_MAX_CONNECTIONS = 3;
const closedCheckpointOwners = new WeakSet<object>();
/** The bounded controller checkpoint pool; transactions and test doubles keep
* their existing database when they have no owning client factory. */
export function checkpointDb(db: Db): Db {
const existing = checkpointDbs.get(db);
if (existing) return existing;
const factory = checkpointDbFactories.get(db);
if (!factory) return db;
if (closedCheckpointOwners.has(db)) throw new Error("checkpoint_pool_closed");
const created = factory();
checkpointDbs.set(db, created);
return created;
}
/** Fence lazy creation immediately, and keep any ended pool cached. */
export async function closeCheckpointDb(db: Db, timeoutSeconds: number): Promise<void> {
closedCheckpointOwners.add(db);
await checkpointDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
}
// Managed file publication holds an agent lock while transferring bytes. A
// stalled object store must not consume request or controller-renewal capacity.
const fileStorageDbFactories = new WeakMap<object, () => Db>();
const fileStorageDbs = new WeakMap<object, Db>();
const closedFileStorageOwners = new WeakSet<object>();
export function fileStorageDb(db: Db): Db {
const existing = fileStorageDbs.get(db);
if (existing) return existing;
const factory = fileStorageDbFactories.get(db);
if (!factory) return db; // Preserve the caller's transaction/savepoint.
if (closedFileStorageOwners.has(db)) throw new Error("file_storage_pool_closed");
const created = factory();
fileStorageDbs.set(db, created);
fileStorageDbs.set(created, created);
return created;
}
export async function closeFileStorageDb(db: Db, timeoutSeconds: number): Promise<void> {
closedFileStorageOwners.add(db);
await fileStorageDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
}
export function createDb(url: string, options?: DatabaseClientOptions) {
const resolved = resolveDatabaseClientOptions(options ?? databaseClientOptionsFromEnv());
return createDbClient(url, resolved);
}
function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: object) {
const sql = postgres(url, postgresJsOptions(resolved));
if (resolved.prepare === true) installPreparedQueries(sql);
const key = hostPortKeyOrNull(url);
if (key) registerClient(key, sql);
// A disconnect can lose the response after a statement has committed.
// postgres.js calls that error "write CONNECTION_CLOSED" too, so the
// message cannot establish that replay is safe. Leave retries to callers
// that know the complete operation is idempotent.
const db = installDatabaseWorkSignals(drizzleWithLazyTransactionQueries(sql, schema), owner);
installDatabaseResponseCapture(db, schema);
dedicatedDbFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: 1, applicationName: "paperclip-workspace-finalization-lock",
}, db));
ownedDatabaseListenerFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: 1,
}, db));
livenessDbFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: LIVENESS_POOL_MAX_CONNECTIONS, applicationName: "paperclip-controller-lease",
}, db));
controllerLeaseDbFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: 1, applicationName: "paperclip-legacy-controller-lease",
}, db));
controllerInstanceLeaseDbFactories.set(db, statementTimeoutMs => createDbClient(url, {
...resolved, maxConnections: 1, applicationName: "paperclip-controller-instance-lease",
...(statementTimeoutMs !== null ? { statementTimeoutMs } : {}),
}, db));
checkpointDbFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: CHECKPOINT_POOL_MAX_CONNECTIONS, applicationName: "paperclip-runner-checkpoint",
}, db));
fileStorageDbFactories.set(db, () => createDbClient(url, {
...resolved, maxConnections: 2, applicationName: "paperclip-file-storage",
}, db));
return db;
}
export async function getPostgresDataDirectory(url: string): Promise<string | null> {
const sql = createUtilitySql(url);
try {
const rows = await sql<{ data_directory: string | null }[]>`
SELECT current_setting('data_directory', true) AS data_directory
`;
const actual = rows[0]?.data_directory;
return typeof actual === "string" && actual.length > 0 ? actual : null;
} catch {
return null;
} finally {
await sql.end();
}
}
async function listMigrationFiles(): Promise<string[]> {
const entries = await readdir(MIGRATIONS_FOLDER, { withFileTypes: true });
return entries
.filter((entry) => entry.isFile() && entry.name.endsWith(".sql"))
.map((entry) => entry.name)
.sort((a, b) => a.localeCompare(b));
}
type MigrationJournalFile = {
entries?: Array<{ idx?: number; tag?: string; when?: number }>;
};
type JournalMigrationEntry = {
fileName: string;
folderMillis: number;
order: number;
};
async function listJournalMigrationEntries(): Promise<JournalMigrationEntry[]> {
try {
const raw = await readFile(MIGRATIONS_JOURNAL_JSON, "utf8");
const parsed = JSON.parse(raw) as MigrationJournalFile;
if (!Array.isArray(parsed.entries)) return [];
return parsed.entries
.map((entry, entryIndex) => {
if (typeof entry?.tag !== "string") return null;
if (typeof entry?.when !== "number" || !Number.isFinite(entry.when)) return null;
const order = Number.isInteger(entry.idx) ? Number(entry.idx) : entryIndex;
return { fileName: `${entry.tag}.sql`, folderMillis: entry.when, order };
})
.filter((entry): entry is JournalMigrationEntry => entry !== null);
} catch {
return [];
}
}
async function listJournalMigrationFiles(): Promise<string[]> {
const entries = await listJournalMigrationEntries();
return entries.map((entry) => entry.fileName);
}
async function readMigrationFileContent(migrationFile: string): Promise<string> {
return readFile(new URL(`./migrations/${migrationFile}`, import.meta.url), "utf8");
}
async function orderMigrationsByJournal(migrationFiles: string[]): Promise<string[]> {
const journalEntries = await listJournalMigrationEntries();
const orderByFileName = new Map(journalEntries.map((entry) => [entry.fileName, entry.order]));
return [...migrationFiles].sort((left, right) => {
const leftOrder = orderByFileName.get(left);
const rightOrder = orderByFileName.get(right);
if (leftOrder === undefined && rightOrder === undefined) return left.localeCompare(right);
if (leftOrder === undefined) return 1;
if (rightOrder === undefined) return -1;
if (leftOrder === rightOrder) return left.localeCompare(right);
return leftOrder - rightOrder;
});
}
type SqlExecutor = Pick<ReturnType<typeof postgres>, "unsafe">;
async function runInTransaction(sql: SqlExecutor, action: () => Promise<void>): Promise<void> {
await sql.unsafe("BEGIN");
try {
await action();
await sql.unsafe("COMMIT");
} catch (error) {
try {
await sql.unsafe("ROLLBACK");
} catch {
// Ignore rollback failures and surface the original error.
}
throw error;
}
}
async function latestMigrationCreatedAt(
sql: SqlExecutor,
qualifiedTable: string,
): Promise<number | null> {
const rows = await sql.unsafe<{ created_at: string | number | null }[]>(
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC NULLS LAST LIMIT 1`,
);
const value = Number(rows[0]?.created_at ?? Number.NaN);
return Number.isFinite(value) ? value : null;
}
function normalizeFolderMillis(value: number | null | undefined): number {
if (typeof value === "number" && Number.isFinite(value) && value >= 0) {
return Math.trunc(value);
}
return Date.now();
}
async function ensureMigrationJournalTable(
sql: ReturnType<typeof postgres>,
): Promise<{ migrationTableSchema: string; columnNames: Set<string> }> {
let migrationTableSchema = await discoverMigrationTableSchema(sql);
if (!migrationTableSchema) {
const drizzleSchema = quoteIdentifier("drizzle");
const migrationTable = quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE);
await sql.unsafe(`CREATE SCHEMA IF NOT EXISTS ${drizzleSchema}`);
await sql.unsafe(
`CREATE TABLE IF NOT EXISTS ${drizzleSchema}.${migrationTable} (id SERIAL PRIMARY KEY, hash text NOT NULL, created_at bigint)`,
);
migrationTableSchema = (await discoverMigrationTableSchema(sql)) ?? "drizzle";
}
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
return { migrationTableSchema, columnNames };
}
async function migrationHistoryEntryExists(
sql: SqlExecutor,
qualifiedTable: string,
columnNames: Set<string>,
migrationFile: string,
hash: string,
): Promise<boolean> {
const predicates: string[] = [];
if (columnNames.has("hash")) predicates.push(`hash = ${quoteLiteral(hash)}`);
if (columnNames.has("name")) predicates.push(`name = ${quoteLiteral(migrationFile)}`);
if (predicates.length === 0) return false;
const rows = await sql.unsafe<{ one: number }[]>(
`SELECT 1 AS one FROM ${qualifiedTable} WHERE ${predicates.join(" OR ")} LIMIT 1`,
);
return rows.length > 0;
}
async function recordMigrationHistoryEntry(
sql: SqlExecutor,
qualifiedTable: string,
columnNames: Set<string>,
migrationFile: string,
hash: string,
folderMillis: number,
): Promise<void> {
const insertColumns: string[] = [];
const insertValues: string[] = [];
if (columnNames.has("hash")) {
insertColumns.push(quoteIdentifier("hash"));
insertValues.push(quoteLiteral(hash));
}
if (columnNames.has("name")) {
insertColumns.push(quoteIdentifier("name"));
insertValues.push(quoteLiteral(migrationFile));
}
if (columnNames.has("created_at")) {
const latestCreatedAt = await latestMigrationCreatedAt(sql, qualifiedTable);
const createdAt = latestCreatedAt === null
? normalizeFolderMillis(folderMillis)
: Math.max(latestCreatedAt + 1, normalizeFolderMillis(folderMillis));
insertColumns.push(quoteIdentifier("created_at"));
insertValues.push(quoteLiteral(String(createdAt)));
}
if (insertColumns.length === 0) return;
await sql.unsafe(
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
);
}
// These idempotent migrations require top-level statements for batch commits
// or concurrent index builds. History is recorded only after completion.
const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
"0313_private_task_access.sql",
"0314_private_task_draft_assets.sql",
"0332_little_night_nurse.sql",
"0345_luxuriant_carmella_unuscione.sql",
"0369_late_logan.sql",
"0381_mean_scream.sql",
"0382_narrow_the_executioner.sql",
"0385_open_madame_web.sql",
"0386_steady_epoch.sql",
"0387_boring_professor_monster.sql",
"0388_ambitious_micromax.sql",
"0389_brave_hulk.sql",
"0390_salty_killraven.sql",
"0391_third_roulette.sql",
"0392_small_frightful_four.sql",
"0393_open_grandmaster.sql",
"0395_tranquil_firestar.sql",
"0396_tan_paladin.sql",
"0397_known_madrox.sql",
"0398_yielding_random.sql",
"0399_ambitious_onslaught.sql",
"0400_absent_phantom_reporter.sql",
"0401_yummy_ultimates.sql",
"0402_ancient_malice.sql",
"0403_natural_cerise.sql",
"0406_zippy_war_machine.sql",
"0407_greedy_star_brand.sql",
"0408_short_red_skull.sql",
"0409_uneven_zuras.sql",
"0410_freezing_kinsey_walden.sql",
"0411_real_scrambler.sql",
"0412_tearful_squadron_supreme.sql",
"0413_tired_kronos.sql",
"0414_closed_edwin_jarvis.sql",
"0419_slippery_impossible_man.sql",
"0432_strong_mole_man.sql",
"0435_premium_puff_adder.sql",
"0436_legal_reavers.sql",
"0447_legacy_controller_process.sql",
"0450_petite_frightful_four.sql",
"0453_curious_ben_parker.sql",
"0454_lumpy_ronan.sql",
"0455_pending_budget_cancellation_index.sql",
"0456_chat_review_lookup_indexes.sql",
"0457_native_session_run_index.sql",
"0458_chat_admission_issue_index.sql",
"0459_environment_lease_backup_scope_index.sql",
"0460_workspace_activity_context_index.sql",
"0461_created_from_legacy_origin_run_index.sql",
"0470_heartbeat_runs_queued_created_index.sql",
"0472_runner_api_capture_pending_deletes.sql",
"0476_wakeup_request_run_index.sql",
"0469_zippy_photon.sql",
"0384_funny_adam_warlock.sql",
"0383_tool_review_company_expiry.sql",
]);
async function applyPendingMigrationsManually(
url: string,
pendingMigrations: string[],
): Promise<void> {
if (pendingMigrations.length === 0) return;
const orderedPendingMigrations = await orderMigrationsByJournal(pendingMigrations);
const journalEntries = await listJournalMigrationEntries();
const folderMillisByFileName = new Map(
journalEntries.map((entry) => [entry.fileName, normalizeFolderMillis(entry.folderMillis)]),
);
const pool = createUtilitySql(url);
try {
const sql = await pool.reserve();
try {
// A session lock survives each batch COMMIT. Reserve the connection so
// pool rotation cannot release it while another migrator is waiting.
// A blocking lock SELECT retains its statement snapshot while waiting.
// A partial concurrent index build can wait for that snapshot, forming a
// deadlock with this lock. Each failed try ends its snapshot before sleep.
while (!(await sql<{ locked: boolean }[]>`
SELECT pg_try_advisory_lock(hashtextextended('paperclip:migrations', 0)) AS locked`)[0]?.locked) {
await new Promise((resolve) => setTimeout(resolve, 100));
}
const { migrationTableSchema, columnNames } = await ensureMigrationJournalTable(sql);
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
for (const migrationFile of orderedPendingMigrations) {
const migrationContent = await readMigrationFileContent(migrationFile);
const hash = createHash("sha256").update(migrationContent).digest("hex");
const existingEntry = await migrationHistoryEntryExists(
sql,
qualifiedTable,
columnNames,
migrationFile,
hash,
);
if (existingEntry) continue;
const applyMigration = async () => {
for (const statement of splitMigrationStatements(migrationContent)) {
if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
// A cancelled concurrent build leaves an invalid index. IF NOT
// EXISTS alone would skip it on retry and journal an incomplete index.
const index = statement.replace(/^\s*--.*$/gm, "").trim()
.match(/^CREATE (?:UNIQUE )?INDEX CONCURRENTLY IF NOT EXISTS "([A-Za-z_][A-Za-z0-9_]*)"/i);
if (index) {
const invalid = await sql<{ invalid: boolean }[]>`
SELECT NOT i.indisvalid AS invalid FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public' AND c.relname = ${index[1]}`;
if (invalid[0]?.invalid) {
await sql.unsafe(`DROP INDEX CONCURRENTLY ${quoteIdentifier("public")}.${quoteIdentifier(index[1])}`);
}
}
}
// Older dev schemas can lack a replaced constraint. Preserve the
// published SQL/hash while allowing the harmless drop to proceed.
const executable = statement.replace(
/^(\s*ALTER TABLE "[^"]+" DROP CONSTRAINT) (?!IF EXISTS\b)/i,
"$1 IF EXISTS ",
);
await sql.unsafe(executable);
}
await recordMigrationHistoryEntry(
sql,
qualifiedTable,
columnNames,
migrationFile,
hash,
folderMillisByFileName.get(migrationFile) ?? Date.now(),
);
};
if (migrationFile === PROJECT_ENVIRONMENT_OVERLAP.dropFile) {
const repairFile = PROJECT_ENVIRONMENT_OVERLAP.repairFile;
const repairContent = await readMigrationFileContent(repairFile);
const repairHash = createHash("sha256").update(repairContent).digest("hex");
const dropIndex = orderedPendingMigrations.indexOf(migrationFile);
if (hash !== PROJECT_ENVIRONMENT_OVERLAP.dropSha256
|| repairHash !== PROJECT_ENVIRONMENT_OVERLAP.repairSha256
|| orderedPendingMigrations[dropIndex + 1] !== repairFile
|| await migrationHistoryEntryExists(sql, qualifiedTable, columnNames, repairFile, repairHash)) {
throw new Error("Project environment migration pair does not match the qualified overlap repair");
}
await runInTransaction(sql, async () => {
await sql.unsafe('LOCK TABLE "public"."projects" IN ACCESS EXCLUSIVE MODE');
await sql.unsafe(`CREATE TEMP TABLE paperclip_project_environment_overlap ON COMMIT DROP AS
SELECT id, environment_id FROM "public"."projects" WHERE environment_id IS NOT NULL`);
await applyMigration();
for (const statement of splitMigrationStatements(repairContent)) await sql.unsafe(statement);
await sql.unsafe(`UPDATE "public"."projects" AS project SET environment_id = legacy.environment_id
FROM pg_temp.paperclip_project_environment_overlap AS legacy WHERE project.id = legacy.id`);
await recordMigrationHistoryEntry(sql, qualifiedTable, columnNames, repairFile, repairHash,
folderMillisByFileName.get(repairFile) ?? Date.now());
});
} else if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
await applyMigration();
} else {
await runInTransaction(sql, applyMigration);
}
}
} finally {
try {
await sql`SELECT pg_advisory_unlock(hashtextextended('paperclip:migrations', 0))`;
} catch {
// Preserve the migration error if the session died; PostgreSQL releases
// its lock on disconnect, and pool.end() closes any remaining connection.
}
sql.release();
}
} finally {
await pool.end();
}
}
async function mapHashesToMigrationFiles(migrationFiles: string[]): Promise<Map<string, string>> {
const mapped = new Map<string, string>();
await Promise.all(
migrationFiles.map(async (migrationFile) => {
const content = await readMigrationFileContent(migrationFile);
const hash = createHash("sha256").update(content).digest("hex");
mapped.set(hash, migrationFile);
if (migrationFile === "0355_agent_tool_definition_hooks.sql") {
mapped.set("7ac76adac0220da522c78e51da20490aeb0f90512b4370a05af9f8dd0f75eda6", migrationFile);
}
}),
);
return mapped;
}
async function getMigrationTableColumnNames(
sql: ReturnType<typeof postgres>,
migrationTableSchema: string,
): Promise<Set<string>> {
const columns = await sql.unsafe<{ column_name: string }[]>(
`
SELECT column_name
FROM information_schema.columns
WHERE table_schema = ${quoteLiteral(migrationTableSchema)}
AND table_name = ${quoteLiteral(DRIZZLE_MIGRATIONS_TABLE)}
`,
);
return new Set(columns.map((column) => column.column_name));
}
async function tableExists(
sql: ReturnType<typeof postgres>,
tableName: string,
): Promise<boolean> {
const rows = await sql<{ exists: boolean }[]>`
SELECT EXISTS (
SELECT 1
FROM information_schema.tables
WHERE table_schema = 'public'
AND table_name = ${tableName}
) AS exists
`;
return rows[0]?.exists ?? false;
}
async function columnExists(
sql: ReturnType<typeof postgres>,
tableName: string,
columnName: string,
): Promise<boolean> {
const rows = await sql<{ exists: boolean }[]>`
SELECT EXISTS (
SELECT 1
FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = ${tableName}
AND column_name = ${columnName}
) AS exists
`;
return rows[0]?.exists ?? false;
}
async function columnHasDataType(
sql: ReturnType<typeof postgres>,
tableName: string,
columnName: string,
dataType: string,
): Promise<boolean> {
const rows = await sql<{ dataType: string; udtName: string }[]>`
SELECT data_type AS "dataType", udt_name AS "udtName"
FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = ${tableName}
AND column_name = ${columnName}
`;
const expected = dataType.toLowerCase();
return rows.some((row) => (
row.dataType.toLowerCase() === expected || row.udtName.toLowerCase() === expected
));
}
async function indexExists(
sql: ReturnType<typeof postgres>,
indexName: string,
): Promise<boolean> {
const rows = await sql<{ exists: boolean }[]>`
SELECT EXISTS (
SELECT 1
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public'
AND c.relkind = 'i'
AND c.relname = ${indexName}
) AS exists
`;
return rows[0]?.exists ?? false;
}
async function constraintExists(
sql: ReturnType<typeof postgres>,
constraintName: string,
): Promise<boolean> {
const rows = await sql<{ exists: boolean }[]>`
SELECT EXISTS (
SELECT 1
FROM pg_constraint c
JOIN pg_namespace n ON n.oid = c.connamespace
WHERE n.nspname = 'public'
AND c.conname = ${constraintName}
) AS exists
`;
return rows[0]?.exists ?? false;
}
async function functionExists(
sql: ReturnType<typeof postgres>,
functionName: string,
): Promise<boolean> {
const rows = await sql<{ exists: boolean }[]>`
SELECT EXISTS (
SELECT 1
FROM pg_proc p
JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname = 'public'
AND p.proname = ${functionName}
) AS exists
`;
return rows[0]?.exists ?? false;
}
async function triggerExists(
sql: ReturnType<typeof postgres>,
triggerName: string,
): Promise<boolean> {
const rows = await sql<{ exists: boolean }[]>`
SELECT EXISTS (
SELECT 1
FROM pg_trigger t
JOIN pg_class c ON c.oid = t.tgrelid
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public'
AND t.tgname = ${triggerName}
AND NOT t.tgisinternal
) AS exists
`;
return rows[0]?.exists ?? false;
}
async function heartbeatEventSequencesAreUnique(
sql: ReturnType<typeof postgres>,
): Promise<boolean> {
const rows = await sql<{ unique: boolean }[]>`
SELECT NOT EXISTS (
SELECT 1
FROM heartbeat_run_events
GROUP BY run_id, seq
HAVING count(*) > 1
) AS unique
`;
return rows[0]?.unique ?? false;
}
async function heartbeatNextEventSequencesAreCurrent(
sql: ReturnType<typeof postgres>,
): Promise<boolean> {
const rows = await sql<{ current: boolean }[]>`
SELECT NOT EXISTS (
SELECT 1
FROM heartbeat_runs run
WHERE run.next_event_seq IS DISTINCT FROM COALESCE((
SELECT max(event.seq) + 1
FROM heartbeat_run_events event
WHERE event.run_id = run.id
), 1)
) AS current
`;
return rows[0]?.current ?? false;
}
async function migrationStatementAlreadyApplied(
sql: ReturnType<typeof postgres>,
statement: string,
): Promise<boolean> {
const normalized = statement
.replace(/^\s*--.*$/gm, "")
.replace(/\s+/g, " ")
.trim();
const createTableMatch = normalized.match(/^CREATE TABLE(?: IF NOT EXISTS)? "([^"]+)"/i);
if (createTableMatch) {
return tableExists(sql, createTableMatch[1]);
}
const addColumnMatch = normalized.match(
/^ALTER TABLE "([^"]+)" ADD COLUMN(?: IF NOT EXISTS)? "([^"]+)"/i,
);
if (addColumnMatch) {
return columnExists(sql, addColumnMatch[1], addColumnMatch[2]);
}
const alterColumnTypeMatch = normalized.match(
/^ALTER TABLE "([^"]+)" ALTER COLUMN "([^"]+)" SET DATA TYPE ([A-Za-z0-9_]+)/i,
);
if (alterColumnTypeMatch) {
return columnHasDataType(
sql,
alterColumnTypeMatch[1],
alterColumnTypeMatch[2],
alterColumnTypeMatch[3],
);
}
const createIndexMatch = normalized.match(/^CREATE (?:UNIQUE )?INDEX(?: IF NOT EXISTS)? "([^"]+)"/i);
if (createIndexMatch) {
return indexExists(sql, createIndexMatch[1]);
}
const addConstraintMatch = normalized.match(/^ALTER TABLE "([^"]+)" ADD CONSTRAINT "([^"]+)"/i);
if (addConstraintMatch) {
return constraintExists(sql, addConstraintMatch[2]);
}
const createFunctionMatch = normalized.match(
/^CREATE OR REPLACE FUNCTION "?([A-Za-z_][A-Za-z0-9_]*)"?\s*\(/i,
);
if (createFunctionMatch) {
return functionExists(sql, createFunctionMatch[1]);
}
const createTriggerMatch = normalized.match(
/^CREATE TRIGGER "?([A-Za-z_][A-Za-z0-9_]*)"?/i,
);
if (createTriggerMatch) {
return triggerExists(sql, createTriggerMatch[1]);
}
// These native-runner repairs have persistent postconditions. Verify them
// instead of replaying them when a restored database is missing only the
// migration-history row.
if (
normalized.startsWith("WITH ranked AS (")
&& normalized.includes('UPDATE "heartbeat_run_events" AS event')
) {
return heartbeatEventSequencesAreUnique(sql);
}
if (
normalized.startsWith('UPDATE "heartbeat_runs" AS run')
&& normalized.includes('SET "next_event_seq" = COALESCE')
) {
return heartbeatNextEventSequencesAreCurrent(sql);
}
// If we cannot reason about a statement safely, require manual migration.
return false;
}
async function migrationContentAlreadyApplied(
sql: ReturnType<typeof postgres>,
migrationContent: string,
): Promise<boolean> {
const statements = splitMigrationStatements(migrationContent);
if (statements.length === 0) return false;
for (const statement of statements) {
const applied = await migrationStatementAlreadyApplied(sql, statement);
if (!applied) return false;
}
return true;
}
async function loadAppliedMigrations(
sql: ReturnType<typeof postgres>,
migrationTableSchema: string,
availableMigrations: string[],
): Promise<string[]> {
const quotedSchema = quoteIdentifier(migrationTableSchema);
const qualifiedTable = `${quotedSchema}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
if (columnNames.has("name")) {
const rows = await sql.unsafe<{ name: string }[]>(`SELECT name FROM ${qualifiedTable} ORDER BY id`);
return rows.map((row) => row.name).filter((name): name is string => Boolean(name));
}
if (columnNames.has("hash")) {
const rows = await sql.unsafe<{ hash: string }[]>(`SELECT hash FROM ${qualifiedTable} ORDER BY id`);
const hashesToMigrationFiles = await mapHashesToMigrationFiles(availableMigrations);
const appliedFromHashes = rows
.map((row) => hashesToMigrationFiles.get(row.hash))
.filter((name): name is string => Boolean(name));
if (appliedFromHashes.length > 0) {
// Best-effort: when all hashes resolve, this is authoritative.
if (appliedFromHashes.length === rows.length) return appliedFromHashes;
// Partial hash resolution can happen when files have changed; return what we can trust.
return appliedFromHashes;
}
// Fallback only when hashes are unavailable/unresolved.
if (columnNames.has("created_at")) {
const journalEntries = await listJournalMigrationEntries();
if (journalEntries.length > 0) {
const lastDbRows = await sql.unsafe<{ created_at: string | number | null }[]>(
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC LIMIT 1`,
);
const lastCreatedAt = Number(lastDbRows[0]?.created_at ?? -1);
if (Number.isFinite(lastCreatedAt) && lastCreatedAt >= 0) {
return journalEntries
.filter((entry) => availableMigrations.includes(entry.fileName))
.filter((entry) => entry.folderMillis <= lastCreatedAt)
.map((entry) => entry.fileName)
.slice(0, rows.length);
}
}
}
}
const rows = await sql.unsafe<{ id: number }[]>(`SELECT id FROM ${qualifiedTable} ORDER BY id`);
const journalMigrationFiles = await listJournalMigrationFiles();
const appliedFromIds = rows
.map((row) => journalMigrationFiles[row.id - 1])
.filter((name): name is string => Boolean(name));
if (appliedFromIds.length > 0) return appliedFromIds;
return availableMigrations.slice(0, Math.max(0, rows.length));
}
export type MigrationHistoryReconcileResult = {
repairedMigrations: string[];
remainingMigrations: string[];
};
export async function reconcilePendingMigrationHistory(
url: string,
): Promise<MigrationHistoryReconcileResult> {
const state = await inspectMigrations(url);
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
return { repairedMigrations: [], remainingMigrations: [] };
}
const sql = createUtilitySql(url);
const repairedMigrations: string[] = [];
try {
const journalEntries = await listJournalMigrationEntries();
const folderMillisByFile = new Map(journalEntries.map((entry) => [entry.fileName, entry.folderMillis]));
const migrationTableSchema = await discoverMigrationTableSchema(sql);
if (!migrationTableSchema) {
return { repairedMigrations, remainingMigrations: state.pendingMigrations };
}
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
for (const migrationFile of state.pendingMigrations) {
const migrationContent = await readMigrationFileContent(migrationFile);
const alreadyApplied = await migrationContentAlreadyApplied(sql, migrationContent);
if (!alreadyApplied) break;
const hash = createHash("sha256").update(migrationContent).digest("hex");
const folderMillis = folderMillisByFile.get(migrationFile) ?? Date.now();
const existingByHash = columnNames.has("hash")
? await sql.unsafe<{ created_at: string | number | null }[]>(
`SELECT created_at FROM ${qualifiedTable} WHERE hash = ${quoteLiteral(hash)} ORDER BY created_at DESC LIMIT 1`,
)
: [];
const existingByName = columnNames.has("name")
? await sql.unsafe<{ created_at: string | number | null }[]>(
`SELECT created_at FROM ${qualifiedTable} WHERE name = ${quoteLiteral(migrationFile)} ORDER BY created_at DESC LIMIT 1`,
)
: [];
if (existingByHash.length > 0 || existingByName.length > 0) {
if (columnNames.has("created_at")) {
const existingHashCreatedAt = Number(existingByHash[0]?.created_at ?? -1);
if (existingByHash.length > 0 && Number.isFinite(existingHashCreatedAt) && existingHashCreatedAt < folderMillis) {
await sql.unsafe(
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE hash = ${quoteLiteral(hash)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
);
}
const existingNameCreatedAt = Number(existingByName[0]?.created_at ?? -1);
if (existingByName.length > 0 && Number.isFinite(existingNameCreatedAt) && existingNameCreatedAt < folderMillis) {
await sql.unsafe(
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE name = ${quoteLiteral(migrationFile)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
);
}
}
repairedMigrations.push(migrationFile);
continue;
}
const insertColumns: string[] = [];
const insertValues: string[] = [];
if (columnNames.has("hash")) {
insertColumns.push(quoteIdentifier("hash"));
insertValues.push(quoteLiteral(hash));
}
if (columnNames.has("name")) {
insertColumns.push(quoteIdentifier("name"));
insertValues.push(quoteLiteral(migrationFile));
}
if (columnNames.has("created_at")) {
insertColumns.push(quoteIdentifier("created_at"));
insertValues.push(quoteLiteral(String(folderMillis)));
}
if (insertColumns.length === 0) break;
await sql.unsafe(
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
);
repairedMigrations.push(migrationFile);
}
} finally {
await sql.end();
}
const refreshed = await inspectMigrations(url);
return {
repairedMigrations,
remainingMigrations:
refreshed.status === "needsMigrations" ? refreshed.pendingMigrations : [],
};
}
async function discoverMigrationTableSchema(sql: ReturnType<typeof postgres>): Promise<string | null> {
const rows = await sql<{ schemaName: string }[]>`
SELECT n.nspname AS "schemaName"
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE c.relname = ${DRIZZLE_MIGRATIONS_TABLE} AND c.relkind = 'r'
`;
if (rows.length === 0) return null;
const drizzleSchema = rows.find(({ schemaName }) => schemaName === "drizzle");
if (drizzleSchema) return drizzleSchema.schemaName;
const publicSchema = rows.find(({ schemaName }) => schemaName === "public");
if (publicSchema) return publicSchema.schemaName;
return rows[0]?.schemaName ?? null;
}
export async function inspectMigrations(url: string): Promise<MigrationState> {
const sql = createUtilitySql(url);
try {
const availableMigrations = await listMigrationFiles();
const tableCountResult = await sql<{ count: number }[]>`
select count(*)::int as count
from information_schema.tables
where table_schema = 'public'
and table_type = 'BASE TABLE'
`;
const tableCount = tableCountResult[0]?.count ?? 0;
const migrationTableSchema = await discoverMigrationTableSchema(sql);
if (!migrationTableSchema) {
if (tableCount > 0) {
return {
status: "needsMigrations",
tableCount,
availableMigrations,
appliedMigrations: [],
pendingMigrations: availableMigrations,
journalEntryCount: 0,
reason: "no-migration-journal-non-empty-db",
};
}
return {
status: "needsMigrations",
tableCount,
availableMigrations,
appliedMigrations: [],
pendingMigrations: availableMigrations,
journalEntryCount: 0,
reason: "no-migration-journal-empty-db",
};
}
const qualifiedMigrationTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
const journalCountRows = await sql.unsafe<{ count: number }[]>(
`SELECT count(*)::int AS count FROM ${qualifiedMigrationTable}`,
);
const journalEntryCount = journalCountRows[0]?.count ?? 0;
const appliedMigrations = await loadAppliedMigrations(sql, migrationTableSchema, availableMigrations);
const pendingMigrations = availableMigrations.filter((name) => !appliedMigrations.includes(name));
if (pendingMigrations.length === 0) {
return {
status: "upToDate",
tableCount,
availableMigrations,
appliedMigrations,
journalEntryCount,
};
}
return {
status: "needsMigrations",
tableCount,
availableMigrations,
appliedMigrations,
pendingMigrations,
journalEntryCount,
reason: "pending-migrations",
};
} finally {
await sql.end();
}
}
export async function applyPendingMigrations(url: string): Promise<void> {
const initialState = await inspectMigrations(url);
if (initialState.status === "upToDate") return;
if (initialState.reason === "no-migration-journal-empty-db") {
// Use the same per-file executor for bootstrap and upgrades: Drizzle wraps
// the whole journal in one transaction and cannot run committing backfills.
await applyPendingMigrationsManually(url, initialState.pendingMigrations);
const bootstrappedState = await inspectMigrations(url);
if (bootstrappedState.status === "upToDate") return;
throw new Error(
`Failed to bootstrap migrations: ${bootstrappedState.pendingMigrations.join(", ")}`,
);
}
if (initialState.reason === "no-migration-journal-non-empty-db") {
throw new Error(
"Database has tables but no migration journal; automatic migration is unsafe. Initialize migration history manually.",
);
}
let state = await inspectMigrations(url);
if (state.status === "upToDate") return;
const repair = await reconcilePendingMigrationHistory(url);
if (repair.repairedMigrations.length > 0) {
state = await inspectMigrations(url);
if (state.status === "upToDate") return;
}
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
throw new Error("Migrations are still pending after migration-history reconciliation; run inspectMigrations for details.");
}
await applyPendingMigrationsManually(url, state.pendingMigrations);
const finalState = await inspectMigrations(url);
if (finalState.status !== "upToDate") {
throw new Error(
`Failed to apply pending migrations: ${finalState.pendingMigrations.join(", ")}`,
);
}
}
export type MigrationBootstrapResult =
| { migrated: true; reason: "migrated-empty-db"; tableCount: 0 }
| { migrated: false; reason: "already-migrated"; tableCount: number }
| { migrated: false; reason: "not-empty-no-migration-journal"; tableCount: number };
export async function migratePostgresIfEmpty(url: string): Promise<MigrationBootstrapResult> {
const sql = createUtilitySql(url);
try {
const migrationTableSchema = await discoverMigrationTableSchema(sql);
const tableCountResult = await sql<{ count: number }[]>`
select count(*)::int as count
from information_schema.tables
where table_schema = 'public'
and table_type = 'BASE TABLE'
`;
const tableCount = tableCountResult[0]?.count ?? 0;
if (migrationTableSchema) {
return { migrated: false, reason: "already-migrated", tableCount };
}
if (tableCount > 0) {
return { migrated: false, reason: "not-empty-no-migration-journal", tableCount };
}
await applyPendingMigrationsManually(url, await listMigrationFiles());
return { migrated: true, reason: "migrated-empty-db", tableCount: 0 };
} finally {
await sql.end();
}
}
export async function ensurePostgresDatabase(
url: string,
databaseName: string,
): Promise<"created" | "exists"> {
if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(databaseName)) {
throw new Error(`Unsafe database name: ${databaseName}`);
}
const sql = createUtilitySql(url);
try {
const existing = await sql<{ one: number }[]>`
select 1 as one from pg_database where datname = ${databaseName} limit 1
`;
if (existing.length > 0) return "exists";
await sql.unsafe(`create database "${databaseName}" encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
return "created";
} finally {
await sql.end();
}
}
export async function resetPostgresDatabase(
url: string,
databaseName: string,
): Promise<"reset"> {
const quotedDatabaseName = quoteIdentifier(databaseName);
const sql = createUtilitySql(url);
try {
await sql`
select pg_terminate_backend(pid)
from pg_stat_activity
where datname = ${databaseName}
and pid <> pg_backend_pid()
`;
await sql.unsafe(`drop database if exists ${quotedDatabaseName}`);
await sql.unsafe(`create database ${quotedDatabaseName} encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
return "reset";
} finally {
await sql.end();
}
}
export type Db = ReturnType<typeof createDb>;