mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once
## Linked Issues or Issue Description
No public issue exists. Bug report:
**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.
**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.
**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.
`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.
**Paperclip version or commit**
`master` at 3c3e6c2d75.
**Deployment mode**
Any. The cost grows with every deploy of a hosted instance.
## What Changed
- `server/src/services/native-local-process-stop.ts`
- New `settledNativeProcessCandidates`: one fixed-text statement that
finds releasable runs. It reads only runs recorded in this server's PID
space (`process_host_hash`), sandbox runs (`remote`, with a non-local
lease), and rows saved before the hash (their latest identity event must
name this host). It excludes runs with an unsettled coordinator phase
and runs with no checkpoint and no provider evidence. Each PID-space
branch reads its own range of the new partial index.
- `releaseSettledNativeRunProcess` now reads the run, phase, evidence,
lease facts and identity match in one `for update` statement (it was 6
statements). It reads the lease rows again only for a sandbox run. The
new `inspectSettledNativeRunProcess` also returns whether a remote stop
acknowledgement can apply.
- Startup discovery uses the same candidate statement, so it opens no
transaction for an excluded run.
- `server/src/services/heartbeat/live-recovery-work.ts` (settled kind
only)
- The load runs the candidate statement and a separate indexed statement
for cancelled `cursor_cloud_api` runs whose Stop is still `requested`.
- The check uses the loaded entry. It no longer runs the load again for
each entry.
- A "pending" run backs off 30 s, 60 s, 120 s, up to 15 min. A
run-scoped hint (identity event, lease cleanup, stop receipt) resets the
backoff.
- A "skipped" run that the predicate cannot exclude (for example a birth
time mismatch on this host) waits for a run-scoped hint or a restart.
- `server/src/services/heartbeat/recovery.ts` (the settled handler
only): returns the outcome. It calls `acknowledgeRemoteStop` only when
the run has a non-local lease or is a cancelled `cursor_cloud_api` run.
For any other run that function reads two rows and writes nothing.
- `packages/db`: migration 0487 builds
`heartbeat_runs_settled_process_identity_idx` and
`heartbeat_runs_pending_api_stop_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. The migration is `online_compatible` and is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. `doc/DATABASE.md` describes both
indexes.
End state for each reason that a check cannot finish:
| Reason | Before | After | What re-admits the run |
| --- | --- | --- | --- |
| Recorded by another PID space | checked on every hint | not read |
only a server in that PID space reads it (a live older server in a
deploy overlap still releases its own runs) |
| Coordinator phase not settled | checked on every hint | not read | the
phase write; the finalizer sends a company hint in the same transaction
|
| No checkpoint and no provider evidence | checked on every hint | not
read | the next hint after a provider or turn event |
| Live local process, unreleased lease, missing sandbox receipt | every
hint and every 30 s | backoff 30 s to 15 min | a run-scoped hint resets
the backoff |
| Other same-host skip | checked on every hint | parked | a run-scoped
hint (an identity event sends one) or a restart |
## Verification
- `pnpm exec vitest run
src/__tests__/settled-native-process-rechecks.test.ts` (new, 12 tests,
real PostgreSQL, real queue and coordinator, fake clock):
- each exclusion: replaced container, unsettled phase, no evidence: 0
checks over 4 hints (red on `master`: 3 to 5 checks);
- a live older server and a stopped older server (different hash): the
new server never reads the run and writes no receipt; this server's own
run is released at start;
- re-admission: phase settles, late provider evidence, legacy rows with
no hash (this host released, another host not read), sandbox run
released from any server with its receipt;
- live process: at most 4 checks over 8 hints (red: 17), released on the
run-scoped hint after the process exits;
- one check is BEGIN, one `for update` statement and COMMIT (3
statements); the next hints do not check it; a run-scoped hint checks it
once more (red: 5 statements and a check on every hint);
- a restart costs one check for each pending run; startup discovery with
50 excluded runs sends 1 statement (red: 302).
- On `master` 11 of the 12 tests fail.
- Measurement on embedded PostgreSQL, 300 stuck runs, one company hint
per second for 300 s (fake clock), same harness before and after:
- before: 9.0 checks/s, 84.3 statements/s, 9.4 statements per check, 5.9
s CPU (2.2 ms per check);
- after: 0 checks/s, 3.2 statements/s for all seven residual queues
together, 0.16 s CPU.
- Load plan with 12,000 excluded finished runs (10% without a hash) in a
31,000-row `heartbeat_runs`: the candidate statement reads only index
entries for this server's hash, `remote` and `unknown`, and a bitmap
scan of the 1,200 legacy rows: 2.7 ms, 4,416 buffers. Without the index
the same exclusion read all 31,000 rows (31,127 buffers).
- Existing suites, one file at a time: all pass.
`live-recovery-work.test.ts` 51/51 (an earlier run under machine load
hit the `afterAll` hook timeout after all tests passed; the rerun passed
clean, and `native-safe-replacement.test.ts` hits the same hook timeout
on the base commit), `native-settled-process-release.test.ts` 31/31,
`native-local-process-stop.test.ts` 5/5, `heartbeat/recovery.test.ts`
107/107, `native-runner-restart-recovery.integration.test.ts` 231 passed
/ 14 skipped, `native-restart-recovery.test.ts` 49/49,
`native-controller-recovery.integration.test.ts` 30/30,
`retained-native-session-cleanup-work.test.ts` 29/29,
`native-provider-failure.integration.test.ts` 16/16,
`native-safe-replacement.test.ts` 70/70 (with the `afterAll` hook
timeout that also occurs on the base commit),
`native-remote-retry-admission.test.ts` 93/93.
- `pnpm check:module-boundaries`, `pnpm check:migrations` (packages/db),
server typecheck: pass.
- Not run: the full suite and browser suites.
## Risks
- Unresolved by design: a finished run whose server was replaced keeps
its process fields indefinitely and is never released, because only the
host that recorded a process can prove that it stopped, and the shared
prover refuses a controller-lease proof for a run with a process group.
This pull request only stops paying for it. These runs keep no stop
receipt (`native.local_process_stopped`), so what waits for that receipt
still waits: accounting close of the run's usage source, retained native
session cleanup, and instruction or directory working copies that wait
for proof that the run stopped. A follow-up must decide the ownership
rule for a replaced container.
- Rows saved before `process_host_hash` exist are read through the
legacy branch, which checks the latest identity event for each of them
on each load. On an instance with many such rows the load costs one
index lookup per row. The count can be measured with `select count(*)
from heartbeat_runs where process_host_hash is null and process_pid is
not null and status in
('failed','cancelled','timed_out','interrupted')`.
- A run that this server recorded but whose `process_host_hash` changed
(a new PID namespace on the same host, for example a container restart)
is no longer read here. Before, it was read and skipped, so no release
is lost.
- The backoff and parked state live in memory. A restart checks each
pending run once.
- The migration builds two partial indexes concurrently on
`heartbeat_runs`. It holds no long lock.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1520 lines
57 KiB
TypeScript
1520 lines
57 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { readFile, readdir } from "node:fs/promises";
|
|
import { fileURLToPath } from "node:url";
|
|
import postgres from "postgres";
|
|
import { installDatabaseWorkSignals } from "./work-signals.js";
|
|
import { installDatabaseResponseCapture } from "./response-capture.js";
|
|
import { installPreparedQueries } from "./prepared-queries.js";
|
|
import { drizzleWithLazyTransactionQueries } from "./lazy-transaction-queries.js";
|
|
import * as schema from "./schema/index.js";
|
|
|
|
const MIGRATIONS_FOLDER = fileURLToPath(new URL("./migrations", import.meta.url));
|
|
const DRIZZLE_MIGRATIONS_TABLE = "__drizzle_migrations";
|
|
const MIGRATIONS_JOURNAL_JSON = fileURLToPath(new URL("./migrations/meta/_journal.json", import.meta.url));
|
|
const PROJECT_ENVIRONMENT_OVERLAP = {
|
|
dropFile: "0466_drop_project_environment.sql",
|
|
dropSha256: "07ca1c666c6471fdb9ce7a0a5fb31bf1a074138466fa8daae50b53199b8e0d7c",
|
|
repairFile: "0467_fair_human_torch.sql",
|
|
repairSha256: "50ce0eb2567570b098bea1de92fe6a220d139e4ba83438195fbbad1be51e31a1",
|
|
} as const;
|
|
|
|
function createUtilitySql(url: string) {
|
|
return postgres(url, { max: 1, onnotice: () => {} });
|
|
}
|
|
|
|
type RegisteredPostgresClient = ReturnType<typeof postgres>;
|
|
|
|
/**
|
|
* Derives a registry key from a connection URL's host and port only. We must
|
|
* not retain or log the full URL, because it carries credentials.
|
|
*/
|
|
function hostPortKey(url: string): string {
|
|
const parsed = new URL(url);
|
|
return `${parsed.hostname}:${parsed.port || "5432"}`;
|
|
}
|
|
|
|
/**
|
|
* Same as `hostPortKey`, but returns `null` instead of throwing when the URL
|
|
* does not parse. `postgres(url)` tolerates a value `new URL()` rejects (an
|
|
* empty string falls back to the `PG*` environment variables), so `createDb`
|
|
* must tolerate it too: skip the registry entry and let the driver decide
|
|
* the outcome, instead of throwing an error the driver itself would not.
|
|
*/
|
|
function hostPortKeyOrNull(url: string): string | null {
|
|
try {
|
|
return hostPortKey(url);
|
|
} catch (error) {
|
|
if (error instanceof TypeError && (error as NodeJS.ErrnoException).code === "ERR_INVALID_URL") return null;
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
// Tracks every client `createDb` hands out, keyed by host and port, so a test
|
|
// fixture can end them before it stops the Postgres cluster they point at. A
|
|
// `WeakRef` plus `FinalizationRegistry` means a long-lived process (a real
|
|
// server) retains nothing extra: an unreferenced client is pruned on its own.
|
|
const clientsByHostPort = new Map<string, Set<WeakRef<RegisteredPostgresClient>>>();
|
|
const clientFinalizer = new FinalizationRegistry<{ hostPortKey: string; ref: WeakRef<RegisteredPostgresClient> }>(
|
|
({ hostPortKey, ref }) => {
|
|
const refs = clientsByHostPort.get(hostPortKey);
|
|
if (!refs) return;
|
|
refs.delete(ref);
|
|
if (refs.size === 0) clientsByHostPort.delete(hostPortKey);
|
|
},
|
|
);
|
|
|
|
function registerClient(key: string, client: RegisteredPostgresClient): void {
|
|
const ref = new WeakRef(client);
|
|
let refs = clientsByHostPort.get(key);
|
|
if (!refs) {
|
|
refs = new Set();
|
|
clientsByHostPort.set(key, refs);
|
|
}
|
|
refs.add(ref);
|
|
clientFinalizer.register(client, { hostPortKey: key, ref }, ref);
|
|
}
|
|
|
|
/**
|
|
* Ends every live client `createDb` handed out for the given URL's host and
|
|
* port, then forgets them. Call this before stopping a Postgres cluster: a
|
|
* client that outlives the cluster it points at can crash the process (a
|
|
* reserved connection's deferred write firing after the socket is gone).
|
|
* Swallows individual `end()` errors so one bad client cannot block the rest.
|
|
* An owned database on a shared test host must select database scope instead;
|
|
* its teardown cannot close another fixture's clients on that host.
|
|
*/
|
|
export async function closeRegisteredClients(url: string, options?: { scope: "database" }): Promise<void> {
|
|
const key = hostPortKeyOrNull(url);
|
|
if (key === null) return;
|
|
const database = options?.scope === "database" ? decodeURIComponent(new URL(url).pathname.slice(1)) : undefined;
|
|
if (database === "") throw new Error("Database-scoped teardown requires an explicit database name");
|
|
const refs = clientsByHostPort.get(key);
|
|
if (!refs) return;
|
|
|
|
const clients: RegisteredPostgresClient[] = [];
|
|
for (const ref of refs) {
|
|
const client = ref.deref();
|
|
if (client && database !== undefined && client.options.database !== database) continue;
|
|
refs.delete(ref);
|
|
clientFinalizer.unregister(ref);
|
|
if (client) clients.push(client);
|
|
}
|
|
if (refs.size === 0) clientsByHostPort.delete(key);
|
|
|
|
await Promise.all(clients.map((client) => client.end({ timeout: 1 }).catch(() => {})));
|
|
}
|
|
|
|
function isSafeIdentifier(value: string): boolean {
|
|
return /^[A-Za-z_][A-Za-z0-9_]*$/.test(value);
|
|
}
|
|
|
|
function quoteIdentifier(value: string): string {
|
|
if (!isSafeIdentifier(value)) throw new Error(`Unsafe SQL identifier: ${value}`);
|
|
return `"${value.replaceAll("\"", "\"\"")}"`;
|
|
}
|
|
|
|
function quoteLiteral(value: string): string {
|
|
return `'${value.replaceAll("'", "''")}'`;
|
|
}
|
|
|
|
function splitMigrationStatements(content: string): string[] {
|
|
return content
|
|
.split("--> statement-breakpoint")
|
|
.map((statement) => statement.trim())
|
|
.filter((statement) => statement.length > 0);
|
|
}
|
|
|
|
export type MigrationState =
|
|
| {
|
|
status: "upToDate";
|
|
tableCount: number;
|
|
availableMigrations: string[];
|
|
appliedMigrations: string[];
|
|
journalEntryCount: number;
|
|
}
|
|
| {
|
|
status: "needsMigrations";
|
|
tableCount: number;
|
|
availableMigrations: string[];
|
|
appliedMigrations: string[];
|
|
pendingMigrations: string[];
|
|
journalEntryCount: number;
|
|
reason: "no-migration-journal-empty-db" | "no-migration-journal-non-empty-db" | "pending-migrations";
|
|
};
|
|
|
|
export interface DatabaseClientOptions {
|
|
prepare?: boolean;
|
|
/** postgres.js `max` — connection pool size (driver default: 10). */
|
|
maxConnections?: number;
|
|
/** postgres.js `idle_timeout` in seconds (driver default: disabled). */
|
|
idleTimeoutSeconds?: number;
|
|
/** postgres.js `connect_timeout` in seconds (driver default: 30). */
|
|
connectTimeoutSeconds?: number;
|
|
/**
|
|
* postgres.js `max_lifetime` in seconds. Bounds how long one pooled
|
|
* connection is reused before the client replaces it (driver default: a
|
|
* random value between 30 and 60 minutes).
|
|
*/
|
|
maxLifetimeSeconds?: number;
|
|
/**
|
|
* postgres.js `connection.application_name`, shown in
|
|
* `pg_stat_activity.application_name`. Lets an operator tell Paperclip's
|
|
* pool apart from other clients of the same database (driver default:
|
|
* `postgres.js`).
|
|
*/
|
|
applicationName?: string;
|
|
/**
|
|
* Connection-level `statement_timeout` in milliseconds, sent as a startup
|
|
* parameter (no extra round trip). For dedicated single-purpose pools only.
|
|
*/
|
|
statementTimeoutMs?: number;
|
|
}
|
|
|
|
/**
|
|
* Idle pooled connections close after this many seconds unless
|
|
* `DATABASE_IDLE_TIMEOUT_SECONDS` says otherwise. The driver default keeps an
|
|
* idle connection open forever, so a process that stops issuing queries still
|
|
* holds every backend it ever opened. Set `DATABASE_IDLE_TIMEOUT_SECONDS=0`
|
|
* to restore the driver default.
|
|
*/
|
|
export const DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS = 60;
|
|
/** `application_name` reported to PostgreSQL unless `DATABASE_APPLICATION_NAME` overrides it. */
|
|
export const DEFAULT_DATABASE_APPLICATION_NAME = "paperclip";
|
|
|
|
function envBoolean(env: NodeJS.ProcessEnv, name: string): boolean | undefined {
|
|
const value = env[name]?.trim().toLowerCase();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (value === "true" || value === "1") return true;
|
|
if (value === "false" || value === "0") return false;
|
|
throw new Error(`${name} must be "true" or "false", got: ${env[name]}`);
|
|
}
|
|
|
|
function envPositiveInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (!/^[1-9]\d*$/.test(value)) {
|
|
throw new Error(`${name} must be a positive integer, got: ${env[name]}`);
|
|
}
|
|
return Number.parseInt(value, 10);
|
|
}
|
|
|
|
function envNonNegativeInteger(env: NodeJS.ProcessEnv, name: string): number | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
if (!/^(?:0|[1-9]\d*)$/.test(value)) {
|
|
throw new Error(`${name} must be a non-negative integer, got: ${env[name]}`);
|
|
}
|
|
return Number.parseInt(value, 10);
|
|
}
|
|
|
|
function envNonEmptyString(env: NodeJS.ProcessEnv, name: string): string | undefined {
|
|
const value = env[name]?.trim();
|
|
if (value === undefined || value === "") return undefined;
|
|
return value;
|
|
}
|
|
|
|
/**
|
|
* Database client tuning from the environment, so hosted deployments can
|
|
* adapt to their connection topology (pooled endpoints, network latency)
|
|
* without editing source. Every variable is optional. This function returns
|
|
* only the values the environment sets; `resolveDatabaseClientOptions` adds
|
|
* Paperclip's own defaults on top, and the driver defaults apply to the rest
|
|
* — self-hosted setups need none of these.
|
|
*/
|
|
export function databaseClientOptionsFromEnv(env: NodeJS.ProcessEnv = process.env): DatabaseClientOptions {
|
|
const options: DatabaseClientOptions = {};
|
|
const prepare = envBoolean(env, "DATABASE_PREPARED_STATEMENTS");
|
|
if (prepare !== undefined) options.prepare = prepare;
|
|
const maxConnections = envPositiveInteger(env, "DATABASE_POOL_MAX");
|
|
if (maxConnections !== undefined) options.maxConnections = maxConnections;
|
|
// `0` is allowed here: it disables idle reaping (the driver default).
|
|
const idleTimeoutSeconds = envNonNegativeInteger(env, "DATABASE_IDLE_TIMEOUT_SECONDS");
|
|
if (idleTimeoutSeconds !== undefined) options.idleTimeoutSeconds = idleTimeoutSeconds;
|
|
const connectTimeoutSeconds = envPositiveInteger(env, "DATABASE_CONNECT_TIMEOUT_SECONDS");
|
|
if (connectTimeoutSeconds !== undefined) options.connectTimeoutSeconds = connectTimeoutSeconds;
|
|
const maxLifetimeSeconds = envPositiveInteger(env, "DATABASE_MAX_LIFETIME_SECONDS");
|
|
if (maxLifetimeSeconds !== undefined) options.maxLifetimeSeconds = maxLifetimeSeconds;
|
|
const applicationName = envNonEmptyString(env, "DATABASE_APPLICATION_NAME");
|
|
if (applicationName !== undefined) options.applicationName = applicationName;
|
|
return options;
|
|
}
|
|
|
|
/**
|
|
* Fills in Paperclip's defaults for the options the caller left unset: idle
|
|
* connections are reaped after `DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS`, and the
|
|
* pool identifies itself as `DEFAULT_DATABASE_APPLICATION_NAME`. Everything
|
|
* else stays at the driver default. An explicit value (including
|
|
* `idleTimeoutSeconds: 0`) always wins over the default.
|
|
*/
|
|
export function resolveDatabaseClientOptions(options: DatabaseClientOptions): DatabaseClientOptions {
|
|
return {
|
|
...options,
|
|
idleTimeoutSeconds: options.idleTimeoutSeconds ?? DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS,
|
|
applicationName: options.applicationName ?? DEFAULT_DATABASE_APPLICATION_NAME,
|
|
};
|
|
}
|
|
|
|
export function postgresJsOptions(options: DatabaseClientOptions): Record<string, unknown> {
|
|
const driverOptions: Record<string, unknown> = {};
|
|
if (options.prepare !== undefined) driverOptions.prepare = options.prepare;
|
|
if (options.maxConnections !== undefined) driverOptions.max = options.maxConnections;
|
|
if (options.idleTimeoutSeconds !== undefined) driverOptions.idle_timeout = options.idleTimeoutSeconds;
|
|
if (options.connectTimeoutSeconds !== undefined) driverOptions.connect_timeout = options.connectTimeoutSeconds;
|
|
if (options.maxLifetimeSeconds !== undefined) driverOptions.max_lifetime = options.maxLifetimeSeconds;
|
|
const connection: Record<string, string> = {};
|
|
if (options.applicationName !== undefined) connection.application_name = options.applicationName;
|
|
if (options.statementTimeoutMs !== undefined) connection.statement_timeout = String(options.statementTimeoutMs);
|
|
if (Object.keys(connection).length) driverOptions.connection = connection;
|
|
return driverOptions;
|
|
}
|
|
|
|
// A long advisory-lock transaction must not borrow the normal pool that its
|
|
// work needs for progress writes. Keep connection configuration private to the
|
|
// originating Db lifetime; callers receive neither URLs nor credentials.
|
|
const ownedDatabaseListenerFactories = new WeakMap<object, () => Db>();
|
|
|
|
export const DEDICATED_CONNECTION_IDLE_MAX = 4;
|
|
|
|
type DedicatedConnectionSet = {
|
|
create(): Db;
|
|
idle: Db[];
|
|
ended: WeakSet<Db>;
|
|
closed: boolean;
|
|
};
|
|
const dedicatedConnectionSets = new WeakMap<object, DedicatedConnectionSet>();
|
|
|
|
function trackClientEnd(client: Db, onEnd: (timeoutSeconds: number | undefined) => Promise<void> | void): void {
|
|
const sql = client.$client;
|
|
const end = sql.end;
|
|
sql.end = async (options?: { timeout?: number }) => {
|
|
await Promise.all([onEnd(options?.timeout), end(options)]);
|
|
};
|
|
}
|
|
|
|
async function endDedicatedConnections(clients: Db[], timeoutSeconds: number): Promise<void> {
|
|
await Promise.all(clients.map(client => client.$client.end({ timeout: timeoutSeconds }).catch(() => undefined)));
|
|
}
|
|
|
|
export async function closeDedicatedDbConnections(db: Db, timeoutSeconds: number): Promise<void> {
|
|
const set = dedicatedConnectionSets.get(db);
|
|
if (!set) return;
|
|
set.closed = true;
|
|
await endDedicatedConnections(set.idle.splice(0), timeoutSeconds);
|
|
}
|
|
|
|
export interface OwnedDatabaseListener {
|
|
listen: Db["$client"]["listen"];
|
|
close(): Promise<void>;
|
|
}
|
|
|
|
export function createOwnedDatabaseListener(db: Db): OwnedDatabaseListener {
|
|
const factory = ownedDatabaseListenerFactories.get(db);
|
|
if (!factory) throw new Error("owned_listener_requires_create_db");
|
|
const owned = factory();
|
|
return {
|
|
listen: owned.$client.listen.bind(owned.$client),
|
|
close: () => owned.$client.end({ timeout: 1 }),
|
|
};
|
|
}
|
|
|
|
export async function withDedicatedDbConnection<T>(db: Db, action: (dedicated: Db) => Promise<T>): Promise<T> {
|
|
const set = dedicatedConnectionSets.get(db);
|
|
if (!set) throw new Error("dedicated_connection_requires_create_db");
|
|
let dedicated: Db | undefined;
|
|
while (!dedicated && set.idle.length) {
|
|
const candidate = set.idle.pop()!;
|
|
if (!set.ended.has(candidate)) dedicated = candidate;
|
|
}
|
|
dedicated ??= set.create();
|
|
let completed = false;
|
|
try {
|
|
const result = await action(dedicated);
|
|
completed = true;
|
|
return result;
|
|
} finally {
|
|
if (completed && !set.closed && !set.ended.has(dedicated) && set.idle.length < DEDICATED_CONNECTION_IDLE_MAX) {
|
|
set.idle.push(dedicated);
|
|
} else if (!set.ended.has(dedicated)) {
|
|
await dedicated.$client.end({ timeout: 1 });
|
|
}
|
|
}
|
|
}
|
|
|
|
// postgres.js pipelines a query onto a busy connection when none is idle, so
|
|
// a short liveness write can wait behind an unrelated slow statement with no
|
|
// client-side bound. Liveness writes (controller-lease renewal) get a small
|
|
// private pool, created on first use and shared for the owner's lifetime.
|
|
const livenessDbFactories = new WeakMap<object, () => Db>();
|
|
const livenessDbs = new WeakMap<object, Db>();
|
|
const LIVENESS_POOL_MAX_CONNECTIONS = 2;
|
|
|
|
/** The private liveness pool for a `createDb` client; other clients (test
|
|
* doubles, transactions) are returned unchanged. */
|
|
export function livenessDb(db: Db): Db {
|
|
const existing = livenessDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = livenessDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory();
|
|
livenessDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
const controllerLeaseDbFactories = new WeakMap<object, () => Db>();
|
|
const controllerLeaseDbs = new WeakMap<object, Db>();
|
|
|
|
export function controllerLeaseDb(db: Db): Db {
|
|
const existing = controllerLeaseDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = controllerLeaseDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory();
|
|
controllerLeaseDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
const controllerInstanceLeaseDbFactories = new WeakMap<object, (statementTimeoutMs: number | null) => Db>();
|
|
const controllerInstanceLeaseDbs = new WeakMap<object, Db>();
|
|
|
|
/** One private connection for the server's controller-instance lease, so its
|
|
* renewal never waits behind request or worker traffic on the main pool.
|
|
* `statementTimeoutMs` (first call wins) is a connection-level
|
|
* `statement_timeout`: the server cancels a slow renewal, so it never holds the
|
|
* single connection that the next renewal needs, at no extra round trip. */
|
|
export function controllerInstanceLeaseDb(db: Db, options: { statementTimeoutMs?: number | null } = {}): Db {
|
|
const existing = controllerInstanceLeaseDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = controllerInstanceLeaseDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const created = factory(options.statementTimeoutMs ?? null);
|
|
controllerInstanceLeaseDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
/** Replaces the lease connection with one that sends no `statement_timeout`
|
|
* startup parameter, for a connection pooler that refuses it. */
|
|
export function controllerInstanceLeaseDbWithoutStatementTimeout(db: Db): Db {
|
|
const factory = controllerInstanceLeaseDbFactories.get(db);
|
|
if (!factory) return db;
|
|
const previous = controllerInstanceLeaseDbs.get(db);
|
|
const created = factory(null);
|
|
controllerInstanceLeaseDbs.set(db, created);
|
|
void previous?.$client.end({ timeout: 0 }).catch(() => undefined);
|
|
return created;
|
|
}
|
|
|
|
/** Ends the liveness pool if one was created. The ended pool stays cached, so
|
|
* a late renewal fails like a query on the ended primary pool. */
|
|
export async function closeLivenessDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
await Promise.all([livenessDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
|
|
controllerLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds }),
|
|
controllerInstanceLeaseDbs.get(db)?.$client.end({ timeout: timeoutSeconds })]);
|
|
}
|
|
|
|
// Control checkpoints must not occupy the request or lease-renewal pools
|
|
// during an aligned burst. Share at most three lazy connections per owner.
|
|
const checkpointDbFactories = new WeakMap<object, () => Db>();
|
|
const checkpointDbs = new WeakMap<object, Db>();
|
|
const CHECKPOINT_POOL_MAX_CONNECTIONS = 3;
|
|
const closedCheckpointOwners = new WeakSet<object>();
|
|
|
|
/** The bounded controller checkpoint pool; transactions and test doubles keep
|
|
* their existing database when they have no owning client factory. */
|
|
export function checkpointDb(db: Db): Db {
|
|
const existing = checkpointDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = checkpointDbFactories.get(db);
|
|
if (!factory) return db;
|
|
if (closedCheckpointOwners.has(db)) throw new Error("checkpoint_pool_closed");
|
|
const created = factory();
|
|
checkpointDbs.set(db, created);
|
|
return created;
|
|
}
|
|
|
|
/** Fence lazy creation immediately, and keep any ended pool cached. */
|
|
export async function closeCheckpointDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
closedCheckpointOwners.add(db);
|
|
await checkpointDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
|
|
}
|
|
|
|
// Managed file publication holds an agent lock while transferring bytes. A
|
|
// stalled object store must not consume request or controller-renewal capacity.
|
|
const fileStorageDbFactories = new WeakMap<object, () => Db>();
|
|
const fileStorageDbs = new WeakMap<object, Db>();
|
|
const closedFileStorageOwners = new WeakSet<object>();
|
|
export function fileStorageDb(db: Db): Db {
|
|
const existing = fileStorageDbs.get(db);
|
|
if (existing) return existing;
|
|
const factory = fileStorageDbFactories.get(db);
|
|
if (!factory) return db; // Preserve the caller's transaction/savepoint.
|
|
if (closedFileStorageOwners.has(db)) throw new Error("file_storage_pool_closed");
|
|
const created = factory();
|
|
fileStorageDbs.set(db, created);
|
|
fileStorageDbs.set(created, created);
|
|
return created;
|
|
}
|
|
export async function closeFileStorageDb(db: Db, timeoutSeconds: number): Promise<void> {
|
|
closedFileStorageOwners.add(db);
|
|
await fileStorageDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
|
|
}
|
|
|
|
export function createDb(url: string, options?: DatabaseClientOptions) {
|
|
const resolved = resolveDatabaseClientOptions(options ?? databaseClientOptionsFromEnv());
|
|
return createDbClient(url, resolved);
|
|
}
|
|
|
|
function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: object, dedicatedSet?: DedicatedConnectionSet) {
|
|
const sql = postgres(url, postgresJsOptions(resolved));
|
|
if (resolved.prepare === true) installPreparedQueries(sql);
|
|
const key = hostPortKeyOrNull(url);
|
|
if (key) registerClient(key, sql);
|
|
// A disconnect can lose the response after a statement has committed.
|
|
// postgres.js calls that error "write CONNECTION_CLOSED" too, so the
|
|
// message cannot establish that replay is safe. Leave retries to callers
|
|
// that know the complete operation is idempotent.
|
|
const db = installDatabaseWorkSignals(drizzleWithLazyTransactionQueries(sql, schema), owner);
|
|
installDatabaseResponseCapture(db, schema);
|
|
const ownerSet = owner ? dedicatedConnectionSets.get(owner) : undefined;
|
|
if (dedicatedSet) {
|
|
trackClientEnd(db, () => { dedicatedSet.ended.add(db); });
|
|
dedicatedConnectionSets.set(db, dedicatedSet);
|
|
} else if (ownerSet) {
|
|
dedicatedConnectionSets.set(db, ownerSet);
|
|
} else {
|
|
const set: DedicatedConnectionSet = {
|
|
create: () => createDbClient(url, {
|
|
...resolved, maxConnections: 1, applicationName: "paperclip-workspace-finalization-lock",
|
|
}, db, set),
|
|
idle: [], ended: new WeakSet(), closed: false,
|
|
};
|
|
dedicatedConnectionSets.set(db, set);
|
|
trackClientEnd(db, timeoutSeconds => closeDedicatedDbConnections(db, timeoutSeconds ?? 1));
|
|
}
|
|
ownedDatabaseListenerFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: 1,
|
|
}, db));
|
|
livenessDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: LIVENESS_POOL_MAX_CONNECTIONS, applicationName: "paperclip-controller-lease",
|
|
}, db));
|
|
controllerLeaseDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: 1, applicationName: "paperclip-legacy-controller-lease",
|
|
}, db));
|
|
controllerInstanceLeaseDbFactories.set(db, statementTimeoutMs => createDbClient(url, {
|
|
...resolved, maxConnections: 1, applicationName: "paperclip-controller-instance-lease",
|
|
...(statementTimeoutMs !== null ? { statementTimeoutMs } : {}),
|
|
}, db));
|
|
checkpointDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: CHECKPOINT_POOL_MAX_CONNECTIONS, applicationName: "paperclip-runner-checkpoint",
|
|
}, db));
|
|
fileStorageDbFactories.set(db, () => createDbClient(url, {
|
|
...resolved, maxConnections: 2, applicationName: "paperclip-file-storage",
|
|
}, db));
|
|
return db;
|
|
}
|
|
|
|
export async function getPostgresDataDirectory(url: string): Promise<string | null> {
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
const rows = await sql<{ data_directory: string | null }[]>`
|
|
SELECT current_setting('data_directory', true) AS data_directory
|
|
`;
|
|
const actual = rows[0]?.data_directory;
|
|
return typeof actual === "string" && actual.length > 0 ? actual : null;
|
|
} catch {
|
|
return null;
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
async function listMigrationFiles(): Promise<string[]> {
|
|
const entries = await readdir(MIGRATIONS_FOLDER, { withFileTypes: true });
|
|
return entries
|
|
.filter((entry) => entry.isFile() && entry.name.endsWith(".sql"))
|
|
.map((entry) => entry.name)
|
|
.sort((a, b) => a.localeCompare(b));
|
|
}
|
|
|
|
type MigrationJournalFile = {
|
|
entries?: Array<{ idx?: number; tag?: string; when?: number }>;
|
|
};
|
|
|
|
type JournalMigrationEntry = {
|
|
fileName: string;
|
|
folderMillis: number;
|
|
order: number;
|
|
};
|
|
|
|
async function listJournalMigrationEntries(): Promise<JournalMigrationEntry[]> {
|
|
try {
|
|
const raw = await readFile(MIGRATIONS_JOURNAL_JSON, "utf8");
|
|
const parsed = JSON.parse(raw) as MigrationJournalFile;
|
|
if (!Array.isArray(parsed.entries)) return [];
|
|
return parsed.entries
|
|
.map((entry, entryIndex) => {
|
|
if (typeof entry?.tag !== "string") return null;
|
|
if (typeof entry?.when !== "number" || !Number.isFinite(entry.when)) return null;
|
|
const order = Number.isInteger(entry.idx) ? Number(entry.idx) : entryIndex;
|
|
return { fileName: `${entry.tag}.sql`, folderMillis: entry.when, order };
|
|
})
|
|
.filter((entry): entry is JournalMigrationEntry => entry !== null);
|
|
} catch {
|
|
return [];
|
|
}
|
|
}
|
|
|
|
async function listJournalMigrationFiles(): Promise<string[]> {
|
|
const entries = await listJournalMigrationEntries();
|
|
return entries.map((entry) => entry.fileName);
|
|
}
|
|
|
|
async function readMigrationFileContent(migrationFile: string): Promise<string> {
|
|
return readFile(new URL(`./migrations/${migrationFile}`, import.meta.url), "utf8");
|
|
}
|
|
|
|
async function orderMigrationsByJournal(migrationFiles: string[]): Promise<string[]> {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const orderByFileName = new Map(journalEntries.map((entry) => [entry.fileName, entry.order]));
|
|
return [...migrationFiles].sort((left, right) => {
|
|
const leftOrder = orderByFileName.get(left);
|
|
const rightOrder = orderByFileName.get(right);
|
|
if (leftOrder === undefined && rightOrder === undefined) return left.localeCompare(right);
|
|
if (leftOrder === undefined) return 1;
|
|
if (rightOrder === undefined) return -1;
|
|
if (leftOrder === rightOrder) return left.localeCompare(right);
|
|
return leftOrder - rightOrder;
|
|
});
|
|
}
|
|
|
|
type SqlExecutor = Pick<ReturnType<typeof postgres>, "unsafe">;
|
|
|
|
async function runInTransaction(sql: SqlExecutor, action: () => Promise<void>): Promise<void> {
|
|
await sql.unsafe("BEGIN");
|
|
try {
|
|
await action();
|
|
await sql.unsafe("COMMIT");
|
|
} catch (error) {
|
|
try {
|
|
await sql.unsafe("ROLLBACK");
|
|
} catch {
|
|
// Ignore rollback failures and surface the original error.
|
|
}
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
async function latestMigrationCreatedAt(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
): Promise<number | null> {
|
|
const rows = await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC NULLS LAST LIMIT 1`,
|
|
);
|
|
const value = Number(rows[0]?.created_at ?? Number.NaN);
|
|
return Number.isFinite(value) ? value : null;
|
|
}
|
|
|
|
function normalizeFolderMillis(value: number | null | undefined): number {
|
|
if (typeof value === "number" && Number.isFinite(value) && value >= 0) {
|
|
return Math.trunc(value);
|
|
}
|
|
return Date.now();
|
|
}
|
|
|
|
async function ensureMigrationJournalTable(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<{ migrationTableSchema: string; columnNames: Set<string> }> {
|
|
let migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
const drizzleSchema = quoteIdentifier("drizzle");
|
|
const migrationTable = quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE);
|
|
await sql.unsafe(`CREATE SCHEMA IF NOT EXISTS ${drizzleSchema}`);
|
|
await sql.unsafe(
|
|
`CREATE TABLE IF NOT EXISTS ${drizzleSchema}.${migrationTable} (id SERIAL PRIMARY KEY, hash text NOT NULL, created_at bigint)`,
|
|
);
|
|
migrationTableSchema = (await discoverMigrationTableSchema(sql)) ?? "drizzle";
|
|
}
|
|
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
return { migrationTableSchema, columnNames };
|
|
}
|
|
|
|
async function migrationHistoryEntryExists(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
columnNames: Set<string>,
|
|
migrationFile: string,
|
|
hash: string,
|
|
): Promise<boolean> {
|
|
const predicates: string[] = [];
|
|
if (columnNames.has("hash")) predicates.push(`hash = ${quoteLiteral(hash)}`);
|
|
if (columnNames.has("name")) predicates.push(`name = ${quoteLiteral(migrationFile)}`);
|
|
if (predicates.length === 0) return false;
|
|
|
|
const rows = await sql.unsafe<{ one: number }[]>(
|
|
`SELECT 1 AS one FROM ${qualifiedTable} WHERE ${predicates.join(" OR ")} LIMIT 1`,
|
|
);
|
|
return rows.length > 0;
|
|
}
|
|
|
|
async function recordMigrationHistoryEntry(
|
|
sql: SqlExecutor,
|
|
qualifiedTable: string,
|
|
columnNames: Set<string>,
|
|
migrationFile: string,
|
|
hash: string,
|
|
folderMillis: number,
|
|
): Promise<void> {
|
|
const insertColumns: string[] = [];
|
|
const insertValues: string[] = [];
|
|
|
|
if (columnNames.has("hash")) {
|
|
insertColumns.push(quoteIdentifier("hash"));
|
|
insertValues.push(quoteLiteral(hash));
|
|
}
|
|
if (columnNames.has("name")) {
|
|
insertColumns.push(quoteIdentifier("name"));
|
|
insertValues.push(quoteLiteral(migrationFile));
|
|
}
|
|
if (columnNames.has("created_at")) {
|
|
const latestCreatedAt = await latestMigrationCreatedAt(sql, qualifiedTable);
|
|
const createdAt = latestCreatedAt === null
|
|
? normalizeFolderMillis(folderMillis)
|
|
: Math.max(latestCreatedAt + 1, normalizeFolderMillis(folderMillis));
|
|
insertColumns.push(quoteIdentifier("created_at"));
|
|
insertValues.push(quoteLiteral(String(createdAt)));
|
|
}
|
|
|
|
if (insertColumns.length === 0) return;
|
|
|
|
await sql.unsafe(
|
|
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
|
|
);
|
|
}
|
|
|
|
// These idempotent migrations require top-level statements for batch commits
|
|
// or concurrent index builds. History is recorded only after completion.
|
|
const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
|
|
"0313_private_task_access.sql",
|
|
"0314_private_task_draft_assets.sql",
|
|
"0332_little_night_nurse.sql",
|
|
"0345_luxuriant_carmella_unuscione.sql",
|
|
"0369_late_logan.sql",
|
|
"0381_mean_scream.sql",
|
|
"0382_narrow_the_executioner.sql",
|
|
"0385_open_madame_web.sql",
|
|
"0386_steady_epoch.sql",
|
|
"0387_boring_professor_monster.sql",
|
|
"0388_ambitious_micromax.sql",
|
|
"0389_brave_hulk.sql",
|
|
"0390_salty_killraven.sql",
|
|
"0391_third_roulette.sql",
|
|
"0392_small_frightful_four.sql",
|
|
"0393_open_grandmaster.sql",
|
|
"0395_tranquil_firestar.sql",
|
|
"0396_tan_paladin.sql",
|
|
"0397_known_madrox.sql",
|
|
"0398_yielding_random.sql",
|
|
"0399_ambitious_onslaught.sql",
|
|
"0400_absent_phantom_reporter.sql",
|
|
"0401_yummy_ultimates.sql",
|
|
"0402_ancient_malice.sql",
|
|
"0403_natural_cerise.sql",
|
|
"0406_zippy_war_machine.sql",
|
|
"0407_greedy_star_brand.sql",
|
|
"0408_short_red_skull.sql",
|
|
"0409_uneven_zuras.sql",
|
|
"0410_freezing_kinsey_walden.sql",
|
|
"0411_real_scrambler.sql",
|
|
"0412_tearful_squadron_supreme.sql",
|
|
"0413_tired_kronos.sql",
|
|
"0414_closed_edwin_jarvis.sql",
|
|
"0419_slippery_impossible_man.sql",
|
|
"0432_strong_mole_man.sql",
|
|
"0435_premium_puff_adder.sql",
|
|
"0436_legal_reavers.sql",
|
|
"0447_legacy_controller_process.sql",
|
|
"0450_petite_frightful_four.sql",
|
|
"0453_curious_ben_parker.sql",
|
|
"0454_lumpy_ronan.sql",
|
|
"0455_pending_budget_cancellation_index.sql",
|
|
"0456_chat_review_lookup_indexes.sql",
|
|
"0457_native_session_run_index.sql",
|
|
"0458_chat_admission_issue_index.sql",
|
|
"0459_environment_lease_backup_scope_index.sql",
|
|
"0460_workspace_activity_context_index.sql",
|
|
"0461_created_from_legacy_origin_run_index.sql",
|
|
"0470_heartbeat_runs_queued_created_index.sql",
|
|
"0472_runner_api_capture_pending_deletes.sql",
|
|
"0476_wakeup_request_run_index.sql",
|
|
"0486_heartbeat_runs_company_finished_index.sql",
|
|
"0487_heartbeat_runs_settled_process_indexes.sql",
|
|
"0469_zippy_photon.sql",
|
|
|
|
|
|
"0384_funny_adam_warlock.sql",
|
|
"0383_tool_review_company_expiry.sql",
|
|
]);
|
|
|
|
async function applyPendingMigrationsManually(
|
|
url: string,
|
|
pendingMigrations: string[],
|
|
): Promise<void> {
|
|
if (pendingMigrations.length === 0) return;
|
|
|
|
const orderedPendingMigrations = await orderMigrationsByJournal(pendingMigrations);
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const folderMillisByFileName = new Map(
|
|
journalEntries.map((entry) => [entry.fileName, normalizeFolderMillis(entry.folderMillis)]),
|
|
);
|
|
|
|
const pool = createUtilitySql(url);
|
|
try {
|
|
const sql = await pool.reserve();
|
|
try {
|
|
// A session lock survives each batch COMMIT. Reserve the connection so
|
|
// pool rotation cannot release it while another migrator is waiting.
|
|
// A blocking lock SELECT retains its statement snapshot while waiting.
|
|
// A partial concurrent index build can wait for that snapshot, forming a
|
|
// deadlock with this lock. Each failed try ends its snapshot before sleep.
|
|
while (!(await sql<{ locked: boolean }[]>`
|
|
SELECT pg_try_advisory_lock(hashtextextended('paperclip:migrations', 0)) AS locked`)[0]?.locked) {
|
|
await new Promise((resolve) => setTimeout(resolve, 100));
|
|
}
|
|
const { migrationTableSchema, columnNames } = await ensureMigrationJournalTable(sql);
|
|
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
|
|
for (const migrationFile of orderedPendingMigrations) {
|
|
const migrationContent = await readMigrationFileContent(migrationFile);
|
|
const hash = createHash("sha256").update(migrationContent).digest("hex");
|
|
const existingEntry = await migrationHistoryEntryExists(
|
|
sql,
|
|
qualifiedTable,
|
|
columnNames,
|
|
migrationFile,
|
|
hash,
|
|
);
|
|
if (existingEntry) continue;
|
|
|
|
const applyMigration = async () => {
|
|
for (const statement of splitMigrationStatements(migrationContent)) {
|
|
if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
|
|
// A cancelled concurrent build leaves an invalid index. IF NOT
|
|
// EXISTS alone would skip it on retry and journal an incomplete index.
|
|
const index = statement.replace(/^\s*--.*$/gm, "").trim()
|
|
.match(/^CREATE (?:UNIQUE )?INDEX CONCURRENTLY IF NOT EXISTS "([A-Za-z_][A-Za-z0-9_]*)"/i);
|
|
if (index) {
|
|
const invalid = await sql<{ invalid: boolean }[]>`
|
|
SELECT NOT i.indisvalid AS invalid FROM pg_index i
|
|
JOIN pg_class c ON c.oid = i.indexrelid
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public' AND c.relname = ${index[1]}`;
|
|
if (invalid[0]?.invalid) {
|
|
await sql.unsafe(`DROP INDEX CONCURRENTLY ${quoteIdentifier("public")}.${quoteIdentifier(index[1])}`);
|
|
}
|
|
}
|
|
}
|
|
// Older dev schemas can lack a replaced constraint. Preserve the
|
|
// published SQL/hash while allowing the harmless drop to proceed.
|
|
const executable = statement.replace(
|
|
/^(\s*ALTER TABLE "[^"]+" DROP CONSTRAINT) (?!IF EXISTS\b)/i,
|
|
"$1 IF EXISTS ",
|
|
);
|
|
await sql.unsafe(executable);
|
|
}
|
|
|
|
await recordMigrationHistoryEntry(
|
|
sql,
|
|
qualifiedTable,
|
|
columnNames,
|
|
migrationFile,
|
|
hash,
|
|
folderMillisByFileName.get(migrationFile) ?? Date.now(),
|
|
);
|
|
};
|
|
if (migrationFile === PROJECT_ENVIRONMENT_OVERLAP.dropFile) {
|
|
const repairFile = PROJECT_ENVIRONMENT_OVERLAP.repairFile;
|
|
const repairContent = await readMigrationFileContent(repairFile);
|
|
const repairHash = createHash("sha256").update(repairContent).digest("hex");
|
|
const dropIndex = orderedPendingMigrations.indexOf(migrationFile);
|
|
if (hash !== PROJECT_ENVIRONMENT_OVERLAP.dropSha256
|
|
|| repairHash !== PROJECT_ENVIRONMENT_OVERLAP.repairSha256
|
|
|| orderedPendingMigrations[dropIndex + 1] !== repairFile
|
|
|| await migrationHistoryEntryExists(sql, qualifiedTable, columnNames, repairFile, repairHash)) {
|
|
throw new Error("Project environment migration pair does not match the qualified overlap repair");
|
|
}
|
|
await runInTransaction(sql, async () => {
|
|
await sql.unsafe('LOCK TABLE "public"."projects" IN ACCESS EXCLUSIVE MODE');
|
|
await sql.unsafe(`CREATE TEMP TABLE paperclip_project_environment_overlap ON COMMIT DROP AS
|
|
SELECT id, environment_id FROM "public"."projects" WHERE environment_id IS NOT NULL`);
|
|
await applyMigration();
|
|
for (const statement of splitMigrationStatements(repairContent)) await sql.unsafe(statement);
|
|
await sql.unsafe(`UPDATE "public"."projects" AS project SET environment_id = legacy.environment_id
|
|
FROM pg_temp.paperclip_project_environment_overlap AS legacy WHERE project.id = legacy.id`);
|
|
await recordMigrationHistoryEntry(sql, qualifiedTable, columnNames, repairFile, repairHash,
|
|
folderMillisByFileName.get(repairFile) ?? Date.now());
|
|
});
|
|
} else if (MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS.has(migrationFile)) {
|
|
await applyMigration();
|
|
} else {
|
|
await runInTransaction(sql, applyMigration);
|
|
}
|
|
}
|
|
} finally {
|
|
try {
|
|
await sql`SELECT pg_advisory_unlock(hashtextextended('paperclip:migrations', 0))`;
|
|
} catch {
|
|
// Preserve the migration error if the session died; PostgreSQL releases
|
|
// its lock on disconnect, and pool.end() closes any remaining connection.
|
|
}
|
|
sql.release();
|
|
}
|
|
} finally {
|
|
await pool.end();
|
|
}
|
|
}
|
|
|
|
async function mapHashesToMigrationFiles(migrationFiles: string[]): Promise<Map<string, string>> {
|
|
const mapped = new Map<string, string>();
|
|
|
|
await Promise.all(
|
|
migrationFiles.map(async (migrationFile) => {
|
|
const content = await readMigrationFileContent(migrationFile);
|
|
const hash = createHash("sha256").update(content).digest("hex");
|
|
mapped.set(hash, migrationFile);
|
|
if (migrationFile === "0355_agent_tool_definition_hooks.sql") {
|
|
mapped.set("7ac76adac0220da522c78e51da20490aeb0f90512b4370a05af9f8dd0f75eda6", migrationFile);
|
|
}
|
|
}),
|
|
);
|
|
|
|
return mapped;
|
|
}
|
|
|
|
async function getMigrationTableColumnNames(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationTableSchema: string,
|
|
): Promise<Set<string>> {
|
|
const columns = await sql.unsafe<{ column_name: string }[]>(
|
|
`
|
|
SELECT column_name
|
|
FROM information_schema.columns
|
|
WHERE table_schema = ${quoteLiteral(migrationTableSchema)}
|
|
AND table_name = ${quoteLiteral(DRIZZLE_MIGRATIONS_TABLE)}
|
|
`,
|
|
);
|
|
return new Set(columns.map((column) => column.column_name));
|
|
}
|
|
|
|
async function tableExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM information_schema.tables
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function columnExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
columnName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM information_schema.columns
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
AND column_name = ${columnName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function columnHasDataType(
|
|
sql: ReturnType<typeof postgres>,
|
|
tableName: string,
|
|
columnName: string,
|
|
dataType: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ dataType: string; udtName: string }[]>`
|
|
SELECT data_type AS "dataType", udt_name AS "udtName"
|
|
FROM information_schema.columns
|
|
WHERE table_schema = 'public'
|
|
AND table_name = ${tableName}
|
|
AND column_name = ${columnName}
|
|
`;
|
|
const expected = dataType.toLowerCase();
|
|
return rows.some((row) => (
|
|
row.dataType.toLowerCase() === expected || row.udtName.toLowerCase() === expected
|
|
));
|
|
}
|
|
|
|
async function indexExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
indexName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_class c
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public'
|
|
AND c.relkind = 'i'
|
|
AND c.relname = ${indexName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function constraintExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
constraintName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_constraint c
|
|
JOIN pg_namespace n ON n.oid = c.connamespace
|
|
WHERE n.nspname = 'public'
|
|
AND c.conname = ${constraintName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function functionExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
functionName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_proc p
|
|
JOIN pg_namespace n ON n.oid = p.pronamespace
|
|
WHERE n.nspname = 'public'
|
|
AND p.proname = ${functionName}
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function triggerExists(
|
|
sql: ReturnType<typeof postgres>,
|
|
triggerName: string,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ exists: boolean }[]>`
|
|
SELECT EXISTS (
|
|
SELECT 1
|
|
FROM pg_trigger t
|
|
JOIN pg_class c ON c.oid = t.tgrelid
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public'
|
|
AND t.tgname = ${triggerName}
|
|
AND NOT t.tgisinternal
|
|
) AS exists
|
|
`;
|
|
return rows[0]?.exists ?? false;
|
|
}
|
|
|
|
async function heartbeatEventSequencesAreUnique(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ unique: boolean }[]>`
|
|
SELECT NOT EXISTS (
|
|
SELECT 1
|
|
FROM heartbeat_run_events
|
|
GROUP BY run_id, seq
|
|
HAVING count(*) > 1
|
|
) AS unique
|
|
`;
|
|
return rows[0]?.unique ?? false;
|
|
}
|
|
|
|
async function heartbeatNextEventSequencesAreCurrent(
|
|
sql: ReturnType<typeof postgres>,
|
|
): Promise<boolean> {
|
|
const rows = await sql<{ current: boolean }[]>`
|
|
SELECT NOT EXISTS (
|
|
SELECT 1
|
|
FROM heartbeat_runs run
|
|
WHERE run.next_event_seq IS DISTINCT FROM COALESCE((
|
|
SELECT max(event.seq) + 1
|
|
FROM heartbeat_run_events event
|
|
WHERE event.run_id = run.id
|
|
), 1)
|
|
) AS current
|
|
`;
|
|
return rows[0]?.current ?? false;
|
|
}
|
|
|
|
async function migrationStatementAlreadyApplied(
|
|
sql: ReturnType<typeof postgres>,
|
|
statement: string,
|
|
): Promise<boolean> {
|
|
const normalized = statement
|
|
.replace(/^\s*--.*$/gm, "")
|
|
.replace(/\s+/g, " ")
|
|
.trim();
|
|
|
|
const createTableMatch = normalized.match(/^CREATE TABLE(?: IF NOT EXISTS)? "([^"]+)"/i);
|
|
if (createTableMatch) {
|
|
return tableExists(sql, createTableMatch[1]);
|
|
}
|
|
|
|
const addColumnMatch = normalized.match(
|
|
/^ALTER TABLE "([^"]+)" ADD COLUMN(?: IF NOT EXISTS)? "([^"]+)"/i,
|
|
);
|
|
if (addColumnMatch) {
|
|
return columnExists(sql, addColumnMatch[1], addColumnMatch[2]);
|
|
}
|
|
|
|
const alterColumnTypeMatch = normalized.match(
|
|
/^ALTER TABLE "([^"]+)" ALTER COLUMN "([^"]+)" SET DATA TYPE ([A-Za-z0-9_]+)/i,
|
|
);
|
|
if (alterColumnTypeMatch) {
|
|
return columnHasDataType(
|
|
sql,
|
|
alterColumnTypeMatch[1],
|
|
alterColumnTypeMatch[2],
|
|
alterColumnTypeMatch[3],
|
|
);
|
|
}
|
|
|
|
const createIndexMatch = normalized.match(/^CREATE (?:UNIQUE )?INDEX(?: IF NOT EXISTS)? "([^"]+)"/i);
|
|
if (createIndexMatch) {
|
|
return indexExists(sql, createIndexMatch[1]);
|
|
}
|
|
|
|
const addConstraintMatch = normalized.match(/^ALTER TABLE "([^"]+)" ADD CONSTRAINT "([^"]+)"/i);
|
|
if (addConstraintMatch) {
|
|
return constraintExists(sql, addConstraintMatch[2]);
|
|
}
|
|
|
|
const createFunctionMatch = normalized.match(
|
|
/^CREATE OR REPLACE FUNCTION "?([A-Za-z_][A-Za-z0-9_]*)"?\s*\(/i,
|
|
);
|
|
if (createFunctionMatch) {
|
|
return functionExists(sql, createFunctionMatch[1]);
|
|
}
|
|
|
|
const createTriggerMatch = normalized.match(
|
|
/^CREATE TRIGGER "?([A-Za-z_][A-Za-z0-9_]*)"?/i,
|
|
);
|
|
if (createTriggerMatch) {
|
|
return triggerExists(sql, createTriggerMatch[1]);
|
|
}
|
|
|
|
// These native-runner repairs have persistent postconditions. Verify them
|
|
// instead of replaying them when a restored database is missing only the
|
|
// migration-history row.
|
|
if (
|
|
normalized.startsWith("WITH ranked AS (")
|
|
&& normalized.includes('UPDATE "heartbeat_run_events" AS event')
|
|
) {
|
|
return heartbeatEventSequencesAreUnique(sql);
|
|
}
|
|
if (
|
|
normalized.startsWith('UPDATE "heartbeat_runs" AS run')
|
|
&& normalized.includes('SET "next_event_seq" = COALESCE')
|
|
) {
|
|
return heartbeatNextEventSequencesAreCurrent(sql);
|
|
}
|
|
|
|
// If we cannot reason about a statement safely, require manual migration.
|
|
return false;
|
|
}
|
|
|
|
async function migrationContentAlreadyApplied(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationContent: string,
|
|
): Promise<boolean> {
|
|
const statements = splitMigrationStatements(migrationContent);
|
|
if (statements.length === 0) return false;
|
|
|
|
for (const statement of statements) {
|
|
const applied = await migrationStatementAlreadyApplied(sql, statement);
|
|
if (!applied) return false;
|
|
}
|
|
|
|
return true;
|
|
}
|
|
|
|
async function loadAppliedMigrations(
|
|
sql: ReturnType<typeof postgres>,
|
|
migrationTableSchema: string,
|
|
availableMigrations: string[],
|
|
): Promise<string[]> {
|
|
const quotedSchema = quoteIdentifier(migrationTableSchema);
|
|
const qualifiedTable = `${quotedSchema}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
|
|
if (columnNames.has("name")) {
|
|
const rows = await sql.unsafe<{ name: string }[]>(`SELECT name FROM ${qualifiedTable} ORDER BY id`);
|
|
return rows.map((row) => row.name).filter((name): name is string => Boolean(name));
|
|
}
|
|
|
|
if (columnNames.has("hash")) {
|
|
const rows = await sql.unsafe<{ hash: string }[]>(`SELECT hash FROM ${qualifiedTable} ORDER BY id`);
|
|
const hashesToMigrationFiles = await mapHashesToMigrationFiles(availableMigrations);
|
|
const appliedFromHashes = rows
|
|
.map((row) => hashesToMigrationFiles.get(row.hash))
|
|
.filter((name): name is string => Boolean(name));
|
|
|
|
if (appliedFromHashes.length > 0) {
|
|
// Best-effort: when all hashes resolve, this is authoritative.
|
|
if (appliedFromHashes.length === rows.length) return appliedFromHashes;
|
|
|
|
// Partial hash resolution can happen when files have changed; return what we can trust.
|
|
return appliedFromHashes;
|
|
}
|
|
|
|
// Fallback only when hashes are unavailable/unresolved.
|
|
if (columnNames.has("created_at")) {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
if (journalEntries.length > 0) {
|
|
const lastDbRows = await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} ORDER BY created_at DESC LIMIT 1`,
|
|
);
|
|
const lastCreatedAt = Number(lastDbRows[0]?.created_at ?? -1);
|
|
if (Number.isFinite(lastCreatedAt) && lastCreatedAt >= 0) {
|
|
return journalEntries
|
|
.filter((entry) => availableMigrations.includes(entry.fileName))
|
|
.filter((entry) => entry.folderMillis <= lastCreatedAt)
|
|
.map((entry) => entry.fileName)
|
|
.slice(0, rows.length);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
const rows = await sql.unsafe<{ id: number }[]>(`SELECT id FROM ${qualifiedTable} ORDER BY id`);
|
|
const journalMigrationFiles = await listJournalMigrationFiles();
|
|
const appliedFromIds = rows
|
|
.map((row) => journalMigrationFiles[row.id - 1])
|
|
.filter((name): name is string => Boolean(name));
|
|
if (appliedFromIds.length > 0) return appliedFromIds;
|
|
|
|
return availableMigrations.slice(0, Math.max(0, rows.length));
|
|
}
|
|
|
|
export type MigrationHistoryReconcileResult = {
|
|
repairedMigrations: string[];
|
|
remainingMigrations: string[];
|
|
};
|
|
|
|
export async function reconcilePendingMigrationHistory(
|
|
url: string,
|
|
): Promise<MigrationHistoryReconcileResult> {
|
|
const state = await inspectMigrations(url);
|
|
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
|
|
return { repairedMigrations: [], remainingMigrations: [] };
|
|
}
|
|
|
|
const sql = createUtilitySql(url);
|
|
const repairedMigrations: string[] = [];
|
|
|
|
try {
|
|
const journalEntries = await listJournalMigrationEntries();
|
|
const folderMillisByFile = new Map(journalEntries.map((entry) => [entry.fileName, entry.folderMillis]));
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
return { repairedMigrations, remainingMigrations: state.pendingMigrations };
|
|
}
|
|
|
|
const columnNames = await getMigrationTableColumnNames(sql, migrationTableSchema);
|
|
const qualifiedTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
|
|
for (const migrationFile of state.pendingMigrations) {
|
|
const migrationContent = await readMigrationFileContent(migrationFile);
|
|
const alreadyApplied = await migrationContentAlreadyApplied(sql, migrationContent);
|
|
if (!alreadyApplied) break;
|
|
|
|
const hash = createHash("sha256").update(migrationContent).digest("hex");
|
|
const folderMillis = folderMillisByFile.get(migrationFile) ?? Date.now();
|
|
const existingByHash = columnNames.has("hash")
|
|
? await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} WHERE hash = ${quoteLiteral(hash)} ORDER BY created_at DESC LIMIT 1`,
|
|
)
|
|
: [];
|
|
const existingByName = columnNames.has("name")
|
|
? await sql.unsafe<{ created_at: string | number | null }[]>(
|
|
`SELECT created_at FROM ${qualifiedTable} WHERE name = ${quoteLiteral(migrationFile)} ORDER BY created_at DESC LIMIT 1`,
|
|
)
|
|
: [];
|
|
if (existingByHash.length > 0 || existingByName.length > 0) {
|
|
if (columnNames.has("created_at")) {
|
|
const existingHashCreatedAt = Number(existingByHash[0]?.created_at ?? -1);
|
|
if (existingByHash.length > 0 && Number.isFinite(existingHashCreatedAt) && existingHashCreatedAt < folderMillis) {
|
|
await sql.unsafe(
|
|
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE hash = ${quoteLiteral(hash)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
|
|
);
|
|
}
|
|
|
|
const existingNameCreatedAt = Number(existingByName[0]?.created_at ?? -1);
|
|
if (existingByName.length > 0 && Number.isFinite(existingNameCreatedAt) && existingNameCreatedAt < folderMillis) {
|
|
await sql.unsafe(
|
|
`UPDATE ${qualifiedTable} SET created_at = ${quoteLiteral(String(folderMillis))} WHERE name = ${quoteLiteral(migrationFile)} AND created_at < ${quoteLiteral(String(folderMillis))}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
repairedMigrations.push(migrationFile);
|
|
continue;
|
|
}
|
|
|
|
const insertColumns: string[] = [];
|
|
const insertValues: string[] = [];
|
|
|
|
if (columnNames.has("hash")) {
|
|
insertColumns.push(quoteIdentifier("hash"));
|
|
insertValues.push(quoteLiteral(hash));
|
|
}
|
|
if (columnNames.has("name")) {
|
|
insertColumns.push(quoteIdentifier("name"));
|
|
insertValues.push(quoteLiteral(migrationFile));
|
|
}
|
|
if (columnNames.has("created_at")) {
|
|
insertColumns.push(quoteIdentifier("created_at"));
|
|
insertValues.push(quoteLiteral(String(folderMillis)));
|
|
}
|
|
|
|
if (insertColumns.length === 0) break;
|
|
|
|
await sql.unsafe(
|
|
`INSERT INTO ${qualifiedTable} (${insertColumns.join(", ")}) VALUES (${insertValues.join(", ")})`,
|
|
);
|
|
repairedMigrations.push(migrationFile);
|
|
}
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
|
|
const refreshed = await inspectMigrations(url);
|
|
return {
|
|
repairedMigrations,
|
|
remainingMigrations:
|
|
refreshed.status === "needsMigrations" ? refreshed.pendingMigrations : [],
|
|
};
|
|
}
|
|
|
|
async function discoverMigrationTableSchema(sql: ReturnType<typeof postgres>): Promise<string | null> {
|
|
const rows = await sql<{ schemaName: string }[]>`
|
|
SELECT n.nspname AS "schemaName"
|
|
FROM pg_class c
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE c.relname = ${DRIZZLE_MIGRATIONS_TABLE} AND c.relkind = 'r'
|
|
`;
|
|
|
|
if (rows.length === 0) return null;
|
|
|
|
const drizzleSchema = rows.find(({ schemaName }) => schemaName === "drizzle");
|
|
if (drizzleSchema) return drizzleSchema.schemaName;
|
|
|
|
const publicSchema = rows.find(({ schemaName }) => schemaName === "public");
|
|
if (publicSchema) return publicSchema.schemaName;
|
|
|
|
return rows[0]?.schemaName ?? null;
|
|
}
|
|
|
|
export async function inspectMigrations(url: string): Promise<MigrationState> {
|
|
const sql = createUtilitySql(url);
|
|
|
|
try {
|
|
const availableMigrations = await listMigrationFiles();
|
|
const tableCountResult = await sql<{ count: number }[]>`
|
|
select count(*)::int as count
|
|
from information_schema.tables
|
|
where table_schema = 'public'
|
|
and table_type = 'BASE TABLE'
|
|
`;
|
|
const tableCount = tableCountResult[0]?.count ?? 0;
|
|
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
if (!migrationTableSchema) {
|
|
if (tableCount > 0) {
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations: [],
|
|
pendingMigrations: availableMigrations,
|
|
journalEntryCount: 0,
|
|
reason: "no-migration-journal-non-empty-db",
|
|
};
|
|
}
|
|
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations: [],
|
|
pendingMigrations: availableMigrations,
|
|
journalEntryCount: 0,
|
|
reason: "no-migration-journal-empty-db",
|
|
};
|
|
}
|
|
|
|
const qualifiedMigrationTable = `${quoteIdentifier(migrationTableSchema)}.${quoteIdentifier(DRIZZLE_MIGRATIONS_TABLE)}`;
|
|
const journalCountRows = await sql.unsafe<{ count: number }[]>(
|
|
`SELECT count(*)::int AS count FROM ${qualifiedMigrationTable}`,
|
|
);
|
|
const journalEntryCount = journalCountRows[0]?.count ?? 0;
|
|
const appliedMigrations = await loadAppliedMigrations(sql, migrationTableSchema, availableMigrations);
|
|
const pendingMigrations = availableMigrations.filter((name) => !appliedMigrations.includes(name));
|
|
if (pendingMigrations.length === 0) {
|
|
return {
|
|
status: "upToDate",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations,
|
|
journalEntryCount,
|
|
};
|
|
}
|
|
|
|
return {
|
|
status: "needsMigrations",
|
|
tableCount,
|
|
availableMigrations,
|
|
appliedMigrations,
|
|
pendingMigrations,
|
|
journalEntryCount,
|
|
reason: "pending-migrations",
|
|
};
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function applyPendingMigrations(url: string): Promise<void> {
|
|
const initialState = await inspectMigrations(url);
|
|
if (initialState.status === "upToDate") return;
|
|
|
|
if (initialState.reason === "no-migration-journal-empty-db") {
|
|
// Use the same per-file executor for bootstrap and upgrades: Drizzle wraps
|
|
// the whole journal in one transaction and cannot run committing backfills.
|
|
await applyPendingMigrationsManually(url, initialState.pendingMigrations);
|
|
const bootstrappedState = await inspectMigrations(url);
|
|
if (bootstrappedState.status === "upToDate") return;
|
|
throw new Error(
|
|
`Failed to bootstrap migrations: ${bootstrappedState.pendingMigrations.join(", ")}`,
|
|
);
|
|
}
|
|
|
|
if (initialState.reason === "no-migration-journal-non-empty-db") {
|
|
throw new Error(
|
|
"Database has tables but no migration journal; automatic migration is unsafe. Initialize migration history manually.",
|
|
);
|
|
}
|
|
|
|
let state = await inspectMigrations(url);
|
|
if (state.status === "upToDate") return;
|
|
|
|
const repair = await reconcilePendingMigrationHistory(url);
|
|
if (repair.repairedMigrations.length > 0) {
|
|
state = await inspectMigrations(url);
|
|
if (state.status === "upToDate") return;
|
|
}
|
|
|
|
if (state.status !== "needsMigrations" || state.reason !== "pending-migrations") {
|
|
throw new Error("Migrations are still pending after migration-history reconciliation; run inspectMigrations for details.");
|
|
}
|
|
|
|
await applyPendingMigrationsManually(url, state.pendingMigrations);
|
|
|
|
const finalState = await inspectMigrations(url);
|
|
if (finalState.status !== "upToDate") {
|
|
throw new Error(
|
|
`Failed to apply pending migrations: ${finalState.pendingMigrations.join(", ")}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
export type MigrationBootstrapResult =
|
|
| { migrated: true; reason: "migrated-empty-db"; tableCount: 0 }
|
|
| { migrated: false; reason: "already-migrated"; tableCount: number }
|
|
| { migrated: false; reason: "not-empty-no-migration-journal"; tableCount: number };
|
|
|
|
export async function migratePostgresIfEmpty(url: string): Promise<MigrationBootstrapResult> {
|
|
const sql = createUtilitySql(url);
|
|
|
|
try {
|
|
const migrationTableSchema = await discoverMigrationTableSchema(sql);
|
|
|
|
const tableCountResult = await sql<{ count: number }[]>`
|
|
select count(*)::int as count
|
|
from information_schema.tables
|
|
where table_schema = 'public'
|
|
and table_type = 'BASE TABLE'
|
|
`;
|
|
|
|
const tableCount = tableCountResult[0]?.count ?? 0;
|
|
|
|
if (migrationTableSchema) {
|
|
return { migrated: false, reason: "already-migrated", tableCount };
|
|
}
|
|
|
|
if (tableCount > 0) {
|
|
return { migrated: false, reason: "not-empty-no-migration-journal", tableCount };
|
|
}
|
|
|
|
await applyPendingMigrationsManually(url, await listMigrationFiles());
|
|
|
|
return { migrated: true, reason: "migrated-empty-db", tableCount: 0 };
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function ensurePostgresDatabase(
|
|
url: string,
|
|
databaseName: string,
|
|
): Promise<"created" | "exists"> {
|
|
if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(databaseName)) {
|
|
throw new Error(`Unsafe database name: ${databaseName}`);
|
|
}
|
|
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
const existing = await sql<{ one: number }[]>`
|
|
select 1 as one from pg_database where datname = ${databaseName} limit 1
|
|
`;
|
|
if (existing.length > 0) return "exists";
|
|
|
|
await sql.unsafe(`create database "${databaseName}" encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
|
|
return "created";
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export async function resetPostgresDatabase(
|
|
url: string,
|
|
databaseName: string,
|
|
): Promise<"reset"> {
|
|
const quotedDatabaseName = quoteIdentifier(databaseName);
|
|
const sql = createUtilitySql(url);
|
|
try {
|
|
await sql`
|
|
select pg_terminate_backend(pid)
|
|
from pg_stat_activity
|
|
where datname = ${databaseName}
|
|
and pid <> pg_backend_pid()
|
|
`;
|
|
await sql.unsafe(`drop database if exists ${quotedDatabaseName}`);
|
|
await sql.unsafe(`create database ${quotedDatabaseName} encoding 'UTF8' lc_collate 'C' lc_ctype 'C' template template0`);
|
|
return "reset";
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}
|
|
|
|
export type Db = ReturnType<typeof createDb>;
|